Greetings, I am running a very high-volume FreeRADIUS installation on RHEL 5 (not my choice), and noticed that FreeRADIUS was periodically dying after instantiating a great many worker servers. I looked at the output in GDB and got: Continuing. [New LWP 11400] [tcsetpgrp failed in terminal_inferior: Operation not permitted] (no debugging symbols found) [Thread debugging using libthread_db enabled] [New Thread 47272748113904 (LWP 11350)] [New Thread 1328978240 (LWP 11441)] [New Thread 1318488384 (LWP 11439)] [New Thread 1370937664 (LWP 11438)] [New Thread 1307998528 (LWP 11437)] [New Thread 1360447808 (LWP 11436)] [New Thread 1140160832 (LWP 11435)] [New Thread 1297508672 (LWP 11434)] [New Thread 1287018816 (LWP 11433)] [New Thread 1276528960 (LWP 11432)] [New Thread 1266039104 (LWP 11431)] [New Thread 1119181120 (LWP 11430)] [New Thread 1255549248 (LWP 11429)] [New Thread 1245059392 (LWP 11428)] [New Thread 1234569536 (LWP 11427)] [New Thread 1129670976 (LWP 11423)] [New Thread 1203099968 (LWP 11422)] [New Thread 1339468096 (LWP 11421)] [New Thread 1108691264 (LWP 11420)] [New Thread 1349957952 (LWP 11419)] [New Thread 1213589824 (LWP 11412)] [New Thread 1224079680 (LWP 11411)] [New Thread 1094392128 (LWP 11410)] [New Thread 1538775360 (LWP 11406)] [New Thread 1528285504 (LWP 11405)] [New Thread 1517795648 (LWP 11404)] [New Thread 1192610112 (LWP 11403)] [New Thread 1507305792 (LWP 11402)] [New Thread 1496815936 (LWP 11401)] [New Thread 1486326080 (LWP 11400)] [New Thread 1475836224 (LWP 11399)] [New Thread 1465346368 (LWP 11398)] [New Thread 1454856512 (LWP 11397)] [New Thread 1444366656 (LWP 11396)] [New Thread 1433876800 (LWP 11395)] [New Thread 1423386944 (LWP 11394)] [New Thread 1412897088 (LWP 11393)] [New Thread 1161140544 (LWP 11392)] Program received signal SIGTRAP, Trace/breakpoint trap. [Switching to Thread 1486326080 (LWP 11400)] 0x00002afe89997d19 in dl_open_worker () from /lib64/ld-linux-x86-64.so.2 (gdb) Continuing. Program received signal SIGABRT, Aborted. 0x00002afe8ad3d155 in raise () from /lib64/libc.so.6 Is this a bug in the x64 libdl? -- Alex -- Alex Balashov Evariste Systems Web : http://www.evaristesys.com/ Tel : (+1) (678) 954-0670 Direct : (+1) (678) 954-0671 Mobile : (+1) (706) 338-8599
Alex Balashov wrote:
Greetings,
I am running a very high-volume FreeRADIUS installation on RHEL 5 (not my choice), and noticed that FreeRADIUS was periodically dying after instantiating a great many worker servers. I looked at the output in GDB and got:
Continuing.
[New LWP 11400] [tcsetpgrp failed in terminal_inferior: Operation not permitted] (no debugging symbols found) [Thread debugging using libthread_db enabled] [New Thread 47272748113904 (LWP 11350)] [New Thread 1328978240 (LWP 11441)] [New Thread 1318488384 (LWP 11439)] [New Thread 1370937664 (LWP 11438)] [New Thread 1307998528 (LWP 11437)] [New Thread 1360447808 (LWP 11436)] [New Thread 1140160832 (LWP 11435)] [New Thread 1297508672 (LWP 11434)] [New Thread 1287018816 (LWP 11433)] [New Thread 1276528960 (LWP 11432)] [New Thread 1266039104 (LWP 11431)] [New Thread 1119181120 (LWP 11430)] [New Thread 1255549248 (LWP 11429)] [New Thread 1245059392 (LWP 11428)] [New Thread 1234569536 (LWP 11427)] [New Thread 1129670976 (LWP 11423)] [New Thread 1203099968 (LWP 11422)] [New Thread 1339468096 (LWP 11421)] [New Thread 1108691264 (LWP 11420)] [New Thread 1349957952 (LWP 11419)] [New Thread 1213589824 (LWP 11412)] [New Thread 1224079680 (LWP 11411)] [New Thread 1094392128 (LWP 11410)] [New Thread 1538775360 (LWP 11406)] [New Thread 1528285504 (LWP 11405)] [New Thread 1517795648 (LWP 11404)] [New Thread 1192610112 (LWP 11403)] [New Thread 1507305792 (LWP 11402)] [New Thread 1496815936 (LWP 11401)] [New Thread 1486326080 (LWP 11400)] [New Thread 1475836224 (LWP 11399)] [New Thread 1465346368 (LWP 11398)] [New Thread 1454856512 (LWP 11397)] [New Thread 1444366656 (LWP 11396)] [New Thread 1433876800 (LWP 11395)] [New Thread 1423386944 (LWP 11394)] [New Thread 1412897088 (LWP 11393)] [New Thread 1161140544 (LWP 11392)]
Program received signal SIGTRAP, Trace/breakpoint trap. [Switching to Thread 1486326080 (LWP 11400)] 0x00002afe89997d19 in dl_open_worker () from /lib64/ld-linux-x86-64.so.2 (gdb) Continuing.
Program received signal SIGABRT, Aborted. 0x00002afe8ad3d155 in raise () from /lib64/libc.so.6
Is this a bug in the x64 libdl?
-- Alex
Clarification: I ask because I did not reset any breakpoints. -- Alex Balashov Evariste Systems Web : http://www.evaristesys.com/ Tel : (+1) (678) 954-0670 Direct : (+1) (678) 954-0671 Mobile : (+1) (706) 338-8599
Ah, for a clearer picture, I tried running under gdb with the no-fork option: [root@radius2 radius]# gdb /usr/sbin/radiusd GNU gdb Red Hat Linux (6.5-37.el5_2.2rh) Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"...(no debugging symbols found) Using host libthread_db library "/lib64/libthread_db.so.1". (gdb) r -f Starting program: /usr/sbin/radiusd -f (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) [Thread debugging using libthread_db enabled] (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) Thu Aug 14 20:17:47 2008 : Info: Starting - reading configuration files ... [New Thread 47044196995056 (LWP 11633)] (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) ---Type <return> to continue, or q <return> to quit--- (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) [New Thread 1098144064 (LWP 11662)] [New Thread 1108633920 (LWP 11663)] [New Thread 1119123776 (LWP 11664)] [New Thread 1129613632 (LWP 11665)] [New Thread 1140103488 (LWP 11666)] [New Thread 1150593344 (LWP 11667)] [New Thread 1161083200 (LWP 11668)] [New Thread 1171573056 (LWP 11669)] [New Thread 1182062912 (LWP 11670)] [New Thread 1192552768 (LWP 11671)] [New Thread 1203042624 (LWP 11677)] [New Thread 1213532480 (LWP 11678)] [Thread 1098144064 (LWP 11662) exited] [New Thread 1224022336 (LWP 11679)] [New Thread 1098144064 (LWP 11680)] [New Thread 1234512192 (LWP 11684)] [Thread 1108633920 (LWP 11663) exited] [New Thread 1245002048 (LWP 11685)] [New Thread 1108633920 (LWP 11686)] [New Thread 1255491904 (LWP 11687)] [New Thread 1265981760 (LWP 11688)] [New Thread 1276471616 (LWP 11689)] [New Thread 1286961472 (LWP 11690)] [Thread 1119123776 (LWP 11664) exited] [New Thread 1297451328 (LWP 11691)] [New Thread 1119123776 (LWP 11692)] [New Thread 1307941184 (LWP 11693)] [New Thread 1318431040 (LWP 11694)] [New Thread 1328920896 (LWP 11695)] [New Thread 1339410752 (LWP 11696)] [New Thread 1349900608 (LWP 11698)] *** glibc detected *** /usr/sbin/radiusd: free(): invalid pointer: 0x00002ac95891ca60 *** (no debugging symbols found) ======= Backtrace: ========= /lib64/libc.so.6[0x2ac954229634] /lib64/libc.so.6(cfree+0x8c)[0x2ac95422cc5c] /usr/lib64/libpq.so.4[0x2ac95726cdf2] /usr/lib64/rlm_sql_postgresql.so[0x2ac957061e12] /usr/lib64/rlm_sql.so(rlm_sql_query+0x63)[0x2ac956e5db33] /usr/lib64/rlm_sql.so[0x2ac956e5c271] /usr/sbin/radiusd(modcall+0xb2)[0x2ac952c0dc52] /usr/sbin/radiusd[0x2ac952c0e241] /usr/sbin/radiusd(modcall+0x27c)[0x2ac952c0de1c] /usr/sbin/radiusd(rad_accounting+0xcd)[0x2ac952c056ed] /usr/sbin/radiusd(rad_respond+0x11a)[0x2ac952c0f1ea] /usr/sbin/radiusd[0x2ac952c15e51] /lib64/libpthread.so.0[0x2ac9534812f7] /lib64/libc.so.6(clone+0x6d)[0x2ac954289b6d] ======= Memory map: ======== 40d45000-40d46000 ---p 40d45000 00:00 0 40d46000-41746000 rw-p 40d46000 00:00 0 41746000-41747000 ---p 41746000 00:00 0 41747000-42147000 rw-p 41747000 00:00 0 42147000-42148000 ---p 42147000 00:00 0 42148000-42b48000 rw-p 42148000 00:00 0 42b48000-42b49000 ---p 42b48000 00:00 0 42b49000-43549000 rw-p 42b49000 00:00 0 43549000-4354a000 ---p 43549000 00:00 0 4354a000-43f4a000 rw-p 4354a000 00:00 0 43f4a000-43f4b000 ---p 43f4a000 00:00 0 43f4b000-4494b000 rw-p 43f4b000 00:00 0 4494b000-4494c000 ---p 4494b000 00:00 0 4494c000-4534c000 rw-p 4494c000 00:00 0 4534c000-4534d000 ---p 4534c000 00:00 0 4534d000-45d4d000 rw-p 4534d000 00:00 0 45d4d000-45d4e000 ---p 45d4d000 00:00 0 45d4e000-4674e000 rw-p 45d4e000 00:00 0 4674e000-4674f000 ---p 4674e000 00:00 0 4674f000-4714f000 rw-p 4674f000 00:00 0 4714f000-47150000 ---p 4714f000 00:00 0 47150000-47b50000 rw-p 47150000 00:00 0 47b50000-47b51000 ---p 47b50000 00:00 0 47b51000-48551000 rw-p 47b51000 00:00 0 48551000-48552000 ---p 48551000 00:00 0 48552000-48f52000 rw-p 48552000 00:00 0 48f52000-48f53000 ---p 48f52000 00:00 0 48f53000-49953000 rw-p 48f53000 00:00 0 49953000-49954000 ---p 49953000 00:00 0 49954000-4a354000 rw-p 49954000 00:00 0 4a354000-4a355000 ---p 4a354000 00:00 0 4a355000-4ad55000 rw-p 4a355000 00:00 0 4ad55000-4ad56000 ---p 4ad55000 00:00 0 4ad56000-4b756000 rw-p 4ad56000 00:00 0 4b756000-4b757000 ---p 4b756000 00:00 0 4b757000-4c157000 rw-p 4b757000 00:00 0 4c157000-4c158000 ---p 4c157000 00:00 0 4c158000-4cb58000 rw-p 4c158000 00:00 0 4cb58000-4cb59000 ---p 4cb58000 00:00 0 4cb59000-4d559000 rw-p 4cb59000 00:00 0 4d559000-4d55a000 ---p 4d559000 00:00 0 4d55a000-4df5a000 rw-p 4d55a000 00:00 0 4df5a000-4df5b000 ---p 4df5a000 00:00 0 4df5b000-4e95b000 rw-p 4df5b000 00:00 0 4e95b000-4e95c000 ---p 4e95b000 00:00 0 4e95c000-4f35c000 rw-p 4e95c000 00:00 0 4f35c000-4f35d000 ---p 4f35c000 00:00 0 4f35d000-4fd5d000 rw-p 4f35d000 00:00 0 4fd5d000-4fd5e000 ---p 4fd5d000 00:00 0 4fd5e000-5075e000 rw-p 4fd5e000 00:00 0 2aaaaaaae000-2aaaaabae000 rw-p 2aaaaaaae000 00:00 0 2aaaaabcb000-2aaaaabd8000 r-xp 00000000 fd:00 2193674 /lib64/libgcc_s-4.1.2-20080102.so.1 2aaaaabd8000-2aaaaadd8000 ---p 0000d000 fd:00 2193674 /lib64/libgcc_s-4.1.2-20080102.so.1 2aaaaadd8000-2aaaaadd9000 rw-p 0000d000 fd:00 2193674 /lib64/libgcc_s-4.1.2-20080102.so.1 2aaaac000000-2aaaac03e000 rw-p 2aaaac000000 00:00 0 2aaaac03e000-2aaab0000000 ---p 2aaaac03e000 00:00 0 2aaab0000000-2aaab0044000 rw-p 2aaab0000000 00:00 0 2aaab0044000-2aaab4000000 ---p 2aaab0044000 00:00 0 2aaab4000000-2aaab4038000 rw-p 2aaab4000000 00:00 0 2aaab4038000-2aaab8000000 ---p 2aaab4038000 00:00 0 2ac952bfe000-2ac952c21000 r-xp 00000000 fd:00 12332504 /usr/sbin/radiusd 2ac952e21000-2ac952e2f000 rw-p 00023000 fd:00 12332504 /usr/sbin/radiusd 2ac952e2f000-2ac952e32000 rw-p 2ac952e2f000 00:00 0 2ac952e32000-2ac952e4c000 r-xp 00000000 fd:00 2193651 /lib64/ld-2.5.so 2ac952e4c000-2ac952e4e000 rw-p 2ac952e4c000 00:00 0 2ac952e67000-2ac952e68000 rw-p 2ac952e67000 00:00 0 2ac95304c000-2ac95304d000 r--p 0001a000 fd:00 2193651 /lib64/ld-2.5.so 2ac95304d000-2ac95304e000 rw-p 0001b000 fd:00 2193651 Program received signal SIGABRT, Aborted. [Switching to Thread 1129613632 (LWP 11665)] 0x00002ac9541e8155 in raise () from /lib64/libc.so.6 Any idea what gives there? -- Alex Balashov Evariste Systems Web : http://www.evaristesys.com/ Tel : (+1) (678) 954-0670 Direct : (+1) (678) 954-0671 Mobile : (+1) (706) 338-8599
Alex Balashov wrote:
Ah, for a clearer picture, I tried running under gdb with the no-fork option:
*** glibc detected *** /usr/sbin/radiusd: free(): invalid pointer: 0x00002ac95891ca60 *** (no debugging symbols found) ======= Backtrace: ========= /lib64/libc.so.6[0x2ac954229634] /lib64/libc.so.6(cfree+0x8c)[0x2ac95422cc5c] /usr/lib64/libpq.so.4[0x2ac95726cdf2] /usr/lib64/rlm_sql_postgresql.so[0x2ac957061e12] /usr/lib64/rlm_sql.so(rlm_sql_query+0x63)[0x2ac956e5db33] /usr/lib64/rlm_sql.so[0x2ac956e5c271] /usr/sbin/radiusd(modcall+0xb2)[0x2ac952c0dc52] /usr/sbin/radiusd[0x2ac952c0e241] /usr/sbin/radiusd(modcall+0x27c)[0x2ac952c0de1c] /usr/sbin/radiusd(rad_accounting+0xcd)[0x2ac952c056ed] /usr/sbin/radiusd(rad_respond+0x11a)[0x2ac952c0f1ea] /usr/sbin/radiusd[0x2ac952c15e51] /lib64/libpthread.so.0[0x2ac9534812f7] /lib64/libc.so.6(clone+0x6d)[0x2ac954289b6d] ======= Memory map: ========
Any idea what gives there?
Well, it's an invalid "free", obviously, which indicates pointer corruption or something similar. What's odd is that we're running a relatively high-volume server on RHEL5 & Postgres and it's fine. Which version of FreeRadius is this?
On Fri, August 15, 2008 5:56 am, Phil Mayers wrote:
Well, it's an invalid "free", obviously, which indicates pointer corruption or something similar. What's odd is that we're running a relatively high-volume server on RHEL5 & Postgres and it's fine.
2.0.5 (tried both from binary and source0. Postgres 8.3.3. -- Alex Balashov Evariste Systems Web : http://www.evaristesys.com/ Tel : (+1) (678) 954-0670 Direct : (+1) (678) 954-0671 Mobile : (+1) (706) 338-8599
On Fri, Aug 15, 2008 at 01:00:06PM -0400, Alex Balashov wrote:
On Fri, August 15, 2008 5:56 am, Phil Mayers wrote:
Well, it's an invalid "free", obviously, which indicates pointer corruption or something similar. What's odd is that we're running a relatively high-volume server on RHEL5 & Postgres and it's fine.
2.0.5 (tried both from binary and source0. Postgres 8.3.3.
I'm running it against the RHEL5 postgres 8.1 - that seems to be the only difference betwee your setup and mine. As Alan says, you'll need to rebuild with debugging symbols enabled and extract the line number under GDB.
I'm running it against the RHEL5 postgres 8.1 - that seems to be the only difference betwee your setup and mine.
As Alan says, you'll need to rebuild with debugging symbols enabled and extract the line number under GDB.
Actually, on Fedora & RHEL you don't need to rebuild with debugging symbols on. All packages built for Fedora & RHEL always have matching debuginfo packages which can optionally be installed, which once installed will give you line number information in the debugger. The packages are still optimized, all installing the debuginfo packages does it add symbol information in a standard location. You should be able to install like this: % yum install freeradius-debuginfo postgresql-debuginfo -- John Dennis <jdennis@redhat.com>
John Dennis wrote:
Actually, on Fedora & RHEL you don't need to rebuild with debugging symbols on. All packages built for Fedora & RHEL always have matching debuginfo packages which can optionally be installed, which once installed will give you line number information in the debugger.
Nice. That should help. Alan DeKok.
Alan DeKok wrote:
John Dennis wrote:
Actually, on Fedora & RHEL you don't need to rebuild with debugging symbols on. All packages built for Fedora & RHEL always have matching debuginfo packages which can optionally be installed, which once installed will give you line number information in the debugger.
Nice. That should help.
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Whatever the problem was, it definitely went away on 32-bit Debian. Same version of Postgres, libpq, FreeRADIUS, etc. About the only thing that could possibly be different is the precise glibc2 revision. -- Alex Balashov Evariste Systems Web : http://www.evaristesys.com/ Tel : (+1) (678) 954-0670 Direct : (+1) (678) 954-0671 Mobile : (+1) (706) 338-8599
Alex Balashov wrote:
Alan DeKok wrote:
John Dennis wrote:
Actually, on Fedora & RHEL you don't need to rebuild with debugging symbols on. All packages built for Fedora & RHEL always have matching debuginfo packages which can optionally be installed, which once installed will give you line number information in the debugger.
Nice. That should help.
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Whatever the problem was, it definitely went away on 32-bit Debian. Same version of Postgres, libpq, FreeRADIUS, etc. About the only thing that could possibly be different is the precise glibc2 revision.
FWIW, I recently investigated a similar memory corruption problem with freeradius reported by glibc on RHEL 5. I was not able to reproduce it on a current RHEL 5.2 system. The fact I couldn't reproduce it does not in and of itself mean anything, the memory corruption could require a specific series of events to trigger it, but I do believe glibc had been updated between the original report and my efforts at reproducing it. This might corroborate your observation (or might not :-) -- John Dennis <jdennis@redhat.com>
Alex Balashov wrote:
Ah, for a clearer picture, I tried running under gdb with the no-fork option: ... [New Thread 1349900608 (LWP 11698)] *** glibc detected *** /usr/sbin/radiusd: free(): invalid pointer: 0x00002ac95891ca60 *** (no debugging symbols found) ======= Backtrace: ========= /lib64/libc.so.6[0x2ac954229634] /lib64/libc.so.6(cfree+0x8c)[0x2ac95422cc5c] /usr/lib64/libpq.so.4[0x2ac95726cdf2] /usr/lib64/rlm_sql_postgresql.so[0x2ac957061e12]
Can you find out what line number that's at? You'll have to build with debugging symbols.
Any idea what gives there?
a) libpq is broken b) rlm_sql_postgresql is broken Alan DeKok.
participants (4)
-
Alan DeKok -
Alex Balashov -
John Dennis -
Phil Mayers