FreeRADIUS exiting with Signal 11 on FreeBSD
I am running FreeRADIUS 2.1.10 on 64bit FreeBSD 8.1-RELEASE-p1 with LDAP auth and accounting records being sent to file and MySQL. I get the following in radius.log: Sun Dec 5 05:14:57 2010 : Info: Exiting normally. and the following in messages: Dec 5 05:14:57 cat kernel: pid 68292 (radiusd), uid 133: exited on signal 11 This has also occurred on two other occasions. I found somone else in the list archives who had a similar sounding problem on FreeBSD but there didn't appear to be an issue. Any suggestions as to what I can do to avoid this problem? Thanks, Hugh Blandford -- Hugh Blandford Island Internet ph 1300 130 428 mb 0412 016 875
I've been having the same problem for quite some time. I'm currently running FreeRADIUS 2.1.10 on i386 FreeBSD 8.1-RELEASE. Accounting packets are written file + two extra detail logs which are proxied. For me I think this "Exiting normally" problem started happening after upgrading FreeBSD from either 7.x to 8 or 6.x to 7.x. It's been happening for over a year now. Sometimes it happens three times a day and sometimes it runs fine for a month. This accounts for the last two months: Tue Oct 12 23:47:18 2010 : Info: Exiting normally. Wed Oct 20 14:16:56 2010 : Info: Exiting normally. Thu Oct 28 09:44:02 2010 : Info: Exiting normally. Mon Nov 29 15:51:27 2010 : Info: Exiting normally. Mon Nov 29 19:47:59 2010 : Info: Exiting normally. Tue Nov 30 08:07:55 2010 : Info: Exiting normally. Wed Dec 1 13:37:36 2010 : Info: Exiting normally. Thu Dec 2 02:25:16 2010 : Info: Exiting normally. Fri Dec 3 02:40:00 2010 : Info: Exiting normally. Fri Dec 3 13:37:48 2010 : Info: Exiting normally. Fri Dec 3 20:04:01 2010 : Info: Exiting normally. Sun Dec 12 07:30:27 2010 : Info: Exiting normally. I don't see any system in the behaviour. It happens at random hours. This is a secondary server, so the number of accounting requests per day varies between 0 and 20000, whereas on a normal day it serves around 100. If nobody has a clue what might cause this behaviour, perhaps somebody is willing to support me/us in debugging this! Thank you, Dánial -- View this message in context: http://freeradius.1045715.n5.nabble.com/FreeRADIUS-exiting-with-Signal-11-on... Sent from the FreeRadius - User mailing list archive at Nabble.com.
On Sun, Dec 12, 2010 at 02:14:25PM -0800, Danial wrote:
I've been having the same problem for quite some time.
I'm currently running FreeRADIUS 2.1.10 on i386 FreeBSD 8.1-RELEASE. Accounting packets are written file + two extra detail logs which are proxied.
For me I think this "Exiting normally" problem started happening after upgrading FreeBSD from either 7.x to 8 or 6.x to 7.x. It's been happening for over a year now. Sometimes it happens three times a day and sometimes it runs fine for a month. This accounts for the last two months:
Tue Oct 12 23:47:18 2010 : Info: Exiting normally. Wed Oct 20 14:16:56 2010 : Info: Exiting normally. Thu Oct 28 09:44:02 2010 : Info: Exiting normally. Mon Nov 29 15:51:27 2010 : Info: Exiting normally. Mon Nov 29 19:47:59 2010 : Info: Exiting normally. Tue Nov 30 08:07:55 2010 : Info: Exiting normally. Wed Dec 1 13:37:36 2010 : Info: Exiting normally. Thu Dec 2 02:25:16 2010 : Info: Exiting normally. Fri Dec 3 02:40:00 2010 : Info: Exiting normally. Fri Dec 3 13:37:48 2010 : Info: Exiting normally. Fri Dec 3 20:04:01 2010 : Info: Exiting normally. Sun Dec 12 07:30:27 2010 : Info: Exiting normally.
I don't see any system in the behaviour. It happens at random hours. This is a secondary server, so the number of accounting requests per day varies between 0 and 20000, whereas on a normal day it serves around 100.
If nobody has a clue what might cause this behaviour, perhaps somebody is willing to support me/us in debugging this!
I've not had any issues with FreeRADIUS on FreeBSD 7 since version 2.0ish. Around 2.0 FreeRADIUS ate memory and would be killed when the box ran out of swap. That was a known bug in FreeRADIUS and since I upgraded beyond the fix for that, no problems. What is logged in the 3 or 4 hundred lines before the exits? Any pattern in those messages? What is logged in /var/log/messages at those times? -- Scott Lambert KC5MLE Unix SysAdmin lambert@lambertfam.org
Scott Lambert wrote:
What is logged in the 3 or 4 hundred lines before the exits? Any pattern in those messages? What is logged in /var/log/messages at those times?
On the last exit, Des 12, NO entry is written in /var/log/messages. On Des 3 there were, in my case, signal 11 entries, but the usual exits happen without one. I've tried on several occations to search logs to correlate with the exiting events and also with the, often quite few, radius accounting packets, to no avail. I'm prepared to debug this, but I need someone who's willing to tell me what to do and to interpret the output. It's no point for me to start running ktrace, gdb and whatever, without being able to interpret it. -- View this message in context: http://freeradius.1045715.n5.nabble.com/FreeRADIUS-exiting-with-Signal-11-on... Sent from the FreeRadius - User mailing list archive at Nabble.com.
Scott Lambert wrote:
What is logged in the 3 or 4 hundred lines before the exits? Any pattern in those messages? What is logged in /var/log/messages at those times?
On the last exit, Des 12, NO entry is written in /var/log/messages. On Des 3 there were, in my case, signal 11 entries, but the usual exits happen without one. I've tried on several occations to search logs to correlate with the exiting events and also with the, often quite few, radius accounting packets, to no avail. I'm prepared to debug this, but I need someone who's willing to tell me what to do and to interpret the output. It's no point for me to start running ktrace, gdb and whatever, without being able to interpret it. -- View this message in context: http://freeradius.1045715.n5.nabble.com/FreeRADIUS-exiting-with-Signal-11-on... Sent from the FreeRadius - User mailing list archive at Nabble.com.
Hi Scott, On 14/12/2010 11:21, Scott Lambert wrote:
On Sun, Dec 12, 2010 at 02:14:25PM -0800, Danial wrote:
I've been having the same problem for quite some time.
I'm currently running FreeRADIUS 2.1.10 on i386 FreeBSD 8.1-RELEASE. Accounting packets are written file + two extra detail logs which are proxied.
For me I think this "Exiting normally" problem started happening after upgrading FreeBSD from either 7.x to 8 or 6.x to 7.x. It's been happening for over a year now. Sometimes it happens three times a day and sometimes it runs fine for a month. This accounts for the last two months:
Tue Oct 12 23:47:18 2010 : Info: Exiting normally. Wed Oct 20 14:16:56 2010 : Info: Exiting normally. Thu Oct 28 09:44:02 2010 : Info: Exiting normally. Mon Nov 29 15:51:27 2010 : Info: Exiting normally. Mon Nov 29 19:47:59 2010 : Info: Exiting normally. Tue Nov 30 08:07:55 2010 : Info: Exiting normally. Wed Dec 1 13:37:36 2010 : Info: Exiting normally. Thu Dec 2 02:25:16 2010 : Info: Exiting normally. Fri Dec 3 02:40:00 2010 : Info: Exiting normally. Fri Dec 3 13:37:48 2010 : Info: Exiting normally. Fri Dec 3 20:04:01 2010 : Info: Exiting normally. Sun Dec 12 07:30:27 2010 : Info: Exiting normally.
I don't see any system in the behaviour. It happens at random hours. This is a secondary server, so the number of accounting requests per day varies between 0 and 20000, whereas on a normal day it serves around 100.
If nobody has a clue what might cause this behaviour, perhaps somebody is willing to support me/us in debugging this! I've not had any issues with FreeRADIUS on FreeBSD 7 since version 2.0ish. Around 2.0 FreeRADIUS ate memory and would be killed when the box ran out of swap. That was a known bug in FreeRADIUS and since I upgraded beyond the fix for that, no problems.
What is logged in the 3 or 4 hundred lines before the exits? Any pattern in those messages? What is logged in /var/log/messages at those times?
I can see no pattern in my radius.log that is consistent for all of the exits. As this server is fairly quiet /var/log/messages is really quite empty: an example Dec 4 12:54:26 cata su: test to root on /dev/pts/1 Dec 5 05:14:57 cata kernel: pid 68292 (radiusd), uid 133: exited on signal 11 Dec 5 08:39:50 cata su: test to root on /dev/pts/1 Dec 5 08:51:23 cata slapd[93791]: OTP unavailable because can't read/write key database /etc/opiekeys: Permission denied Dec 5 20:47:11 cata kernel: pid 79883 (radiusd), uid 133: exited on signal 11 top is also similarly boring.....at the present, I don't know what it looks like just after FreeRADIUS exits. I will investigate it next time it happens. Regards, Hugh -- Hugh Blandford Island Internet ph 1300 130 428 mb 0412 016 875
Hugh Blandford wrote:
I can see no pattern in my radius.log that is consistent for all of the exits. As this server is fairly quiet /var/log/messages is really quite empty:
Watch the CPU and memory usage by the server.
top is also similarly boring.....at the present, I don't know what it looks like just after FreeRADIUS exits. I will investigate it next time it happens.
If you can, monitor any signals being sent to the server. Alan DeKok.
Ok. I've been able to monitor another "Exiting normally" event: Fri Dec 17 06:38:39 2010 : Info: Exiting normally. Alan DeKok-2 wrote:
Watch the CPU and memory usage by the server.
I've been monitoring the server using Cacti for the last few days. Memory usage shows plenty of free memory. CPU usage and Load averages were next to nothing.
If you can, monitor any signals being sent to the server.
I've had a tcpdump session running and have been trying to analyse it. As stated above, FreeRadius exited at 06:38:39. Prior to this, between 06:38:36.427 and 06:38:37.419, there was a successful Accounting-Request (type Stop), followed by this request being successfully proxied twice - behaviour as expected. (Marked by red square in attached image). Then, at 06:38:39.264, there's a Accounting-Request (type Stop), followed by another Accounting-Request (type Start), at 06:38:39.294, and a third Accounting-Request (type Start), at 06:38:41.023. The ip addresses in the image attached are: .46 is the radius server .61 is where the radius requests are sent from. .106 is where the requests are proxied to. http://freeradius.1045715.n5.nabble.com/file/n3309272/radius.jpg The failover radius server received the first request at 06:38:54. Is this helpful in any way? -- View this message in context: http://freeradius.1045715.n5.nabble.com/FreeRADIUS-exiting-with-Signal-11-on... Sent from the FreeRadius - User mailing list archive at Nabble.com.
Danial wrote:
If you can, monitor any signals being sent to the server.
I've had a tcpdump session running
Uh... packets are not signals. Use "ktrace" to monitor signals sent to a process. See http://www.gsp.com/cgi-bin/man.cgi?section=1&topic=ktrace Alan DeKok.
Alan DeKok-2 wrote:
Uh... packets are not signals. Use "ktrace" to monitor signals sent to a process.
Well, I've now monitored an "Exiting normally" event using ktrace: /var/log/radius.log: Sat Dec 18 02:43:43 2010 : Info: Exiting normally. /var/log/messages: Dec 18 02:43:43 radius kernel: pid 655 (radiusd), uid 133: exited on signal 11 $ kdump | tail -20 655 radiusd CALL fstat(0xa,0xbf8fb280) 655 radiusd STRU struct stat {dev=83, ino=2378852, mode=-rw------- , nlink=1, uid=133, gid=133, rdev=0, atime=1292640223, stime=1292640223, ctime=1292640223, birthtime=1292640223, size=0, blksize=16384, blocks=0, flags=0x0 } 655 radiusd RET fstat 0 655 radiusd CALL lseek(0xa,0,SEEK_SET,0x2) 655 radiusd RET lseek 0 655 radiusd CALL write(0xa,0x28752000,0x54) 655 radiusd GIO fd 10 wrote 84 bytes "Sat Dec 18 02:43:43 2010 Timestamp = 1292640223 Request-Authenticator = Verified " 655 radiusd RET write 84/0x54 655 radiusd CALL close(0xa) 655 radiusd RET close 0 655 radiusd PSIG SIGSEGV SIG_DFL 655 radiusd RET _umtx_op -1 errno 4 Interrupted system call 655 radiusd RET _umtx_op -1 errno 4 Interrupted system call 655 radiusd RET _umtx_op -1 errno 4 Interrupted system call 655 radiusd RET _umtx_op -1 errno 4 Interrupted system call It seems I can semi-reliably replicate this scenario. When I force this radius server to become the primary, it exits after a few hours. / Danial -- View this message in context: http://freeradius.1045715.n5.nabble.com/FreeRADIUS-exiting-with-Signal-11-on... Sent from the FreeRadius - User mailing list archive at Nabble.com.
Danial wrote:
Well, I've now monitored an "Exiting normally" event using ktrace:
That helps.
655 radiusd RET write 84/0x54 655 radiusd CALL close(0xa) 655 radiusd RET close 0 655 radiusd PSIG SIGSEGV SIG_DFL
Ugh.
It seems I can semi-reliably replicate this scenario. When I force this radius server to become the primary, it exits after a few hours.
If you can, see doc/bugs. Re-build with debugging symbols, run it under "gdb", and we should be able to get a backtrace containing the exact line which has a problem. Alan DeKok.
Alan DeKok-2 wrote:
If you can, see doc/bugs. Re-build with debugging symbols, run it under "gdb", and we should be able to get a backtrace containing the exact line which has a problem.
Ok. I've read the doc/bugs file. Removed the installed freeradius. Recompiled with debugging symbols (cd /usr/ports/net/freeradius2 && make -DWITH_DEVELOPER && make install) Now, running under gdb is not working very well. It exits right away: # gdb radiusd GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd"... (gdb) set logging file gdb-radiusd.log (gdb) set logging on Copying output to gdb-radiusd.log. (gdb) run Starting program: /usr/local/sbin/radiusd [New LWP 100077] [New Thread 28401140 (LWP 100077)] Program exited normally. What can I do about this? / Danial -- View this message in context: http://freeradius.1045715.n5.nabble.com/FreeRADIUS-exiting-with-Signal-11-on... Sent from the FreeRadius - User mailing list archive at Nabble.com.
You need to run radiusd in single threaded mode. The server forks a couple processes to run the server as a daemon and then exits normally. If you looked at the process table (ps -ef), you should see a copy of freeradius running on your system. In gdb, set the arguments for radiusd using set args gdb> set args -f gdb> run Tim
-----Original Message----- From: freeradius-users- bounces+tim.sylvester=networkradius.com@lists.freeradius.org [mailto:freeradius-users- bounces+tim.sylvester=networkradius.com@lists.freeradius.org] On Behalf Of Danial Sent: Sunday, December 19, 2010 4:27 PM To: freeradius-users@lists.freeradius.org Subject: Re: FreeRADIUS exiting with Signal 11 on FreeBSD
Alan DeKok-2 wrote:
If you can, see doc/bugs. Re-build with debugging symbols, run it under "gdb", and we should be able to get a backtrace containing the exact line which has a problem.
Ok. I've read the doc/bugs file. Removed the installed freeradius. Recompiled with debugging symbols (cd /usr/ports/net/freeradius2 && make -DWITH_DEVELOPER && make install)
Now, running under gdb is not working very well. It exits right away:
# gdb radiusd GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd"... (gdb) set logging file gdb-radiusd.log (gdb) set logging on Copying output to gdb-radiusd.log. (gdb) run Starting program: /usr/local/sbin/radiusd [New LWP 100077] [New Thread 28401140 (LWP 100077)]
Program exited normally.
What can I do about this?
/ Danial -- View this message in context: http://freeradius.1045715.n5.nabble.com/FreeRADIUS-exiting-with-Signal- 11-on-FreeBSD-tp3293576p3311522.html Sent from the FreeRadius - User mailing list archive at Nabble.com. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Tim Sylvester wrote:
You need to run radiusd in single threaded mode. The server forks a couple processes to run the server as a daemon and then exits normally. If you looked at the process table (ps -ef), you should see a copy of freeradius running on your system.
In gdb, set the arguments for radiusd using set args
gdb> set args -f gdb> run
I did what you suggested and have captured another exit: # gdb radiusd GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd"... (gdb) set logging file gdb-radiusd.log (gdb) set logging on Copying output to gdb-radiusd.log. (gdb) set args -f (gdb) run Starting program: /usr/local/sbin/radiusd -f [New LWP 100056] [New Thread 28401140 (LWP 100056)] [New Thread 28567ac0 (LWP 100051)] [New Thread 28567980 (LWP 100068)] [New Thread 28567840 (LWP 100069)] [New Thread 28567700 (LWP 100070)] [New Thread 285675c0 (LWP 100102)] Program received signal SIGABRT, Aborted. [Switching to Thread 28567ac0 (LWP 100051)] 0x2831866b in thr_kill () from /lib/libc.so.7 (gdb) What do you make of this? -- Danial -- View this message in context: http://freeradius.1045715.n5.nabble.com/FreeRADIUS-exiting-with-Signal-11-on... Sent from the FreeRadius - User mailing list archive at Nabble.com.
Danial wrote:
I did what you suggested and have captured another exit:
# gdb radiusd ... Program received signal SIGABRT, Aborted. [Switching to Thread 28567ac0 (LWP 100051)] 0x2831866b in thr_kill () from /lib/libc.so.7 (gdb)
What do you make of this?
You need to follow the instructions in doc/bugs like you did last time, and do: (gdb) thread apply all bt full The information you posted shows only that it crashed, and nothing more useful after that. Alan DeKok.
Here's the full output from gdb: # gdb radiusd GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd"... (gdb) set logging file gdb-radiusd.log (gdb) set logging on Copying output to gdb-radiusd.log. (gdb) set args -f (gdb) run Starting program: /usr/local/sbin/radiusd -f [New LWP 100056] [New Thread 28401140 (LWP 100056)] [New Thread 28567ac0 (LWP 100051)] [New Thread 28567980 (LWP 100068)] [New Thread 28567840 (LWP 100069)] [New Thread 28567700 (LWP 100070)] [New Thread 285675c0 (LWP 100102)] Program received signal SIGABRT, Aborted. [Switching to Thread 28567ac0 (LWP 100051)] 0x2831866b in thr_kill () from /lib/libc.so.7 (gdb) (gdb) (gdb) where #0 0x2831866b in thr_kill () from /lib/libc.so.7 #1 0x282c7896 in pthread_kill () from /lib/libthr.so.3 #2 0x282c53b3 in raise () from /lib/libthr.so.3 #3 0x283bac4a in abort () from /lib/libc.so.7 #4 0x0806c852 in rad_assert_fail (file=Could not find the frame base for "rad_assert_fail". ) at util.c:365 #5 0x0806af44 in request_dequeue (request=0x28542b7c, fun=0xbf9fef8c) at threads.c:412 #6 0x0806b18b in request_handler_thread (arg=0x28542b60) at threads.c:516 #7 0x282c175f in pthread_getprio () from /lib/libthr.so.3 #8 0x00000000 in ?? () (gdb) thread apply all bt full Thread 7 (Thread 285675c0 (LWP 100102)): #0 0x282cc227 in __error () from /lib/libthr.so.3 No symbol table info available. #1 0x282cbc19 in __error () from /lib/libthr.so.3 No symbol table info available. #2 0x285484fc in ?? () No symbol table info available. #3 0x0000000f in ?? () No symbol table info available. #4 0x00000000 in ?? () No symbol table info available. #5 0x00000000 in ?? () No symbol table info available. #6 0x00000000 in ?? () No symbol table info available. #7 0x282cdbf4 in ?? () from /lib/libthr.so.3 No symbol table info available. #8 0xbf5faf68 in ?? () No symbol table info available. #9 0x282c0fcc in sem_wait () from /lib/libthr.so.3 No symbol table info available. Previous frame identical to this frame (corrupt stack?) Thread 6 (Thread 28567700 (LWP 100070)): #0 0x282cc227 in __error () from /lib/libthr.so.3 No symbol table info available. #1 0x282cbc19 in __error () from /lib/libthr.so.3 No symbol table info available. #2 0x285484fc in ?? () No symbol table info available. #3 0x0000000f in ?? () No symbol table info available. #4 0x00000000 in ?? () No symbol table info available. #5 0x00000000 in ?? () No symbol table info available. #6 0x00000000 in ?? () No symbol table info available. #7 0x282cdbf4 in ?? () from /lib/libthr.so.3 No symbol table info available. #8 0xbf6fbf68 in ?? () No symbol table info available. #9 0x282c0fcc in sem_wait () from /lib/libthr.so.3 No symbol table info available. Previous frame identical to this frame (corrupt stack?) ---Type <return> to continue, or q <return> to quit--- Thread 5 (Thread 28567840 (LWP 100069)): #0 0x282cc227 in __error () from /lib/libthr.so.3 No symbol table info available. #1 0x282cbc19 in __error () from /lib/libthr.so.3 No symbol table info available. #2 0x285484fc in ?? () No symbol table info available. #3 0x0000000f in ?? () No symbol table info available. #4 0x00000000 in ?? () No symbol table info available. #5 0x00000000 in ?? () No symbol table info available. #6 0x00000000 in ?? () No symbol table info available. #7 0x282cdbf4 in ?? () from /lib/libthr.so.3 No symbol table info available. #8 0xbf7fcf68 in ?? () No symbol table info available. #9 0x282c0fcc in sem_wait () from /lib/libthr.so.3 No symbol table info available. Previous frame identical to this frame (corrupt stack?) Thread 4 (Thread 28567980 (LWP 100068)): #0 0x282cc227 in __error () from /lib/libthr.so.3 No symbol table info available. #1 0x282cbc19 in __error () from /lib/libthr.so.3 No symbol table info available. #2 0x285484fc in ?? () No symbol table info available. #3 0x0000000f in ?? () No symbol table info available. #4 0x00000000 in ?? () No symbol table info available. #5 0x00000000 in ?? () No symbol table info available. #6 0x00000000 in ?? () No symbol table info available. #7 0x282cdbf4 in ?? () from /lib/libthr.so.3 No symbol table info available. #8 0xbf8fdf68 in ?? () No symbol table info available. #9 0x282c0fcc in sem_wait () from /lib/libthr.so.3 No symbol table info available. Previous frame identical to this frame (corrupt stack?) Thread 3 (Thread 28567ac0 (LWP 100051)): ---Type <return> to continue, or q <return> to quit--- #0 0x2831866b in thr_kill () from /lib/libc.so.7 No symbol table info available. #1 0x282c7896 in pthread_kill () from /lib/libthr.so.3 No symbol table info available. #2 0x282c53b3 in raise () from /lib/libthr.so.3 No symbol table info available. #3 0x283bac4a in abort () from /lib/libc.so.7 No symbol table info available. #4 0x0806c852 in rad_assert_fail (file=Could not find the frame base for "rad_assert_fail". ) at util.c:365 No locals. #5 0x0806af44 in request_dequeue (request=0x28542b7c, fun=0xbf9fef8c) at threads.c:412 blocked = 134809920 i = RAD_LISTEN_ACCT start = RAD_LISTEN_ACCT entry = (request_queue_t *) 0x0 last_complained = 0 #6 0x0806b18b in request_handler_thread (arg=0x28542b60) at threads.c:516 fun = 0x804e250 <rad_accounting> self = (THREAD_HANDLE *) 0x28542b60 #7 0x282c175f in pthread_getprio () from /lib/libthr.so.3 No symbol table info available. #8 0x00000000 in ?? () No symbol table info available. Thread 2 (Thread 28401140 (LWP 100056)): #0 0x283abf9f in unlink () from /lib/libc.so.7 No symbol table info available. #1 0x0805daf9 in command_socket_free (this=0x28569100) at command.c:1839 sock = (fr_command_socket_t *) 0x285b4800 #2 0x08060053 in listen_free (head=0x8090b7c) at listen.c:2257 next = (rad_listen_t *) 0x285690a0 this = (rad_listen_t *) 0x28569100 #3 0x08061e7c in free_mainconfig () at mainconfig.c:992 cc = (cached_config_t *) 0x0 next = (cached_config_t *) 0x0 #4 0x0806859a in main (argc=2, argv=0xbfbfec74) at radiusd.c:443 rcode = 0 argval = -1 spawn_flag = 1 dont_fork = 1 flag = 0 act = {__sigaction_u = {__sa_handler = 0x80687f0 <sig_fatal>, __sa_sigaction = 0x80687f0 <sig_fatal>}, sa_flags = 0, sa_mask = {__bits = {0, 0, 0, 0}}} #0 0x2831866b in thr_kill () from /lib/libc.so.7 -- Danial -- View this message in context: http://freeradius.1045715.n5.nabble.com/FreeRADIUS-exiting-with-Signal-11-on... Sent from the FreeRadius - User mailing list archive at Nabble.com.
Danial wrote:
Here's the full output from gdb: ..l. #4 0x0806c852 in rad_assert_fail (file=Could not find the frame base for "rad_assert_fail". ) at util.c:365 #5 0x0806af44 in request_dequeue (request=0x28542b7c, fun=0xbf9fef8c) at threads.c:412
Ugh. Something is free'ing the request when it's still queued. This *only* can happen when the request is in the queue for more than 30s. So... why is your server so slow? Fix that, and the problem won't be fixed, but it *will* go away. I can take a look at the code, but this is a very odd edge case, and hard to track down/fix. Alan DeKok.
Hi Alan, Daniel and all, On 21/12/2010 19:46, Alan DeKok wrote:
Danial wrote:
Here's the full output from gdb: ..l. #4 0x0806c852 in rad_assert_fail (file=Could not find the frame base for "rad_assert_fail". ) at util.c:365 #5 0x0806af44 in request_dequeue (request=0x28542b7c, fun=0xbf9fef8c) at threads.c:412 Ugh. Something is free'ing the request when it's still queued. This *only* can happen when the request is in the queue for more than 30s.
So... why is your server so slow? Fix that, and the problem won't be fixed, but it *will* go away.
I have been running a network monitoring program on my server, that tracks cpu load, free memory. None of these have moved out of spec. Load average on the machine hit a high of 0.012. Since this problem effects a number of different FreeBSD boxes, is there something that could be related to the kernel, scheduling, locking etc. I'm guessing here so could be totally wrong. Are the debugs collected by Daniel meaningful to a FreeBSD developer or are they totally FreeRADIUS specific (yes I'm out of my league here)? Daniel, one of your emails suggests that you have two servers. One that seems OK and the other that gives the Signal 11 problem. Do they both run the same version of FreeBSD? Are there differences in the scheduling or SMP options in the kernel? What other differences are there between these two machines? Hugh
I can take a look at the code, but this is a very odd edge case, and hard to track down/fix.
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
-- Hugh Blandford Island Internet ph 1300 130 428 mb 0412 016 875
Hugh Blandford wrote:
I have been running a network monitoring program on my server, that tracks cpu load, free memory. None of these have moved out of spec.
Load average on the machine hit a high of 0.012.
This is similar to what I've monitored.
Daniel, one of your emails suggests that you have two servers. One that seems OK and the other that gives the Signal 11 problem. Do they both run the same version of FreeBSD? Are there differences in the scheduling or SMP options in the kernel? What other differences are there between these two machines?
I do have two servers, but they are quite different. This problematic one is running FreeBSD 8.1 on a VMWare ESX 4 system. I've checked with our vmware admin and there's little load on the whole vmware system. The second, which is running just fine, is freeradius-1.1.6 on OpenBSD 4.1. It's also on a VMWare platform, although a different one. One more thing I'd like to add is that I don't believe I experienced these exiting problems when I was running an older version of FreeBSD. I think it started when I upgraded from 6.x to 7.x - and still continues after reaching 8.1. / Danial -- View this message in context: http://freeradius.1045715.n5.nabble.com/FreeRADIUS-exiting-with-Signal-11-on... Sent from the FreeRadius - User mailing list archive at Nabble.com.
Hi all , We same problem : (radiusd), uid 133: exited on signal 11 Alan say that, this probeme occured, when request still in queue for more than 30s. Do you know if is possible to increase this time or drop requests when exceed 30sec ? Regards, Ben -- View this message in context: http://freeradius.1045715.n5.nabble.com/FreeRADIUS-exiting-with-Signal-11-on... Sent from the FreeRadius - User mailing list archive at Nabble.com.
ben_iss wrote:
We same problem : (radiusd), uid 133: exited on signal 11
On which version?
Alan say that, this probeme occured, when request still in queue for more than 30s.
Where do I say that?
Do you know if is possible to increase this time or drop requests when exceed 30sec ?
Perhaps you could fix your system so that it can handle the load being sent to it. Alan DeKok.
I use FreeBSD 8.0 with Freeradius 2.1.12 version. sorry if i don't understand your sentence : Ugh. Something is free'ing the request when it's still queued. This *only* can happen when the request is in the queue for more than 30s. This problem could be directly causes by network slow moment ? -- View this message in context: http://freeradius.1045715.n5.nabble.com/FreeRADIUS-exiting-with-Signal-11-on... Sent from the FreeRadius - User mailing list archive at Nabble.com.
Thx for yout return, Therefore i forgot to say that we use freeradius only in proxy radius mode. -- View this message in context: http://freeradius.1045715.n5.nabble.com/FreeRADIUS-exiting-with-Signal-11-on... Sent from the FreeRadius - User mailing list archive at Nabble.com.
participants (6)
-
Alan DeKok -
ben_iss -
Danial -
Hugh Blandford -
Scott Lambert -
Tim Sylvester