Seg-fault with proxy and fail-overs
Chris Moules
chris at gms.lu
Mon Mar 2 19:02:06 CET 2009
Hi,
today I have just had the initial 'live' test of a new FreeRADIUS server. The setup had passed all testing performed in the Lab
before re-location.
The server is a dual-core Xeon with 4G RAM running Debian Stable (Lenny):
Linux tas1 2.6.26-1-amd64 #1 SMP Sat Jan 10 17:57:00 UTC 2009 x86_64 GNU/Linux
FreeRADIUS was built with the command: dpkg-buildpackage -rfakeroot -us -uc
The resulting .deb files were installed.
The source is from GIT 'stable' from the build date. This 'stable' branch included patches after the tarball release. I will be
able to supply the GIT information later when my build system is up again.
freeradius -v :
freeradius: FreeRADIUS Version 2.1.4, for host x86_64-pc-linux-gnu, built on Feb 20 2009 at 10:39:25
--
The system seems to run fine for some time. Then I get a Seg-Fault. I was quickly able to isolate one clear cause for a
seg-fault, which was my 'copy-acct-to-home-server' setup. Disabling the virtual server made the system run, apparently, fine.
Setting up this virtual server independently by duplicating and suitably modifying the configuration and running with the '-d'
option I was able to watch this system run. It was happily reading from the log file and sending the data to the home-server.
After a number of records, (I noted 50-300 on various runs), the freeradius process would segfault. This happend, as far as I
could see, always after the home-server failed to respond within the timeout (response_window) and so failed.
Being more sys-admin than developer I though I would pass this on.
Any ideas welcome, I will look more at this tomorrow. I believe that I also had a seg-fault from the system when the
'copy-acct-to-home-server' virtual server was disabled, at least at one point I had no freeradius process running when I am sure
the 'main' one was. The system is currently offline.
--> detail below <--
I have been trying to get a core file from the system without success (very embarrassing):
ulimit -c -> unlimited
allow_core_dumps = yes
cat /proc/sys/kernel/core_pattern -> core
The freeradius user has write access to the location where I execute the program.
Debug output:
-----
Going to the next request
rad_recv: Accounting-Response packet from host <home-server-IP> port 4367, id=234, length=20
Segmentation fault
-----
Doing an strace of the process was not very insightful:
-----
select(26, [19 21 22 23 24 25], NULL, NULL, {0, 28186}) = 0 (Timeout)
select(26, [19 21 22 23 24 25], NULL, NULL, {0, 47114}) = 0 (Timeout)
select(26, [19 21 22 23 24 25], NULL, NULL, {0, 45698}) = 0 (Timeout)
select(26, [19 21 22 23 24 25], NULL, NULL, {0, 43934}) = 0 (Timeout)
select(26, [19 21 22 23 24 25], NULL, NULL, {0, 935}) = 0 (Timeout)
select(26, [19 21 22 23 24 25], NULL, NULL, {0, 41526}) = 0 (Timeout)
select(26, [19 21 22 23 24 25], NULL, NULL, {0, 7897}) = 0 (Timeout)
select(26, [19 21 22 23 24 25], NULL, NULL, {0, 33998}) = 0 (Timeout)
select(26, [19 21 22 23 24 25], NULL, NULL, {0, 18882}) = 0 (Timeout)
select(26, [19 21 22 23 24 25], NULL, NULL, {0, 5182}) = 0 (Timeout)
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2960, ...}) = 0
sendto(3, "<27>Mar 2 14:29:07 freeradius[12"..., 130, MSG_NOSIGNAL, NULL, 0) = 130
read(19, 0x7fff88cb1070, 16) = -1 EAGAIN (Resource temporarily unavailable)
write(20, "\10"..., 1) = 1
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2960, ...}) = 0
sendto(3, "<27>Mar 2 14:29:07 freeradius[12"..., 128, MSG_NOSIGNAL, NULL, 0) = 128
sendmsg(25, {msg_name(16)={sa_family=AF_INET, sin_port=htons(4367), sin_addr=inet_addr("<home-server-IP>")},
msg_iov(1)=[{"\4\312\0H\346e\330\232\230\340X\204\240\v\203\251\211\314\356*(\6\0\0\0\2,\n00000"..., 72}], msg_controllen=32,
{cmsg_len=28, cmsg_level=SOL_IP, cmsg_type=, ...}, msg_flags=0}, 0) = 72
select(26, [19 21 22 23 24 25], NULL, NULL, {0, 20719}) = 1 (in [19], left {0, 20719})
read(19, "\10"..., 32) = 1
select(26, [19 21 22 23 24 25], NULL, NULL, {0, 20673}) = 0 (Timeout)
select(26, [19 21 22 23 24 25], NULL, NULL, {0, 17346}) = 1 (in [25], left {0, 12000})
getsockname(25, {sa_family=AF_INET, sin_port=htons(1814), sin_addr=inet_addr("0.0.0.0")}, [549755813904]) = 0
recvfrom(25, "\5\312\0\24"..., 4, MSG_PEEK, {sa_family=AF_INET, sin_port=htons(4367), sin_addr=inet_addr("<home-server-IP>")},
[16]) = 4
getsockname(25, {sa_family=AF_INET, sin_port=htons(1814), sin_addr=inet_addr("0.0.0.0")}, [107374182416]) = 0
recvmsg(25, {msg_name(16)={sa_family=AF_INET, sin_port=htons(4367), sin_addr=inet_addr("<home-server-IP>")},
msg_iov(1)=[{"\5\312\0\24bTk\201\263\265\r\237\362\r\233\37-\344\356\367"..., 20}], msg_controllen=32, {cmsg_len=28,
cmsg_level=SOL_IP, cmsg_type=, ...}, msg_flags=0}, 0) = 20
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
-----
Doing a 'kill -TERM' to the process when _not_ processing data produced this:
Waking up in 1.2 seconds.
Waking up in 0.1 seconds.
Exiting normally.
*** glibc detected *** freeradius: corrupted double-linked list: 0x000000000083b1f0 ***
======= Backtrace: =========
/lib/libc.so.6[0x7fbe1b341948]
/lib/libc.so.6[0x7fbe1b341c10]
/lib/libc.so.6[0x7fbe1b343708]
/lib/libc.so.6(cfree+0x76)[0x7fbe1b343a56]
/usr/lib/freeradius/libfreeradius-radius-2.1.4.so(fr_hash_table_free+0x93)[0x7fbe1c69db63]
/usr/lib/freeradius/libfreeradius-radius-2.1.4.so(dict_free+0x7a)[0x7fbe1c69c1da]
freeradius(free_mainconfig+0x27)[0x4151b7]
freeradius(main+0x5bf)[0x41a54f]
/lib/libc.so.6(__libc_start_main+0xe6)[0x7fbe1b2ec1a6]
freeradius[0x4085e9]
======= Memory map: ========
00400000-0043b000 r-xp 00000000 fe:01 144730 /usr/sbin/freeradius
0063b000-0063f000 rw-p 0003b000 fe:01 144730 /usr/sbin/freeradius
0081b000-0094b000 rw-p 0081b000 00:00 0 [heap]
7fbe14000000-7fbe14021000 rw-p 7fbe14000000 00:00 0
7fbe14021000-7fbe18000000 ---p 7fbe14021000 00:00 0
7fbe19c71000-7fbe19c87000 r-xp 00000000 fe:00 62788 /lib/libgcc_s.so.1
7fbe19c87000-7fbe19e87000 ---p 00016000 fe:00 62788 /lib/libgcc_s.so.1
7fbe19e87000-7fbe19e88000 rw-p 00016000 fe:00 62788 /lib/libgcc_s.so.1
7fbe19e88000-7fbe19e89000 r-xp 00000000 fe:01 186032 /usr/lib/freeradius/rlm_always-2.1.4.so
7fbe19e89000-7fbe1a089000 ---p 00001000 fe:01 186032 /usr/lib/freeradius/rlm_always-2.1.4.so
7fbe1a089000-7fbe1a08a000 rw-p 00001000 fe:01 186032 /usr/lib/freeradius/rlm_always-2.1.4.so
7fbe1a08a000-7fbe1a08d000 r-xp 00000000 fe:01 186059 /usr/lib/freeradius/rlm_logintime-2.1.4.so
7fbe1a08d000-7fbe1a28c000 ---p 00003000 fe:01 186059 /usr/lib/freeradius/rlm_logintime-2.1.4.so
7fbe1a28c000-7fbe1a28d000 rw-p 00002000 fe:01 186059 /usr/lib/freeradius/rlm_logintime-2.1.4.so
7fbe1a28d000-7fbe1a28f000 r-xp 00000000 fe:01 186053 /usr/lib/freeradius/rlm_expiration-2.1.4.so
7fbe1a28f000-7fbe1a48e000 ---p 00002000 fe:01 186053 /usr/lib/freeradius/rlm_expiration-2.1.4.so
7fbe1a48e000-7fbe1a48f000 rw-p 00001000 fe:01 186053 /usr/lib/freeradius/rlm_expiration-2.1.4.so
7fbe1a48f000-7fbe1a491000 r-xp 00000000 fe:01 186054 /usr/lib/freeradius/rlm_expr-2.1.4.so
7fbe1a491000-7fbe1a691000 ---p 00002000 fe:01 186054 /usr/lib/freeradius/rlm_expr-2.1.4.so
7fbe1a691000-7fbe1a692000 rw-p 00002000 fe:01 186054 /usr/lib/freeradius/rlm_expr-2.1.4.so
7fbe1a692000-7fbe1a694000 r-xp 00000000 fe:01 186052 /usr/lib/freeradius/rlm_exec-2.1.4.so
7fbe1a694000-7fbe1a893000 ---p 00002000 fe:01 186052 /usr/lib/freeradius/rlm_exec-2.1.4.so
7fbe1a893000-7fbe1a894000 rw-p 00001000 fe:01 186052 /usr/lib/freeradius/rlm_exec-2.1.4.so
7fbe1a894000-7fbe1a89e000 r-xp 00000000 fe:00 62759 /lib/libnss_files-2.7.so
7fbe1a89e000-7fbe1aa9e000 ---p 0000a000 fe:00 62759 /lib/libnss_files-2.7.so
7fbe1aa9e000-7fbe1aaa0000 rw-p 0000a000 fe:00 62759 /lib/libnss_files-2.7.so
7fbe1aaa0000-7fbe1aaaa000 r-xp 00000000 fe:00 62748 /lib/libnss_nis-2.7.so
7fbe1aaaa000-7fbe1aca9000 ---p 0000a000 fe:00 62748 /lib/libnss_nis-2.7.so
7fbe1aca9000-7fbe1acab000 rw-p 00009000 fe:00 62748 /lib/libnss_nis-2.7.so
7fbe1acab000-7fbe1acb2000 r-xp 00000000 fe:00 62754 /lib/libnss_compat-2.7.so
7fbe1acb2000-7fbe1aeb1000 ---p 00007000 fe:00 62754 /lib/libnss_compat-2.7.so
7fbe1aeb1000-7fbe1aeb3000 rw-p 00006000 fe:00 62754 /lib/libnss_compat-2.7.so
7fbe1aeb3000-7fbe1aec9000 r-xp 00000000 fe:01 176723 /usr/lib/libz.so.1.2.3.3
7fbe1aec9000-7fbe1b0c9000 ---p 00016000 fe:01 176723 /usr/lib/libz.so.1.2.3.3
7fbe1b0c9000-7fbe1b0ca000 rw-p 00016000 fe:01 176723 /usr/lib/libz.so.1.2.3.3
7fbe1b0ca000-7fbe1b0cc000 r-xp 00000000 fe:00 62766 /lib/libdl-2.7.so
7fbe1b0cc000-7fbe1b2cc000 ---p 00002000 fe:00 62766 /lib/libdl-2.7.so
7fbe1b2cc000-7fbe1b2ce000 rw-p 00002000 fe:00 62766 /lib/libdl-2.7.so
7fbe1b2ce000-7fbe1b418000 r-xp 00000000 fe:00 62762 /lib/libc-2.7.so
7fbe1b418000-7fbe1b617000 ---p 0014a000 fe:00 62762 /lib/libc-2.7.so
7fbe1b617000-7fbe1b61a000 r--p 00149000 fe:00 62762 /lib/libc-2.7.so
7fbe1b61a000-7fbe1b61c000 rw-p 0014c000 fe:00 62762 /lib/libc-2.7.so
7fbe1b61c000-7fbe1b621000 rw-p 7fbe1b61c000 00:00 0
7fbe1b621000-7fbe1b792000 r-xp 00000000 fe:01 177147 /usr/lib/libcrypto.so.0.9.8
7fbe1b792000-7fbe1b992000 ---p 00171000 fe:01 177147 /usr/lib/libcrypto.so.0.9.8
7fbe1b992000-7fbe1b9b8000 rw-p 00171000 fe:01 177147 /usr/lib/libcrypto.so.0.9.8
7fbe1b9b8000-7fbe1b9bc000 rw-p 7fbe1b9b8000 00:00 0
7fbe1b9bc000-7fbe1ba06000 r-xp 00000000 fe:01 177148 /usr/lib/libssl.so.0.9.8
7fbe1ba06000-7fbe1bc06000 ---p 0004a000 fe:01 177148 /usr/lib/libssl.so.0.9.8
7fbe1bc06000-7fbe1bc0d000 rw-p 0004a000 fe:01 177148 /usr/lib/libssl.so.0.9.8
7fbe1bc0d000-7fbe1bc14000 r-xp 00000000 fe:01 178225 /usr/lib/libltdl.so.3.1.6
7fbe1bc14000-7fbe1be13000 ---p 00007000 fe:01 178225 /usr/lib/libltdl.so.3.1.6
7fbe1be13000-7fbe1be14000 rw-p 00006000 fe:01 178225 /usr/lib/libltdl.so.3.1.6
7fbe1be14000-7fbe1be1c000 r-xp 00000000 fe:00 62760 /lib/libcrypt-2.7.so
7fbe1be1c000-7fbe1c01c000 ---p 00008000 fe:00 62760 /lib/libcrypt-2.7.so
7fbe1c01c000-7fbe1c01e000 rw-p 00008000 fe:00 62760 /lib/libcrypt-2.7.so
7fbe1c01e000-7fbe1c04c000 rw-p 7fbe1c01e000 00:00 0
7fbe1c04c000-7fbe1c062000 r-xp 00000000 fe:00 62758 /lib/libpthread-2.7.so
7fbe1c062000-7fbe1c262000 ---p 00016000 fe:00 62758 /lib/libpthread-2.7.so
7fbe1c262000-7fbe1c264000 rw-p 00016000 fe:00 62758 /lib/libpthread-2.7.so
7fbe1c264000-7fbe1c268000 rw-p 7fbe1c264000 00:00 0
7fbe1c268000-7fbe1c278000 r-xp 00000000 fe:00 62756 /lib/libresolv-2.7.so
7fbe1c278000-7fbe1c478000 ---p 00010000 fe:00 62756 /lib/libresolv-2.7.so
7fbe1c478000-7fbe1c47a000 rw-p 00010000 fe:00 62756 /lib/libresolv-2.7.so
7fbe1c47a000-7fbe1c47c000 rw-p 7fbe1c47a000 00:00 0
7fbe1c47c000-7fbe1c491000 r-xp 00000000 fe:00 62752 /lib/libnsl-2.7.so
7fbe1c491000-7fbe1c690000 ---p 00015000 fe:00 62752 /lib/libnsl-2.7.so
7fbe1c690000-7fbe1c692000 rw-p 00014000 fe:00 62752 /lib/libnsl-2.7.so
7fbe1c692000-7fbe1c694000 rw-p 7fbe1c692000 00:00 0
7fbe1c694000-7fbe1c6b3000 r-xp 00000000 fe:01 186029 /usr/lib/freeradius/libfreeradius-radius-2.1.4.so
7fbe1c6b3000-7fbe1c8b3000 ---p 0001f000 fe:01 186029 /usr/lib/freeradius/libfreeradius-radius-2.1.4.so
7fbe1c8b3000-7fbe1c8b5000 rw-p 0001f000 fe:01 186029 /usr/lib/freeradius/libfreeradius-radius-2.1.4.so
7fbe1c8b5000-7fbe1c8b6000 rw-p 7fbe1c8b5000 00:00 0
7fbe1c8b6000-7fbe1c8d2000 r-xp 00000000 fe:00 62765 /lib/ld-2.7.so
7fbe1cac4000-7fbe1caca000 rw-p 7fbe1cac4000 00:00 0
7fbe1cace000-7fbe1cad1000 rw-p 7fbe1cace000 00:00 0
7fbe1cad1000-7fbe1cad3000 rw-p 0001b000 fe:00 62765 /lib/ld-2.7.so
7fff24abb000-7fff24ad2000 rw-p 7ffffffe8000 00:00 0 [stack]
7fff24bfe000-7fff24bff000 r-xp 7fff24bfe000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Aborted
More information about the Freeradius-Devel
mailing list