Hi, today I have just had the initial 'live' test of a new FreeRADIUS server. The setup had passed all testing performed in the Lab before re-location. The server is a dual-core Xeon with 4G RAM running Debian Stable (Lenny): Linux tas1 2.6.26-1-amd64 #1 SMP Sat Jan 10 17:57:00 UTC 2009 x86_64 GNU/Linux FreeRADIUS was built with the command: dpkg-buildpackage -rfakeroot -us -uc The resulting .deb files were installed. The source is from GIT 'stable' from the build date. This 'stable' branch included patches after the tarball release. I will be able to supply the GIT information later when my build system is up again. freeradius -v : freeradius: FreeRADIUS Version 2.1.4, for host x86_64-pc-linux-gnu, built on Feb 20 2009 at 10:39:25 -- The system seems to run fine for some time. Then I get a Seg-Fault. I was quickly able to isolate one clear cause for a seg-fault, which was my 'copy-acct-to-home-server' setup. Disabling the virtual server made the system run, apparently, fine. Setting up this virtual server independently by duplicating and suitably modifying the configuration and running with the '-d' option I was able to watch this system run. It was happily reading from the log file and sending the data to the home-server. After a number of records, (I noted 50-300 on various runs), the freeradius process would segfault. This happend, as far as I could see, always after the home-server failed to respond within the timeout (response_window) and so failed. Being more sys-admin than developer I though I would pass this on. Any ideas welcome, I will look more at this tomorrow. I believe that I also had a seg-fault from the system when the 'copy-acct-to-home-server' virtual server was disabled, at least at one point I had no freeradius process running when I am sure the 'main' one was. The system is currently offline. --> detail below <-- I have been trying to get a core file from the system without success (very embarrassing): ulimit -c -> unlimited allow_core_dumps = yes cat /proc/sys/kernel/core_pattern -> core The freeradius user has write access to the location where I execute the program. Debug output: ----- Going to the next request rad_recv: Accounting-Response packet from host <home-server-IP> port 4367, id=234, length=20 Segmentation fault ----- Doing an strace of the process was not very insightful: ----- select(26, [19 21 22 23 24 25], NULL, NULL, {0, 28186}) = 0 (Timeout) select(26, [19 21 22 23 24 25], NULL, NULL, {0, 47114}) = 0 (Timeout) select(26, [19 21 22 23 24 25], NULL, NULL, {0, 45698}) = 0 (Timeout) select(26, [19 21 22 23 24 25], NULL, NULL, {0, 43934}) = 0 (Timeout) select(26, [19 21 22 23 24 25], NULL, NULL, {0, 935}) = 0 (Timeout) select(26, [19 21 22 23 24 25], NULL, NULL, {0, 41526}) = 0 (Timeout) select(26, [19 21 22 23 24 25], NULL, NULL, {0, 7897}) = 0 (Timeout) select(26, [19 21 22 23 24 25], NULL, NULL, {0, 33998}) = 0 (Timeout) select(26, [19 21 22 23 24 25], NULL, NULL, {0, 18882}) = 0 (Timeout) select(26, [19 21 22 23 24 25], NULL, NULL, {0, 5182}) = 0 (Timeout) stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2960, ...}) = 0 sendto(3, "<27>Mar 2 14:29:07 freeradius[12"..., 130, MSG_NOSIGNAL, NULL, 0) = 130 read(19, 0x7fff88cb1070, 16) = -1 EAGAIN (Resource temporarily unavailable) write(20, "\10"..., 1) = 1 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2960, ...}) = 0 sendto(3, "<27>Mar 2 14:29:07 freeradius[12"..., 128, MSG_NOSIGNAL, NULL, 0) = 128 sendmsg(25, {msg_name(16)={sa_family=AF_INET, sin_port=htons(4367), sin_addr=inet_addr("<home-server-IP>")}, msg_iov(1)=[{"\4\312\0H\346e\330\232\230\340X\204\240\v\203\251\211\314\356*(\6\0\0\0\2,\n00000"..., 72}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_IP, cmsg_type=, ...}, msg_flags=0}, 0) = 72 select(26, [19 21 22 23 24 25], NULL, NULL, {0, 20719}) = 1 (in [19], left {0, 20719}) read(19, "\10"..., 32) = 1 select(26, [19 21 22 23 24 25], NULL, NULL, {0, 20673}) = 0 (Timeout) select(26, [19 21 22 23 24 25], NULL, NULL, {0, 17346}) = 1 (in [25], left {0, 12000}) getsockname(25, {sa_family=AF_INET, sin_port=htons(1814), sin_addr=inet_addr("0.0.0.0")}, [549755813904]) = 0 recvfrom(25, "\5\312\0\24"..., 4, MSG_PEEK, {sa_family=AF_INET, sin_port=htons(4367), sin_addr=inet_addr("<home-server-IP>")}, [16]) = 4 getsockname(25, {sa_family=AF_INET, sin_port=htons(1814), sin_addr=inet_addr("0.0.0.0")}, [107374182416]) = 0 recvmsg(25, {msg_name(16)={sa_family=AF_INET, sin_port=htons(4367), sin_addr=inet_addr("<home-server-IP>")}, msg_iov(1)=[{"\5\312\0\24bTk\201\263\265\r\237\362\r\233\37-\344\356\367"..., 20}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_IP, cmsg_type=, ...}, msg_flags=0}, 0) = 20 --- SIGSEGV (Segmentation fault) @ 0 (0) --- ----- Doing a 'kill -TERM' to the process when _not_ processing data produced this: Waking up in 1.2 seconds. Waking up in 0.1 seconds. Exiting normally. *** glibc detected *** freeradius: corrupted double-linked list: 0x000000000083b1f0 *** ======= Backtrace: ========= /lib/libc.so.6[0x7fbe1b341948] /lib/libc.so.6[0x7fbe1b341c10] /lib/libc.so.6[0x7fbe1b343708] /lib/libc.so.6(cfree+0x76)[0x7fbe1b343a56] /usr/lib/freeradius/libfreeradius-radius-2.1.4.so(fr_hash_table_free+0x93)[0x7fbe1c69db63] /usr/lib/freeradius/libfreeradius-radius-2.1.4.so(dict_free+0x7a)[0x7fbe1c69c1da] freeradius(free_mainconfig+0x27)[0x4151b7] freeradius(main+0x5bf)[0x41a54f] /lib/libc.so.6(__libc_start_main+0xe6)[0x7fbe1b2ec1a6] freeradius[0x4085e9] ======= Memory map: ======== 00400000-0043b000 r-xp 00000000 fe:01 144730 /usr/sbin/freeradius 0063b000-0063f000 rw-p 0003b000 fe:01 144730 /usr/sbin/freeradius 0081b000-0094b000 rw-p 0081b000 00:00 0 [heap] 7fbe14000000-7fbe14021000 rw-p 7fbe14000000 00:00 0 7fbe14021000-7fbe18000000 ---p 7fbe14021000 00:00 0 7fbe19c71000-7fbe19c87000 r-xp 00000000 fe:00 62788 /lib/libgcc_s.so.1 7fbe19c87000-7fbe19e87000 ---p 00016000 fe:00 62788 /lib/libgcc_s.so.1 7fbe19e87000-7fbe19e88000 rw-p 00016000 fe:00 62788 /lib/libgcc_s.so.1 7fbe19e88000-7fbe19e89000 r-xp 00000000 fe:01 186032 /usr/lib/freeradius/rlm_always-2.1.4.so 7fbe19e89000-7fbe1a089000 ---p 00001000 fe:01 186032 /usr/lib/freeradius/rlm_always-2.1.4.so 7fbe1a089000-7fbe1a08a000 rw-p 00001000 fe:01 186032 /usr/lib/freeradius/rlm_always-2.1.4.so 7fbe1a08a000-7fbe1a08d000 r-xp 00000000 fe:01 186059 /usr/lib/freeradius/rlm_logintime-2.1.4.so 7fbe1a08d000-7fbe1a28c000 ---p 00003000 fe:01 186059 /usr/lib/freeradius/rlm_logintime-2.1.4.so 7fbe1a28c000-7fbe1a28d000 rw-p 00002000 fe:01 186059 /usr/lib/freeradius/rlm_logintime-2.1.4.so 7fbe1a28d000-7fbe1a28f000 r-xp 00000000 fe:01 186053 /usr/lib/freeradius/rlm_expiration-2.1.4.so 7fbe1a28f000-7fbe1a48e000 ---p 00002000 fe:01 186053 /usr/lib/freeradius/rlm_expiration-2.1.4.so 7fbe1a48e000-7fbe1a48f000 rw-p 00001000 fe:01 186053 /usr/lib/freeradius/rlm_expiration-2.1.4.so 7fbe1a48f000-7fbe1a491000 r-xp 00000000 fe:01 186054 /usr/lib/freeradius/rlm_expr-2.1.4.so 7fbe1a491000-7fbe1a691000 ---p 00002000 fe:01 186054 /usr/lib/freeradius/rlm_expr-2.1.4.so 7fbe1a691000-7fbe1a692000 rw-p 00002000 fe:01 186054 /usr/lib/freeradius/rlm_expr-2.1.4.so 7fbe1a692000-7fbe1a694000 r-xp 00000000 fe:01 186052 /usr/lib/freeradius/rlm_exec-2.1.4.so 7fbe1a694000-7fbe1a893000 ---p 00002000 fe:01 186052 /usr/lib/freeradius/rlm_exec-2.1.4.so 7fbe1a893000-7fbe1a894000 rw-p 00001000 fe:01 186052 /usr/lib/freeradius/rlm_exec-2.1.4.so 7fbe1a894000-7fbe1a89e000 r-xp 00000000 fe:00 62759 /lib/libnss_files-2.7.so 7fbe1a89e000-7fbe1aa9e000 ---p 0000a000 fe:00 62759 /lib/libnss_files-2.7.so 7fbe1aa9e000-7fbe1aaa0000 rw-p 0000a000 fe:00 62759 /lib/libnss_files-2.7.so 7fbe1aaa0000-7fbe1aaaa000 r-xp 00000000 fe:00 62748 /lib/libnss_nis-2.7.so 7fbe1aaaa000-7fbe1aca9000 ---p 0000a000 fe:00 62748 /lib/libnss_nis-2.7.so 7fbe1aca9000-7fbe1acab000 rw-p 00009000 fe:00 62748 /lib/libnss_nis-2.7.so 7fbe1acab000-7fbe1acb2000 r-xp 00000000 fe:00 62754 /lib/libnss_compat-2.7.so 7fbe1acb2000-7fbe1aeb1000 ---p 00007000 fe:00 62754 /lib/libnss_compat-2.7.so 7fbe1aeb1000-7fbe1aeb3000 rw-p 00006000 fe:00 62754 /lib/libnss_compat-2.7.so 7fbe1aeb3000-7fbe1aec9000 r-xp 00000000 fe:01 176723 /usr/lib/libz.so.1.2.3.3 7fbe1aec9000-7fbe1b0c9000 ---p 00016000 fe:01 176723 /usr/lib/libz.so.1.2.3.3 7fbe1b0c9000-7fbe1b0ca000 rw-p 00016000 fe:01 176723 /usr/lib/libz.so.1.2.3.3 7fbe1b0ca000-7fbe1b0cc000 r-xp 00000000 fe:00 62766 /lib/libdl-2.7.so 7fbe1b0cc000-7fbe1b2cc000 ---p 00002000 fe:00 62766 /lib/libdl-2.7.so 7fbe1b2cc000-7fbe1b2ce000 rw-p 00002000 fe:00 62766 /lib/libdl-2.7.so 7fbe1b2ce000-7fbe1b418000 r-xp 00000000 fe:00 62762 /lib/libc-2.7.so 7fbe1b418000-7fbe1b617000 ---p 0014a000 fe:00 62762 /lib/libc-2.7.so 7fbe1b617000-7fbe1b61a000 r--p 00149000 fe:00 62762 /lib/libc-2.7.so 7fbe1b61a000-7fbe1b61c000 rw-p 0014c000 fe:00 62762 /lib/libc-2.7.so 7fbe1b61c000-7fbe1b621000 rw-p 7fbe1b61c000 00:00 0 7fbe1b621000-7fbe1b792000 r-xp 00000000 fe:01 177147 /usr/lib/libcrypto.so.0.9.8 7fbe1b792000-7fbe1b992000 ---p 00171000 fe:01 177147 /usr/lib/libcrypto.so.0.9.8 7fbe1b992000-7fbe1b9b8000 rw-p 00171000 fe:01 177147 /usr/lib/libcrypto.so.0.9.8 7fbe1b9b8000-7fbe1b9bc000 rw-p 7fbe1b9b8000 00:00 0 7fbe1b9bc000-7fbe1ba06000 r-xp 00000000 fe:01 177148 /usr/lib/libssl.so.0.9.8 7fbe1ba06000-7fbe1bc06000 ---p 0004a000 fe:01 177148 /usr/lib/libssl.so.0.9.8 7fbe1bc06000-7fbe1bc0d000 rw-p 0004a000 fe:01 177148 /usr/lib/libssl.so.0.9.8 7fbe1bc0d000-7fbe1bc14000 r-xp 00000000 fe:01 178225 /usr/lib/libltdl.so.3.1.6 7fbe1bc14000-7fbe1be13000 ---p 00007000 fe:01 178225 /usr/lib/libltdl.so.3.1.6 7fbe1be13000-7fbe1be14000 rw-p 00006000 fe:01 178225 /usr/lib/libltdl.so.3.1.6 7fbe1be14000-7fbe1be1c000 r-xp 00000000 fe:00 62760 /lib/libcrypt-2.7.so 7fbe1be1c000-7fbe1c01c000 ---p 00008000 fe:00 62760 /lib/libcrypt-2.7.so 7fbe1c01c000-7fbe1c01e000 rw-p 00008000 fe:00 62760 /lib/libcrypt-2.7.so 7fbe1c01e000-7fbe1c04c000 rw-p 7fbe1c01e000 00:00 0 7fbe1c04c000-7fbe1c062000 r-xp 00000000 fe:00 62758 /lib/libpthread-2.7.so 7fbe1c062000-7fbe1c262000 ---p 00016000 fe:00 62758 /lib/libpthread-2.7.so 7fbe1c262000-7fbe1c264000 rw-p 00016000 fe:00 62758 /lib/libpthread-2.7.so 7fbe1c264000-7fbe1c268000 rw-p 7fbe1c264000 00:00 0 7fbe1c268000-7fbe1c278000 r-xp 00000000 fe:00 62756 /lib/libresolv-2.7.so 7fbe1c278000-7fbe1c478000 ---p 00010000 fe:00 62756 /lib/libresolv-2.7.so 7fbe1c478000-7fbe1c47a000 rw-p 00010000 fe:00 62756 /lib/libresolv-2.7.so 7fbe1c47a000-7fbe1c47c000 rw-p 7fbe1c47a000 00:00 0 7fbe1c47c000-7fbe1c491000 r-xp 00000000 fe:00 62752 /lib/libnsl-2.7.so 7fbe1c491000-7fbe1c690000 ---p 00015000 fe:00 62752 /lib/libnsl-2.7.so 7fbe1c690000-7fbe1c692000 rw-p 00014000 fe:00 62752 /lib/libnsl-2.7.so 7fbe1c692000-7fbe1c694000 rw-p 7fbe1c692000 00:00 0 7fbe1c694000-7fbe1c6b3000 r-xp 00000000 fe:01 186029 /usr/lib/freeradius/libfreeradius-radius-2.1.4.so 7fbe1c6b3000-7fbe1c8b3000 ---p 0001f000 fe:01 186029 /usr/lib/freeradius/libfreeradius-radius-2.1.4.so 7fbe1c8b3000-7fbe1c8b5000 rw-p 0001f000 fe:01 186029 /usr/lib/freeradius/libfreeradius-radius-2.1.4.so 7fbe1c8b5000-7fbe1c8b6000 rw-p 7fbe1c8b5000 00:00 0 7fbe1c8b6000-7fbe1c8d2000 r-xp 00000000 fe:00 62765 /lib/ld-2.7.so 7fbe1cac4000-7fbe1caca000 rw-p 7fbe1cac4000 00:00 0 7fbe1cace000-7fbe1cad1000 rw-p 7fbe1cace000 00:00 0 7fbe1cad1000-7fbe1cad3000 rw-p 0001b000 fe:00 62765 /lib/ld-2.7.so 7fff24abb000-7fff24ad2000 rw-p 7ffffffe8000 00:00 0 [stack] 7fff24bfe000-7fff24bff000 r-xp 7fff24bfe000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] Aborted