--On 18. November 2015 um 09:13:46 -0500 Alan DeKok <aland@deployingradius.com> wrote:
we've been running FR 3.0.10 in production for about a month now. We have four production servers on RHEL 6. Every few days one of the radiusd processes quits without an error message. We have now enabled the panic action and installed the debuginfo RPMs. Last night we got the first usable gbd.log. Here's the relevant part:
Thanks.
# 22 0x000000396dc15dbc in fr_pair_value_strsteal (vp=0x7f8b48732dc0, # src=0x7f8b4874f5e0 # "leap:session-key=\200hl\240:l\236\255t\256\247\366Z\315%f\233\220q~\3 # 43-\207VΟ\334\366N%i\372p\274Q\242\243\232ީ/\257\362\315\376f\344\06 # 7>0\356\262\v\233\343\240\344pH\213\177") at # /usr/src/debug/freeradius-server-3.0.10/src/lib/pair.c:2125 q = <value optimized out>
LEAP? Don't use that. It's insecure.
Actually we don't use LEAP and were surprised by that message. The session in question was using EAP-TTLS.
As far as we have been able to discern, previous crashes didn't all occur in post-proxy.
OK. So it's something else.
The same user authenticated successfully before, so it also doesn't seem to related to that specific user. I will post future backtraces if they offer new evidence. Please let me know if additional info would be helpful.
Any other back traces would help.
This might be hard to track down without local tests...
I'm willing to test anything you suggest. Meanwhile here's a new back trace: Thread 1 (Thread 0x7f57f27b87e0 (LWP 28274)): #0 0x0000003a1aaac70d in waitpid () from /lib64/libc.so.6 No symbol table info available. #1 0x0000003a1aa3e609 in do_system () from /lib64/libc.so.6 No symbol table info available. #2 0x0000003a1aa3e940 in system () from /lib64/libc.so.6 No symbol table info available. #3 0x000000396dc0c15c in fr_fault (sig=<value optimized out>) at /usr/src/debug/freeradius-server-3.0.10/src/lib/debug.c:725 disable = true cmd = "gdb -silent -x /etc/raddb/panic.gdb /usr/sbin/radiusd 28274 2>&1 | tee /var/log/radius/gdb-radiusd-28274.log", '\000' <repeats 423 times> out = 0x7fff54dafcd8 ".log" left = 428 ret = <value optimized out> p = 0x396de3d962 ".log" q = 0x0 code = <value optimized out> #4 <signal handler called> No symbol table info available. #5 0x0000003a20202d19 in ?? () from /usr/lib64/libtalloc.so.2 No symbol table info available. #6 0x0000003a2020244f in ?? () from /usr/lib64/libtalloc.so.2 No symbol table info available. #7 0x0000003a20202323 in ?? () from /usr/lib64/libtalloc.so.2 No symbol table info available. #8 0x0000003a20202323 in ?? () from /usr/lib64/libtalloc.so.2 No symbol table info available. #9 0x0000000000431257 in request_done (request=0x2bd3de0, action=2) at /usr/src/debug/freeradius-server-3.0.10/src/main/process.c:806 now = {tv_sec = 40078192, tv_usec = 45947520} when = {tv_sec = 140734617027872, tv_usec = 40077872} #10 0x0000000000432c31 in request_receive (ctx=0x2bd1a80, listener=0x2638a30, packet=0x2bd1ae0, client=0x263e340, fun=0x40efb0 <rad_authenticate>) at /usr/src/debug/freeradius-server-3.0.10/src/main/process.c:1656 child_state = <value optimized out> count = <value optimized out> packet_p = <value optimized out> request = <value optimized out> now = {tv_sec = 1447842529, tv_usec = 339180} sock = 0x2638b70 #11 0x0000000000419751 in auth_socket_recv (listener=0x2638a30) at /usr/src/debug/freeradius-server-3.0.10/src/main/listen.c:1568 rcode = <value optimized out> code = 1 src_port = <value optimized out> packet = <value optimized out> fun = 0x40efb0 <rad_authenticate> client = 0x263e340 src_ipaddr = {af = 2, ipaddr = {ip4addr = {s_addr = 36312236}, ip6addr = {__in6_u = {__u6_addr8 = "\254\024*\002", '\000' <repeats 11 times>, __u6_addr16 = {5292, 554, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {36312236, 0, 0, 0}}}}, prefix = 32 ' ', scope = 0} ctx = 0x2bd1a80 #12 0x0000000000430fa0 in event_socket_handler (xel=<value optimized out>, fd=<value optimized out>, ctx=<value optimized out>) at /usr/src/debug/freeradius-server-3.0.10/src/main/process.c:4556 listener = <value optimized out> #13 0x000000396dc28cfb in fr_event_loop (el=0x2535930) at /usr/src/debug/freeradius-server-3.0.10/src/lib/event.c:641 ef = <value optimized out> i = <value optimized out> rcode = 1 when = {tv_sec = 1447842529, tv_usec = 345425} wake = <value optimized out> maxfd = 31 read_fds = {fds_bits = {4194304, 0 <repeats 15 times>}} master_fds = {fds_bits = {4294049680, 0 <repeats 15 times>}} #14 0x0000000000427204 in main (argc=<value optimized out>, argv=<value optimized out>) at /usr/src/debug/freeradius-server-3.0.10/src/main/radiusd.c:579 rcode = 0 status = <value optimized out> argval = <value optimized out> spawn_flag = true display_version = false flag = 0 from_child = {4, 5} state = 0x66bb40 autofree = 0x2264060 A debugging session is active. -- .:.Sebastian Hagedorn - Weyertal 121 (Gebäude 133), Zimmer 2.02.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universität zu Köln / Cologne University - ✆ +49-221-470-89578.:.