Issue with 3.2.6 - dies/stops serving but is still running/listening on port
Since upgrading to 3.2.6 we're seeing an issue where the radiusd program just "dies" and stops receiving/handling requests, yet the program itself is still running, has a PID, and netstat shows it's still listening on the ports. I can see the requests still reaching the machine via tcpdump but nothing from radiusd. We're using TLS to support EAP connections but no clients are using TLS on this particular server so it's all UDP, though TLS is still enabled and listening. OpenSSL version is 3.0.2 I've included some logs from what little I know about gdb and wondered if anyone can advise the likely culprit of the issue. The last entries in radius.log were normal authentications and then it just died and stopped receiving requests. I've also observed the same in debug mode, nothing is showing untoward in the output on screen before it dies. Thu Sep 26 06:49:13 2024 : Info: ... adding new socket auth+acct from client (redacted, 33981) -> (*, 2083, virtual-server=default) Thu Sep 26 06:49:13 2024 : Auth: (328802) Login OK: [redacted] (from client 209.206.60.101 port 0 cli redacted) Thu Sep 26 06:49:14 2024 : Auth: (328809) Login OK: [redacted] (from client 209.206.60.101 port 0 cli redacted) Thu Sep 26 06:49:14 2024 : Auth: (328814) Login OK: [redacted] (from client 209.206.60.101 port 0 cli redacted) Thu Sep 26 06:49:18 2024 : Auth: (328835) Login OK: [redacted] (from client 209.206.60.101 port 0 cli redacted) --- (gdb) bt #0 futex_wait (private=0, expected=2, futex_word=0x572a56ee9c88) at ../sysdeps/nptl/futex-internal.h:146 #1 __GI___lll_lock_wait (futex=futex@entry=0x572a56ee9c88, private=0) at ./nptl/lowlevellock.c:49 #2 0x00007fc3d5098002 in lll_mutex_lock_optimized (mutex=0x572a56ee9c88) at ./nptl/pthread_mutex_lock.c:48 #3 ___pthread_mutex_lock (mutex=mutex@entry=0x572a56ee9c88) at ./nptl/pthread_mutex_lock.c:93 #4 0x0000572a546bfc05 in tls_socket_recv (listener=0x572a56e6f710) at src/main/tls_listen.c:476 #5 dual_tls_recv (listener=0x572a56e6f710) at src/main/tls_listen.c:732 #6 0x00007fc3d59aaee4 in fr_event_loop (el=0x572a56ab72f0) at src/lib/event.c:713 #7 0x0000572a546af1d5 in radius_event_process () at src/main/process.c:6565 #8 0x0000572a546834bc in main (argc=<optimized out>, argv=<optimized out>) at src/main/radiusd.c:641 --- (gdb) info threads Id Target Id Frame * 1 Thread 0x7fc3d5850c00 (LWP 837582) "radiusd" futex_wait (private=0, expected=2, futex_word=0x572a56ee9c88) at ../sysdeps/nptl/futex-internal.h:146 2 Thread 0x7fc3d15f5640 (LWP 922309) "radiusd" __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x572a546f13e8 <thread_pool+168>) at ./nptl/futex-internal.c:57 3 Thread 0x7fc3cd5ed640 (LWP 922310) "radiusd" __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x572a546f13e8 <thread_pool+168>) at ./nptl/futex-internal.c:57 4 Thread 0x7fc3d1df6640 (LWP 922311) "radiusd" __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x572a546f13e8 <thread_pool+168>) at ./nptl/futex-internal.c:57 5 Thread 0x7fc39e7fc640 (LWP 922312) "radiusd" __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x572a546f13e8 <thread_pool+168>) at ./nptl/futex-internal.c:57 6 Thread 0x7fc3cf5f1640 (LWP 922313) "radiusd" __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x572a546f13e8 <thread_pool+168>) at ./nptl/futex-internal.c:57 7 Thread 0x7fc3cedf0640 (LWP 922314) "radiusd" __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x572a546f13e8 <thread_pool+168>) at ./nptl/futex-internal.c:57 8 Thread 0x7fc3957fa640 (LWP 922315) "radiusd" __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x572a546f13e8 <thread_pool+168>) at ./nptl/futex-internal.c:57 9 Thread 0x7fc387fff640 (LWP 922316) "radiusd" __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x572a546f13e8 <thread_pool+168>) at ./nptl/futex-internal.c:57 10 Thread 0x7fc3cfdf2640 (LWP 922317) "radiusd" __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x572a546f13e8 <thread_pool+168>) at ./nptl/futex-internal.c:57 --- Thread 10 (Thread 0x7fc3cfdf2640 (LWP 922317) "radiusd"): #0 __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x572a546f13e8 <thread_pool+168>) at ./nptl/futex-internal.c:57 sc_cancel_oldtype = 0 sc_ret = <optimized out> resultvar = <optimized out> __arg6 = <optimized out> __arg5 = <optimized out> __arg4 = <optimized out> __arg3 = <optimized out> __arg2 = <optimized out> __arg1 = <optimized out> _a6 = <optimized out> _a5 = <optimized out> _a4 = <optimized out> _a3 = <optimized out> _a2 = <optimized out> _a1 = <optimized out> #1 __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x572a546f13e8 <thread_pool+168>) at ./nptl/futex-internal.c:87 err = <optimized out> clockbit = 256 op = 393 err = <optimized out> clockbit = <optimized out> op = <optimized out> #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x572a546f13e8 <thread_pool+168>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139 No locals. #3 0x00007fc3d509cbdf in do_futex_wait (sem=sem@entry=0x572a546f13e8 <thread_pool+168>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111 err = <optimized out> #4 0x00007fc3d509cc78 in __new_sem_wait_slow64 (sem=sem@entry=0x572a546f13e8 <thread_pool+168>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183 _buffer = {__routine = 0x7fc3d509cbb0 <__sem_wait_cleanup>, __arg = 0x572a546f13e8 <thread_pool+168>, __canceltype = 1459787472, __prev = 0x0} err = <optimized out> d = 34359738368 #5 0x00007fc3d509ccf1 in __new_sem_wait (sem=sem@entry=0x572a546f13e8 <thread_pool+168>) at ./nptl/sem_wait.c:42 No locals. #6 0x0000572a546a5080 in request_handler_thread (arg=0x572a570098c0) at src/main/threads.c:741 self = 0x572a570098c0 re_wait = <optimized out> #7 0x00007fc3d5094ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 ret = <optimized out> pd = <optimized out> out = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140720694236800, -4619313545745495514, 140478982858304, 0, 140479069505488, 140720694237152, 4639685579648305702, 4639636563766184486}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> #8 0x00007fc3d5126850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 No locals. Thread 9 (Thread 0x7fc387fff640 (LWP 922316) "radiusd"): #0 __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x572a546f1--Type <RET> for more, q to quit, c to continue without paging--c 3e8 <thread_pool+168>) at ./nptl/futex-internal.c:57 sc_cancel_oldtype = 0 sc_ret = <optimized out> resultvar = <optimized out> __arg6 = <optimized out> __arg5 = <optimized out> __arg4 = <optimized out> __arg3 = <optimized out> __arg2 = <optimized out> __arg1 = <optimized out> _a6 = <optimized out> _a5 = <optimized out> _a4 = <optimized out> _a3 = <optimized out> _a2 = <optimized out> _a1 = <optimized out> #1 __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x572a546f13e8 <thread_pool+168>) at ./nptl/futex-internal.c:87 err = <optimized out> clockbit = 256 op = 393 err = <optimized out> clockbit = <optimized out> op = <optimized out> #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x572a546f13e8 <thread_pool+168>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139 No locals. #3 0x00007fc3d509cbdf in do_futex_wait (sem=sem@entry=0x572a546f13e8 <thread_pool+168>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111 err = <optimized out> #4 0x00007fc3d509cc78 in __new_sem_wait_slow64 (sem=sem@entry=0x572a546f13e8 <thread_pool+168>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183 _buffer = {__routine = 0x7fc3d509cbb0 <__sem_wait_cleanup>, __arg = 0x572a546f13e8 <thread_pool+168>, __canceltype = 1458253584, __prev = 0x0} err = <optimized out> d = 34359738368 #5 0x00007fc3d509ccf1 in __new_sem_wait (sem=sem@entry=0x572a546f13e8 <thread_pool+168>) at ./nptl/sem_wait.c:42 No locals. #6 0x0000572a546a5080 in request_handler_thread (arg=0x572a56e99c30) at src/main/threads.c:741 self = 0x572a56e99c30 re_wait = <optimized out> #7 0x00007fc3d5094ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 ret = <optimized out> pd = <optimized out> out = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140720694236800, -4619313545745495514, 140477777049152, 0, 140479069505488, 140720694237152, 4639808443093387814, 4639636563766184486}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> #8 0x00007fc3d5126850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 No locals. Thread 8 (Thread 0x7fc3957fa640 (LWP 922315) "radiusd"): #0 __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x572a546f13e8 <thread_pool+168>) at ./nptl/futex-internal.c:57 sc_cancel_oldtype = 0 sc_ret = <optimized out> resultvar = <optimized out> __arg6 = <optimized out> __arg5 = <optimized out> __arg4 = <optimized out> __arg3 = <optimized out> __arg2 = <optimized out> __arg1 = <optimized out> _a6 = <optimized out> _a5 = <optimized out> _a4 = <optimized out> _a3 = <optimized out> _a2 = <optimized out> _a1 = <optimized out> #1 __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x572a546f13e8 <thread_pool+168>) at ./nptl/futex-internal.c:87 err = <optimized out> clockbit = 256 op = 393 err = <optimized out> clockbit = <optimized out> op = <optimized out> #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x572a546f13e8 <thread_pool+168>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139 No locals. #3 0x00007fc3d509cbdf in do_futex_wait (sem=sem@entry=0x572a546f13e8 <thread_pool+168>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111 err = <optimized out> #4 0x00007fc3d509cc78 in __new_sem_wait_slow64 (sem=sem@entry=0x572a546f13e8 <thread_pool+168>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183 _buffer = {__routine = 0x7fc3d509cbb0 <__sem_wait_cleanup>, __arg = 0x572a546f13e8 <thread_pool+168>, __canceltype = 1458253568, __prev = 0x0} err = <optimized out> d = 34359738368 #5 0x00007fc3d509ccf1 in __new_sem_wait (sem=sem@entry=0x572a546f13e8 <thread_pool+168>) at ./nptl/sem_wait.c:42 No locals. #6 0x0000572a546a5080 in request_handler_thread (arg=0x572a56f07cd0) at src/main/threads.c:741 self = 0x572a56f07cd0 re_wait = <optimized out> #7 0x00007fc3d5094ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 ret = <optimized out> pd = <optimized out> out = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140720694236800, -4619313545745495514, 140478003521088, 0, 140479069505488, 140720694237152, 4639776559940536870, 4639636563766184486}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> #8 0x00007fc3d5126850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 No locals. Thread 7 (Thread 0x7fc3cedf0640 (LWP 922314) "radiusd"): #0 __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x572a546f13e8 <thread_pool+168>) at ./nptl/futex-internal.c:57 sc_cancel_oldtype = 0 sc_ret = <optimized out> resultvar = <optimized out> __arg6 = <optimized out> __arg5 = <optimized out> __arg4 = <optimized out> __arg3 = <optimized out> __arg2 = <optimized out> __arg1 = <optimized out> _a6 = <optimized out> _a5 = <optimized out> _a4 = <optimized out> _a3 = <optimized out> _a2 = <optimized out> _a1 = <optimized out> #1 __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x572a546f13e8 <thread_pool+168>) at ./nptl/futex-internal.c:87 err = <optimized out> clockbit = 256 op = 393 err = <optimized out> clockbit = <optimized out> op = <optimized out> #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x572a546f13e8 <thread_pool+168>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139 No locals. #3 0x00007fc3d509cbdf in do_futex_wait (sem=sem@entry=0x572a546f13e8 <thread_pool+168>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111 err = <optimized out> #4 0x00007fc3d509cc78 in __new_sem_wait_slow64 (sem=sem@entry=0x572a546f13e8 <thread_pool+168>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183 _buffer = {__routine = 0x7fc3d509cbb0 <__sem_wait_cleanup>, __arg = 0x572a546f13e8 <thread_pool+168>, __canceltype = 1458225792, __prev = 0x0} err = <optimized out> d = 34359738368 #5 0x00007fc3d509ccf1 in __new_sem_wait (sem=sem@entry=0x572a546f13e8 <thread_pool+168>) at ./nptl/sem_wait.c:42 No locals. #6 0x0000572a546a5080 in request_handler_thread (arg=0x572a56e8f190) at src/main/threads.c:741 self = 0x572a56e8f190 re_wait = <optimized out> #7 0x00007fc3d5094ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 ret = <optimized out> pd = <optimized out> out = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140720694236800, -4619313545745495514, 140478966072896, 0, 140479069505488, 140720694237152, 4639687762565433894, 4639636563766184486}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> #8 0x00007fc3d5126850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 No locals. Thread 6 (Thread 0x7fc3cf5f1640 (LWP 922313) "radiusd"): #0 __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x572a546f13e8 <thread_pool+168>) at ./nptl/futex-internal.c:57 sc_cancel_oldtype = 0 sc_ret = <optimized out> resultvar = <optimized out> __arg6 = <optimized out> __arg5 = <optimized out> __arg4 = <optimized out> __arg3 = <optimized out> __arg2 = <optimized out> __arg1 = <optimized out> _a6 = <optimized out> _a5 = <optimized out> _a4 = <optimized out> _a3 = <optimized out> _a2 = <optimized out> _a1 = <optimized out> #1 __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x572a546f13e8 <thread_pool+168>) at ./nptl/futex-internal.c:87 err = <optimized out> clockbit = 256 op = 393 err = <optimized out> clockbit = <optimized out> op = <optimized out> #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x572a546f13e8 <thread_pool+168>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139 No locals. #3 0x00007fc3d509cbdf in do_futex_wait (sem=sem@entry=0x572a546f13e8 <thread_pool+168>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111 err = <optimized out> #4 0x00007fc3d509cc78 in __new_sem_wait_slow64 (sem=sem@entry=0x572a546f13e8 <thread_pool+168>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183 _buffer = {__routine = 0x7fc3d509cbb0 <__sem_wait_cleanup>, __arg = 0x572a546f13e8 <thread_pool+168>, __canceltype = 1459750864, __prev = 0x0} err = <optimized out> d = 34359738368 #5 0x00007fc3d509ccf1 in __new_sem_wait (sem=sem@entry=0x572a546f13e8 <thread_pool+168>) at ./nptl/sem_wait.c:42 No locals. #6 0x0000572a546a5080 in request_handler_thread (arg=0x572a56e68d40) at src/main/threads.c:741 self = 0x572a56e68d40 re_wait = <optimized out> #7 0x00007fc3d5094ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 ret = <optimized out> pd = <optimized out> out = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140720694236800, -4619313545745495514, 140478974465600, 0, 140479069505488, 140720694237152, 4639684480673548838, 4639636563766184486}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> #8 0x00007fc3d5126850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 No locals. Thread 5 (Thread 0x7fc39e7fc640 (LWP 922312) "radiusd"): #0 __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x572a546f13e8 <thread_pool+168>) at ./nptl/futex-internal.c:57 sc_cancel_oldtype = 0 sc_ret = <optimized out> resultvar = <optimized out> __arg6 = <optimized out> __arg5 = <optimized out> __arg4 = <optimized out> __arg3 = <optimized out> __arg2 = <optimized out> __arg1 = <optimized out> _a6 = <optimized out> _a5 = <optimized out> _a4 = <optimized out> _a3 = <optimized out> _a2 = <optimized out> _a1 = <optimized out> #1 __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x572a546f13e8 <thread_pool+168>) at ./nptl/futex-internal.c:87 err = <optimized out> clockbit = 256 op = 393 err = <optimized out> clockbit = <optimized out> op = <optimized out> #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x572a546f13e8 <thread_pool+168>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139 No locals. #3 0x00007fc3d509cbdf in do_futex_wait (sem=sem@entry=0x572a546f13e8 <thread_pool+168>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111 err = <optimized out> #4 0x00007fc3d509cc78 in __new_sem_wait_slow64 (sem=sem@entry=0x572a546f13e8 <thread_pool+168>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183 _buffer = {__routine = 0x7fc3d509cbb0 <__sem_wait_cleanup>, __arg = 0x572a546f13e8 <thread_pool+168>, __canceltype = 1458253728, __prev = 0x0} err = <optimized out> d = 34359738368 #5 0x00007fc3d509ccf1 in __new_sem_wait (sem=sem@entry=0x572a546f13e8 <thread_pool+168>) at ./nptl/sem_wait.c:42 No locals. #6 0x0000572a546a5080 in request_handler_thread (arg=0x572a56eee150) at src/main/threads.c:741 self = 0x572a56eee150 re_wait = <optimized out> #7 0x00007fc3d5094ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 ret = <optimized out> pd = <optimized out> out = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140720694236800, -4619313545745495514, 140478154524224, 0, 140479069505488, 140720694237152, 4639791952029583910, 4639636563766184486}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> #8 0x00007fc3d5126850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 No locals. Thread 4 (Thread 0x7fc3d1df6640 (LWP 922311) "radiusd"): #0 __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x572a546f13e8 <thread_pool+168>) at ./nptl/futex-internal.c:57 sc_cancel_oldtype = 0 sc_ret = <optimized out> resultvar = <optimized out> __arg6 = <optimized out> __arg5 = <optimized out> __arg4 = <optimized out> __arg3 = <optimized out> __arg2 = <optimized out> __arg1 = <optimized out> _a6 = <optimized out> _a5 = <optimized out> _a4 = <optimized out> _a3 = <optimized out> _a2 = <optimized out> _a1 = <optimized out> #1 __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x572a546f13e8 <thread_pool+168>) at ./nptl/futex-internal.c:87 err = <optimized out> clockbit = 256 op = 393 err = <optimized out> clockbit = <optimized out> op = <optimized out> #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x572a546f13e8 <thread_pool+168>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139 No locals. #3 0x00007fc3d509cbdf in do_futex_wait (sem=sem@entry=0x572a546f13e8 <thread_pool+168>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111 err = <optimized out> #4 0x00007fc3d509cc78 in __new_sem_wait_slow64 (sem=sem@entry=0x572a546f13e8 <thread_pool+168>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183 _buffer = {__routine = 0x7fc3d509cbb0 <__sem_wait_cleanup>, __arg = 0x572a546f13e8 <thread_pool+168>, __canceltype = 1458225760, __prev = 0x0} err = <optimized out> d = 34359738368 #5 0x00007fc3d509ccf1 in __new_sem_wait (sem=sem@entry=0x572a546f13e8 <thread_pool+168>) at ./nptl/sem_wait.c:42 No locals. #6 0x0000572a546a5080 in request_handler_thread (arg=0x572a56eab860) at src/main/threads.c:741 self = 0x572a56eab860 re_wait = <optimized out> #7 0x00007fc3d5094ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 ret = <optimized out> pd = <optimized out> out = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140720694236800, -4619313545745495514, 140479016429120, 0, 140479069505488, 140720694237152, 4639628402896177702, 4639636563766184486}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> #8 0x00007fc3d5126850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 No locals. Thread 3 (Thread 0x7fc3cd5ed640 (LWP 922310) "radiusd"): #0 __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x572a546f13e8 <thread_pool+168>) at ./nptl/futex-internal.c:57 sc_cancel_oldtype = 0 sc_ret = <optimized out> resultvar = <optimized out> __arg6 = <optimized out> __arg5 = <optimized out> __arg4 = <optimized out> __arg3 = <optimized out> __arg2 = <optimized out> __arg1 = <optimized out> _a6 = <optimized out> _a5 = <optimized out> _a4 = <optimized out> _a3 = <optimized out> _a2 = <optimized out> _a1 = <optimized out> #1 __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x572a546f13e8 <thread_pool+168>) at ./nptl/futex-internal.c:87 err = <optimized out> clockbit = 256 op = 393 err = <optimized out> clockbit = <optimized out> op = <optimized out> #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x572a546f13e8 <thread_pool+168>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139 No locals. #3 0x00007fc3d509cbdf in do_futex_wait (sem=sem@entry=0x572a546f13e8 <thread_pool+168>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111 err = <optimized out> #4 0x00007fc3d509cc78 in __new_sem_wait_slow64 (sem=sem@entry=0x572a546f13e8 <thread_pool+168>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183 _buffer = {__routine = 0x7fc3d509cbb0 <__sem_wait_cleanup>, __arg = 0x572a546f13e8 <thread_pool+168>, __canceltype = 1459269632, __prev = 0x0} err = <optimized out> d = 34359738368 #5 0x00007fc3d509ccf1 in __new_sem_wait (sem=sem@entry=0x572a546f13e8 <thread_pool+168>) at ./nptl/sem_wait.c:42 No locals. #6 0x0000572a546a5080 in request_handler_thread (arg=0x572a56ec83f0) at src/main/threads.c:741 self = 0x572a56ec83f0 re_wait = <optimized out> #7 0x00007fc3d5094ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 ret = <optimized out> pd = <optimized out> out = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140720694236800, -4619313545745495514, 140478940894784, 0, 140479069505488, 140720694237152, 4639688863687674406, 4639636563766184486}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> #8 0x00007fc3d5126850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 No locals. Thread 2 (Thread 0x7fc3d15f5640 (LWP 922309) "radiusd"): #0 __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x572a546f13e8 <thread_pool+168>) at ./nptl/futex-internal.c:57 sc_cancel_oldtype = 0 sc_ret = <optimized out> resultvar = <optimized out> __arg6 = <optimized out> __arg5 = <optimized out> __arg4 = <optimized out> __arg3 = <optimized out> __arg2 = <optimized out> __arg1 = <optimized out> _a6 = <optimized out> _a5 = <optimized out> _a4 = <optimized out> _a3 = <optimized out> _a2 = <optimized out> _a1 = <optimized out> #1 __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x572a546f13e8 <thread_pool+168>) at ./nptl/futex-internal.c:87 err = <optimized out> clockbit = 256 op = 393 err = <optimized out> clockbit = <optimized out> op = <optimized out> #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x572a546f13e8 <thread_pool+168>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139 No locals. #3 0x00007fc3d509cbdf in do_futex_wait (sem=sem@entry=0x572a546f13e8 <thread_pool+168>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111 err = <optimized out> #4 0x00007fc3d509cc78 in __new_sem_wait_slow64 (sem=sem@entry=0x572a546f13e8 <thread_pool+168>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183 _buffer = {__routine = 0x7fc3d509cbb0 <__sem_wait_cleanup>, __arg = 0x572a546f13e8 <thread_pool+168>, __canceltype = 1458226016, __prev = 0x0} err = <optimized out> d = 34359738368 #5 0x00007fc3d509ccf1 in __new_sem_wait (sem=sem@entry=0x572a546f13e8 <thread_pool+168>) at ./nptl/sem_wait.c:42 No locals. #6 0x0000572a546a5080 in request_handler_thread (arg=0x572a5702e6d0) at src/main/threads.c:741 self = 0x572a5702e6d0 re_wait = <optimized out> #7 0x00007fc3d5094ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 ret = <optimized out> pd = <optimized out> out = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140720694236800, -4619313545745495514, 140479008036416, 0, 140479069505488, 140720694237152, 4639627303921420838, 4639636563766184486}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> #8 0x00007fc3d5126850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 No locals. Thread 1 (Thread 0x7fc3d5850c00 (LWP 837582) "radiusd"): #0 futex_wait (private=0, expected=2, futex_word=0x572a56ee9c88) at ../sysdeps/nptl/futex-internal.h:146 __ret = -512 err = <optimized out> err = <optimized out> __ret = <optimized out> resultvar = <optimized out> __arg4 = <optimized out> __arg3 = <optimized out> __arg2 = <optimized out> __arg1 = <optimized out> _a4 = <optimized out> _a3 = <optimized out> _a2 = <optimized out> _a1 = <optimized out> #1 __GI___lll_lock_wait (futex=futex@entry=0x572a56ee9c88, private=0) at ./nptl/lowlevellock.c:49 No locals. #2 0x00007fc3d5098002 in lll_mutex_lock_optimized (mutex=0x572a56ee9c88) at ./nptl/pthread_mutex_lock.c:48 __futex = 0x572a56ee9c88 private = <optimized out> private = <optimized out> __value = <optimized out> __futex = <optimized out> #3 ___pthread_mutex_lock (mutex=mutex@entry=0x572a56ee9c88) at ./nptl/pthread_mutex_lock.c:93 type = <optimized out> __PRETTY_FUNCTION__ = "___pthread_mutex_lock" id = <optimized out> #4 0x0000572a546bfc05 in tls_socket_recv (listener=0x572a56e6f710) at src/main/tls_listen.c:476 already_read = false packet = <optimized out> data_len = <optimized out> sock = <optimized out> status = <optimized out> doing_init = false rcode = <optimized out> request = <optimized out> doing_init = <optimized out> already_read = <optimized out> rcode = <optimized out> data_len = <optimized out> packet = <optimized out> request = <optimized out> sock = <optimized out> status = <optimized out> get_application_data = <optimized out> read_application_data = <optimized out> check_for_setup = <optimized out> do_close = <optimized out> __talloc_steal_ret = <optimized out> __talloc_steal_ret = <optimized out> host_ipaddr = <optimized out> #5 dual_tls_recv (listener=0x572a56e6f710) at src/main/tls_listen.c:732 packet = 0x0 fun = <optimized out> sock = 0x572a56ee9b60 client = 0x572a56e70df0 rbio = <optimized out> #6 0x00007fc3d59aaee4 in fr_event_loop (el=0x572a56ab72f0) at src/lib/event.c:713 ef = 0x572a56ab7528 i = 8 rcode = <optimized out> when = {tv_sec = 1727333383, tv_usec = 500000} wake = <optimized out> read_fds = {fds_bits = {0, 8192, 0 <repeats 14 times>}} write_fds = {fds_bits = {0 <repeats 16 times>}} #7 0x0000572a546af1d5 in radius_event_process () at src/main/process.c:6565 No locals. #8 0x0000572a546834bc in main (argc=<optimized out>, argv=<optimized out>) at src/main/radiusd.c:641 rcode = 0 status = <optimized out> argval = <optimized out> spawn_flag = true display_version = false flag = 0 from_child = {4, 5} p = <optimized out> state = 0x572a546f0c80 <global_state> autofree = 0x572a5663a330 --- Thanks
On Sep 26, 2024, at 9:12 AM, James Wood via Freeradius-Users <freeradius-users@lists.freeradius.org> wrote:
Since upgrading to 3.2.6 we're seeing an issue where the radiusd program just "dies" and stops receiving/handling requests, yet the program itself is still running, has a PID, and netstat shows it's still listening on the ports.
I can see the requests still reaching the machine via tcpdump but nothing from radiusd.
We're using TLS to support EAP connections but no clients are using TLS on this particular server so it's all UDP, though TLS is still enabled and listening. OpenSSL version is 3.0.2
I've included some logs from what little I know about gdb and wondered if anyone can advise the likely culprit of the issue.
The last entries in radius.log were normal authentications and then it just died and stopped receiving requests. I've also observed the same in debug mode, nothing is showing untoward in the output on screen before it dies
Hmm..
(gdb) bt
#0 futex_wait (private=0, expected=2, futex_word=0x572a56ee9c88) at ../sysdeps/nptl/futex-internal.h:146 #1 __GI___lll_lock_wait (futex=futex@entry=0x572a56ee9c88, private=0) at ./nptl/lowlevellock.c:49 #2 0x00007fc3d5098002 in lll_mutex_lock_optimized (mutex=0x572a56ee9c88) at ./nptl/pthread_mutex_lock.c:48 #3 ___pthread_mutex_lock (mutex=mutex@entry=0x572a56ee9c88) at ./nptl/pthread_mutex_lock.c:93 #4 0x0000572a546bfc05 in tls_socket_recv (listener=0x572a56e6f710) at src/main/tls_listen.c:476
The code I have for 3.2.6 doesn't have any call to proxy_mutex_lock at line 476 of tls_listen.c That being said, it looks like there are a few cases where the function may return without an unlock. I've pushed a patch. Alan DeKok.
Thanks Alan. Rebuilt with the patch and so far been up for 8 hours with no recurrence. Will continue to monitor but looks good. Thanks again.
Trying to move from 3.0 to 3.2 using 3.2.6 Will this patch only be relevant for installations with TLS enabled? --------------------------------------------------------- Dan Mullen Extended Services - GIS - Cyber Security Tech 925-405-6014 -----Original Message----- From: Freeradius-Users <freeradius-users-bounces+daniel.j.mullen=bofa.com@lists.freeradius.org> On Behalf Of James Wood via Freeradius-Users Sent: Thursday, September 26, 2024 10:42 AM To: FreeRadius users mailing list <freeradius-users@lists.freeradius.org> Cc: James Wood <james.wood@purplewifi.com> Subject: Re: Issue with 3.2.6 - dies/stops serving but is still running/listening on port Thanks Alan. Rebuilt with the patch and so far been up for 8 hours with no recurrence. Will continue to monitor but looks good. Thanks again. - List info/subscribe/unsubscribe? See https://urldefense.com/v3/__http://www.freeradius.org/list/users.html__;!!I2... ---------------------------------------------------------------------- This message, and any attachment(s), is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/electronic-disclaimer. If you are not the intended recipient, please delete this message. For more information about how Bank of America protects your privacy, including specific rights that may apply, please visit the following pages: https://business.bofa.com/en-us/content/global-privacy-notices.html (which includes global privacy notices) and https://www.bankofamerica.com/security-center/privacy-overview/ (which includes US State specific privacy notices such as the http://www.bankofamerica.com/ccpa-notice).
participants (3)
-
Alan DeKok -
James Wood -
Mullen, Daniel J