Hi, we've been running FR 3.0.10 in production for about a month now. We have four production servers on RHEL 6. Every few days one of the radiusd processes quits without an error message. We have now enabled the panic action and installed the debuginfo RPMs. Last night we got the first usable gbd.log. Here's the relevant part: Thread 4 (Thread 0x7f8b6cb36700 (LWP 9823)): #0 0x0000003a1ae0cdab in pthread_once () from /lib64/libpthread.so.0 No symbol table info available. #1 0x0000003a1aafea04 in backtrace () from /lib64/libc.so.6 No symbol table info available. #2 0x0000003a1aa7054b in __libc_message () from /lib64/libc.so.6 No symbol table info available. #3 0x0000003a1aa75e66 in malloc_printerr () from /lib64/libc.so.6 No symbol table info available. #4 0x0000003a1aa799a4 in _int_malloc () from /lib64/libc.so.6 No symbol table info available. #5 0x0000003a1aa7a3e6 in calloc () from /lib64/libc.so.6 No symbol table info available. #6 0x0000003a1a60addf in _dl_new_object () from /lib64/ld-linux-x86-64.so.2 No symbol table info available. #7 0x0000003a1a6071de in _dl_map_object_from_fd () from /lib64/ld-linux-x86-64.so.2 No symbol table info available. #8 0x0000003a1a60839a in _dl_map_object () from /lib64/ld-linux-x86-64.so.2 No symbol table info available. #9 0x0000003a1a612b95 in dl_open_worker () from /lib64/ld-linux-x86-64.so.2 No symbol table info available. #10 0x0000003a1a60e286 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2 No symbol table info available. #11 0x0000003a1a61263a in _dl_open () from /lib64/ld-linux-x86-64.so.2 No symbol table info available. #12 0x0000003a1ab26cd0 in do_dlopen () from /lib64/libc.so.6 No symbol table info available. #13 0x0000003a1a60e286 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2 No symbol table info available. #14 0x0000003a1ab26e27 in __libc_dlopen_mode () from /lib64/libc.so.6 No symbol table info available. #15 0x0000003a1aafe8d5 in init () from /lib64/libc.so.6 No symbol table info available. #16 0x0000003a1ae0cdb3 in pthread_once () from /lib64/libpthread.so.0 No symbol table info available. #17 0x0000003a1aafea04 in backtrace () from /lib64/libc.so.6 No symbol table info available. #18 0x0000003a1aa7054b in __libc_message () from /lib64/libc.so.6 No symbol table info available. #19 0x0000003a1aa75e66 in malloc_printerr () from /lib64/libc.so.6 No symbol table info available. #20 0x0000003a1aa789ba in _int_free () from /lib64/libc.so.6 No symbol table info available. #21 0x0000003a20202389 in ?? () from /usr/lib64/libtalloc.so.2 No symbol table info available. #22 0x000000396dc15dbc in fr_pair_value_strsteal (vp=0x7f8b48732dc0, src=0x7f8b4874f5e0 "leap:session-key=\200hl\240:l\236\255t\256\247\366Z\315%f\233\220q~\343-\207VΟ\334\366N%i\372p\274Q\242\243\232ީ/\257\362\315\376f\344\067>0\356\262\v\233\343\240\344pH\213\177") at /usr/src/debug/freeradius-server-3.0.10/src/lib/pair.c:2125 q = <value optimized out> #23 0x00007f8b751ac8da in mod_post_proxy (inst=0x19c0c60, request=0x1c6c590) at /usr/src/debug/freeradius-server-3.0.10/src/modules/rlm_eap/rlm_eap.c:675 i = 34 len = 130 p = 0x7f8b4874f5e0 "leap:session-key=\200hl\240:l\236\255t\256\247\366Z\315%f\233\220q~\343-\207VΟ\334\366N%i\372p\274Q\242\243\232ީ/\257\362\315\376f\344\067>0\356\262\v\233\343\240\344pH\213\177" vp = 0x7f8b48732dc0 handler = <value optimized out> cursor = {first = 0x1c170d8, found = 0x7f8b48732dc0, last = 0x0, current = 0x7f8b48732dc0, next = 0x7f8b48708160} #24 0x0000000000425f4f in call_modsingle (request=0x1c6c590, component=MOD_POST_PROXY, depth=1, entry=0x7f8b6cb359c8, do_next_sibling=true) at /usr/src/debug/freeradius-server-3.0.10/src/main/modcall.c:302 blocked = <value optimized out> indent = 4 #25 modcall_recurse (request=0x1c6c590, component=MOD_POST_PROXY, depth=1, entry=0x7f8b6cb359c8, do_next_sibling=true) at /usr/src/debug/freeradius-server-3.0.10/src/main/modcall.c:574 sp = 0x1a672f0 if_taken = false was_if = false c = <value optimized out> priority = <value optimized out> result = RLM_MODULE_OK #26 0x00000000004267ba in modcall_child (request=<value optimized out>, component=<value optimized out>, depth=<value optimized out>, entry=0x7f8b6cb359b0, c=<value optimized out>, result=0x7f8b6cb3580c, do_next_sibling=true) at /usr/src/debug/freeradius-server-3.0.10/src/main/modcall.c:408 next = 0x7f8b6cb359c8 #27 0x0000000000425575 in modcall_recurse (request=0x1c6c590, component=MOD_POST_PROXY, depth=0, entry=0x7f8b6cb359b0, do_next_sibling=true) at /usr/src/debug/freeradius-server-3.0.10/src/main/modcall.c:785 g = 0x1a67150 if_taken = false was_if = false c = <value optimized out> priority = <value optimized out> result = RLM_MODULE_UNKNOWN #28 0x0000000000426753 in modcall (component=<value optimized out>, c=<value optimized out>, request=<value optimized out>) at /usr/src/debug/freeradius-server-3.0.10/src/main/modcall.c:1130 stack = {{result = RLM_MODULE_NOOP, priority = 0, unwind = 0, c = 0x1a67150}, {result = RLM_MODULE_OK, priority = 3, unwind = 0, c = 0x1a672f0}, {result = 1823693344, priority = 32651, unwind = 29811200, c = 0x0}, {result = 1841447521, priority = 57, unwind = 1823693888, c = 0x3a1d2e8029}, {result = RLM_MODULE_FAIL, priority = 0, unwind = 490166044, c = 0x7f8b48708160}, {result = 1215624496, priority = 32651, unwind = 3, c = 0x1d5}, {result = 24099536, priority = 0, unwind = 1823693448, c = 0x7f8b480008c0}, {result = 24099488, priority = 0, unwind = 24099536, c = 0x3a1d2e7e92}, {result = 1823693504, priority = 32651, unwind = 489586107, c = 0x0}, {result = RLM_MODULE_REJECT, priority = 0, unwind = 0, c = 0x7f8b6cb35d20}, {result = 11, priority = 32651, unwind = 489586811, c = 0x0}, {result = RLM_MODULE_REJECT, priority = 0, unwind = 9, c = 0x2}, {result = RLM_MODULE_REJECT, priority = 0, unwind = 176, c = 0x0}, {result = RLM_MODULE_REJECT, priority = 0, unwind = 0, c = 0x3200000009}, {result = 91, priority = 124, unwind = 119, c = 0x396dc2f86a}, {result = 1215353712, priority = 32651, unwind = 129, c = 0x7f8b48000020}, {result = 29454432, priority = 0, unwind = 1823693888, c = 0x7f8b6cb35c40}, {result = 4382876, priority = 0, unwind = 1215353712, c = 0xe6dfd36399d685d1}, {result = 49, priority = 0, unwind = 29454432, c = 0x1c1727c}, {result = 29811200, priority = 0, unwind = 29454432, c = 0x7f8b6cb35c40}, {result = 1823693888, priority = 32651, unwind = 1841429743, c = 0x3}, {result = RLM_MODULE_HANDLED, priority = 0, unwind = 1823693888, c = 0x10}, {result = RLM_MODULE_USERLOCK, priority = 0, unwind = 26404800, c = 0x8}, {result = 1841415815, priority = 57, unwind = 8, c = 0x7f8b589caf50}, {result = 1841506360, priority = 57, unwind = 1486663584, c = 0x0}, {result = 1843648576, priority = 57, unwind = 1843648576, c = 0x3a20202389}, {result = 1486663584, priority = 32651, unwind = 27885456, c = 0x0}, {result = 1486663584, priority = 32651, unwind = 1843648576, c = 0x396dc20f10}, {result = RLM_MODULE_REJECT, priority = 0, unwind = 1486663584, c = 0x1a97f90}, {result = 27852576, priority = 0, unwind = 29811200, c = 0x1}, {result = 1823693984, priority = 32651, unwind = 1516734484, c = 0x7f8b6cb35ce0}, {result = 4323808, priority = 0, unwind = 1823694016, c = 0x1c6c590}} #29 0x000000000041fd95 in indexed_modcall (comp=MOD_POST_PROXY, idx=0, request=0x1c6c590) at /usr/src/debug/freeradius-server-3.0.10/src/main/modules.c:1009 rcode = <value optimized out> list = 0x1a67150 server = 0x1a5df00 #30 0x000000000042e8f3 in process_proxy_reply (request=0x1c6c590, reply=0x1c17060) at /usr/src/debug/freeradius-server-3.0.10/src/main/process.c:2394 rcode = <value optimized out> post_proxy_type = <value optimized out> vp = <value optimized out> #31 0x00000000004354c9 in proxy_running (request=0x1c6c590, action=<value optimized out>) at /usr/src/debug/freeradius-server-3.0.10/src/main/process.c:2741 __FUNCTION__ = "proxy_running" #32 0x000000000042cb4a in request_handler_thread (arg=0x16fbaa0) at /usr/src/debug/freeradius-server-3.0.10/src/main/threads.c:690 self = 0x16fbaa0 #33 0x0000003a1ae07a51 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #34 0x0000003a1aae89ad in clone () from /lib64/libc.so.6 No symbol table info available. As far as we have been able to discern, previous crashes didn't all occur in post-proxy. The same user authenticated successfully before, so it also doesn't seem to related to that specific user. I will post future backtraces if they offer new evidence. Please let me know if additional info would be helpful. -- .:.Sebastian Hagedorn - Weyertal 121 (Gebäude 133), Zimmer 2.02.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universität zu Köln / Cologne University - ✆ +49-221-470-89578.:.