See if the following gdb log (without debugging symbols) is good enough to trace down the problem. Fu -------------- Reading symbols from /usr/local/sbin/radiusd...done. Attaching to program: /usr/local/sbin/radiusd, process 28551 Reading symbols from /usr/local/lib/libfreeradius-server.so...done. Loaded symbols for /usr/local/lib/libfreeradius-server.so Reading symbols from /usr/local/lib/libfreeradius-radius.so...done. Loaded symbols for /usr/local/lib/libfreeradius-radius.so Reading symbols from /usr/local/lib/libfreeradius-eap.so...done. Loaded symbols for /usr/local/lib/libfreeradius-eap.so Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found).. .done. Loaded symbols for /usr/lib64/libcrypto.so.10 Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...do ne. Loaded symbols for /usr/lib64/libssl.so.10 Reading symbols from /usr/lib64/libtalloc.so.2...(no debugging symbols found)... done. Loaded symbols for /usr/lib64/libtalloc.so.2 Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libnsl.so.1 Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done . Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...don e. [New LWP 28556] [New LWP 28555] [New LWP 28554] [New LWP 28553] [New LWP 28552] [Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libreadline.so.6...(no debugging symbols found)...do ne. Loaded symbols for /lib64/libreadline.so.6 Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libcrypt.so.1 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found). ..done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libz.so.1 Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found).. .done. Loaded symbols for /lib64/libgssapi_krb5.so.2 Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done. Loaded symbols for /lib64/libkrb5.so.3 Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...don e. Loaded symbols for /lib64/libcom_err.so.2 Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...do ne. Loaded symbols for /lib64/libk5crypto.so.3 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /lib64/libtinfo.so.5...(no debugging symbols found)...done. Loaded symbols for /lib64/libtinfo.so.5 Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done. Loaded symbols for /lib64/libfreebl3.so Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found).. .done. Loaded symbols for /lib64/libkrb5support.so.0 Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...do ne. Loaded symbols for /lib64/libkeyutils.so.1 Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...don e. Loaded symbols for /lib64/libselinux.so.1 Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...d one. Loaded symbols for /lib64/libnss_files.so.2 Reading symbols from /usr/local/lib/rlm_realm.so...done. Loaded symbols for /usr/local/lib/rlm_realm.so Reading symbols from /usr/local/lib/rlm_chap.so...done. Loaded symbols for /usr/local/lib/rlm_chap.so Reading symbols from /usr/local/lib/rlm_ldap.so...done. Loaded symbols for /usr/local/lib/rlm_ldap.so Reading symbols from /lib64/libldap-2.4.so.2...(no debugging symbols found)...do ne. Loaded symbols for /lib64/libldap-2.4.so.2 Reading symbols from /lib64/liblber-2.4.so.2...(no debugging symbols found)...do ne. Loaded symbols for /lib64/liblber-2.4.so.2 Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols found)...d one. Loaded symbols for /usr/lib64/libsasl2.so.2 Reading symbols from /usr/lib64/libssl3.so...(no debugging symbols found)...done . Loaded symbols for /usr/lib64/libssl3.so Reading symbols from /usr/lib64/libsmime3.so...(no debugging symbols found)...do ne. Loaded symbols for /usr/lib64/libsmime3.so Reading symbols from /usr/lib64/libnss3.so...(no debugging symbols found)...done . Loaded symbols for /usr/lib64/libnss3.so Reading symbols from /usr/lib64/libnssutil3.so...(no debugging symbols found)... done. Loaded symbols for /usr/lib64/libnssutil3.so Reading symbols from /lib64/libplds4.so...(no debugging symbols found)...done. Loaded symbols for /lib64/libplds4.so Reading symbols from /lib64/libplc4.so...(no debugging symbols found)...done. Loaded symbols for /lib64/libplc4.so Reading symbols from /lib64/libnspr4.so...(no debugging symbols found)...done. Loaded symbols for /lib64/libnspr4.so Reading symbols from /usr/local/lib/rlm_exec.so...done. Loaded symbols for /usr/local/lib/rlm_exec.so Reading symbols from /usr/local/lib/rlm_utf8.so...done. Loaded symbols for /usr/local/lib/rlm_utf8.so Reading symbols from /usr/local/lib/rlm_dhcp.so...done. Loaded symbols for /usr/local/lib/rlm_dhcp.so Reading symbols from /usr/local/lib/rlm_logintime.so...done. Loaded symbols for /usr/local/lib/rlm_logintime.so Reading symbols from /usr/local/lib/rlm_digest.so...done. Loaded symbols for /usr/local/lib/rlm_digest.so Reading symbols from /usr/local/lib/rlm_attr_filter.so...done. Loaded symbols for /usr/local/lib/rlm_attr_filter.so Reading symbols from /usr/local/lib/rlm_radutmp.so...done. Loaded symbols for /usr/local/lib/rlm_radutmp.so Reading symbols from /usr/local/lib/rlm_files.so...done. Loaded symbols for /usr/local/lib/rlm_files.so Reading symbols from /usr/local/lib/rlm_preprocess.so...done. Loaded symbols for /usr/local/lib/rlm_preprocess.so Reading symbols from /usr/local/lib/rlm_cache.so...done. Loaded symbols for /usr/local/lib/rlm_cache.so Reading symbols from /usr/local/lib/rlm_cache_rbtree.so...done. Loaded symbols for /usr/local/lib/rlm_cache_rbtree.so Reading symbols from /usr/local/lib/rlm_passwd.so...done. Loaded symbols for /usr/local/lib/rlm_passwd.so Reading symbols from /usr/local/lib/rlm_pap.so...done. Loaded symbols for /usr/local/lib/rlm_pap.so Reading symbols from /usr/local/lib/rlm_always.so...done. Loaded symbols for /usr/local/lib/rlm_always.so Reading symbols from /usr/local/lib/rlm_eap.so...done. Loaded symbols for /usr/local/lib/rlm_eap.so Reading symbols from /usr/local/lib/rlm_eap_md5.so...done. Loaded symbols for /usr/local/lib/rlm_eap_md5.so Reading symbols from /usr/local/lib/rlm_eap_gtc.so...done. Loaded symbols for /usr/local/lib/rlm_eap_gtc.so Reading symbols from /usr/local/lib/rlm_eap_tls.so...done. Loaded symbols for /usr/local/lib/rlm_eap_tls.so Reading symbols from /usr/local/lib/rlm_eap_ttls.so...done. Loaded symbols for /usr/local/lib/rlm_eap_ttls.so Reading symbols from /usr/local/lib/rlm_eap_peap.so...done. Loaded symbols for /usr/local/lib/rlm_eap_peap.so Reading symbols from /usr/local/lib/rlm_eap_mschapv2.so...done. Loaded symbols for /usr/local/lib/rlm_eap_mschapv2.so Reading symbols from /usr/local/lib/rlm_soh.so...done. Loaded symbols for /usr/local/lib/rlm_soh.so Reading symbols from /usr/local/lib/rlm_mschap.so...done. Loaded symbols for /usr/local/lib/rlm_mschap.so Reading symbols from /usr/local/lib/rlm_perl.so...done. Loaded symbols for /usr/local/lib/rlm_perl.so Reading symbols from /usr/lib64/perl5/CORE/libperl.so...(no debugging symbols fo und)...done. Loaded symbols for /usr/lib64/perl5/CORE/libperl.so Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/libutil.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libutil.so.1 Reading symbols from /usr/lib64/perl5/auto/Data/Dumper/Dumper.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/auto/Data/Dumper/Dumper.so Reading symbols from /usr/local/lib/rlm_detail.so...done. Loaded symbols for /usr/local/lib/rlm_detail.so Reading symbols from /usr/local/lib/rlm_replicate.so...done. Loaded symbols for /usr/local/lib/rlm_replicate.so Reading symbols from /usr/local/lib/rlm_expiration.so...done. Loaded symbols for /usr/local/lib/rlm_expiration.so Reading symbols from /usr/local/lib/rlm_dynamic_clients.so...done. Loaded symbols for /usr/local/lib/rlm_dynamic_clients.so Reading symbols from /usr/local/lib/rlm_unpack.so...done. Loaded symbols for /usr/local/lib/rlm_unpack.so Reading symbols from /usr/local/lib/rlm_expr.so...done. Loaded symbols for /usr/local/lib/rlm_expr.so Reading symbols from /usr/local/lib/rlm_unix.so...done. Loaded symbols for /usr/local/lib/rlm_unix.so Reading symbols from /usr/local/lib/rlm_linelog.so...done. Loaded symbols for /usr/local/lib/rlm_linelog.so 0x00000038374ac65d in waitpid () from /lib64/libc.so.6 No symbol table info available. No symbol table info available. Thread 6 (Thread 0x7f85468a6700 (LWP 28552)): #0 0x000000383780d930 in sem_wait () from /lib64/libpthread.so.0 No symbol table info available. #1 0x000000000042ba2c in request_handler_thread (arg=0x18072d0) at src/main/thr eads.c:621 self = 0x18072d0 #2 0x00000038378079d1 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00000038374e88fd in clone () from /lib64/libc.so.6 No symbol table info available. Thread 5 (Thread 0x7f8545ea5700 (LWP 28553)): #0 0x000000383780d930 in sem_wait () from /lib64/libpthread.so.0 No symbol table info available. #1 0x000000000042ba2c in request_handler_thread (arg=0x18090c0) at src/main/thr eads.c:621 self = 0x18090c0 #2 0x00000038378079d1 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00000038374e88fd in clone () from /lib64/libc.so.6 No symbol table info available. Thread 4 (Thread 0x7f85454a4700 (LWP 28554)): #0 0x000000383780d930 in sem_wait () from /lib64/libpthread.so.0 No symbol table info available. #1 0x000000000042ba2c in request_handler_thread (arg=0x1813450) at src/main/thr eads.c:621 self = 0x1813450 #2 0x00000038378079d1 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00000038374e88fd in clone () from /lib64/libc.so.6 No symbol table info available. Thread 3 (Thread 0x7f8544aa3700 (LWP 28555)): #0 0x000000383780d930 in sem_wait () from /lib64/libpthread.so.0 No symbol table info available. #1 0x000000000042ba2c in request_handler_thread (arg=0x18135e0) at src/main/thr eads.c:621 self = 0x18135e0 #2 0x00000038378079d1 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00000038374e88fd in clone () from /lib64/libc.so.6 No symbol table info available. Thread 2 (Thread 0x7f85440a2700 (LWP 28556)): #0 0x000000383780d930 in sem_wait () from /lib64/libpthread.so.0 No symbol table info available. #1 0x000000000042ba2c in request_handler_thread (arg=0x1813770) at src/main/thr eads.c:621 self = 0x1813770 #2 0x00000038378079d1 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00000038374e88fd in clone () from /lib64/libc.so.6 No symbol table info available. Thread 1 (Thread 0x7f85519dd7e0 (LWP 28551)): #0 0x00000038374ac65d in waitpid () from /lib64/libc.so.6 No symbol table info available. #1 0x000000383743e609 in do_system () from /lib64/libc.so.6 No symbol table info available. #2 0x000000383743e940 in system () from /lib64/libc.so.6 No symbol table info available. #3 0x00007f8551c15469 in fr_fault (sig=11) at src/lib/debug.c:728 disable = true cmd = "gdb -silent -x /usr/local/etc/raddb/panic.gdb /usr/local/sbin/rad iusd 28551 > /usr/local/var/log/radius/gdb-radiusd-28551.log 2>&1", '\000' <repe ats 401 times> out = 0x7fffb634bfd9 ".log 2>&1" left = 411 ret = <value optimized out> p = 0x7f8551e427f3 ".log 2>&1" q = 0x0 code = <value optimized out> #4 <signal handler called> No symbol table info available. #5 _fr_cursor_init (cursor=0x7fffb634c6a0, vp=0x78) at src/lib/cursor.c:81 No locals. #6 0x00007f8551e5bdd4 in tmpl_cursor_init (err=0x7fffb634c708, cursor=0x7fffb63 4c6a0, request=0x190fce0, vpt=0x1910540) at src/main/tmpl.c:1713 vps = 0x78 vp = 0x0 #7 0x00007f8551e640c1 in xlat_getvp (ctx=0x1910a90, request=0x190fce0, node=0x1 910510, escape=0x7f8551e5f220 <rad_filename_escape>, escape_ctx=0x0, lvl=0) at s rc/main/xlat.c:1796 vp = 0x0 virtual = 0x0 packet = 0x0 dv = <value optimized out> ret = 0x0 err = 0 cursor = {first = 0x78, found = 0x0, last = 0x0, current = 0x0, next = 0 x0} quote = 0 '\000' return_null = true #8 xlat_aprint (ctx=0x1910a90, request=0x190fce0, node=0x1910510, escape=0x7f85 51e5f220 <rad_filename_escape>, escape_ctx=0x0, lvl=0) at src/main/xlat.c:2169 rcode = <value optimized out> str = 0x0 child = <value optimized out> p = <value optimized out> #9 0x00007f8551e63e69 in xlat_aprint (ctx=0x1910a90, request=0x190fce0, node=0x 19103b0, escape=0x7f8551e5f220 <rad_filename_escape>, escape_ctx=0x0, lvl=0) at src/main/xlat.c:2270 rcode = <value optimized out> str = 0x0 child = <value optimized out> p = <value optimized out> #10 0x00007f8551e64c68 in xlat_process (out=0x7fffb634c8b8, request=0x190fce0, h ead=<value optimized out>, escape=0x7f8551e5f220 <rad_filename_escape>, escape_c tx=0x0) at src/main/xlat.c:2354 i = <value optimized out> list = 7 total = <value optimized out> array = 0x1910a90 answer = <value optimized out> node = <value optimized out> #11 0x00007f8551e64e24 in xlat_expand_struct (out=0x7fffb634c968, outlen=10240, request=<value optimized out>, node=<value optimized out>, escape=<value optimiz ed out>, escape_ctx=<value optimized out>) at src/main/xlat.c:2408 buff = 0x0 len = <value optimized out> #12 0x00007f8551e6506b in xlat_expand (out=0x7fffb634c968, outlen=10240, request =0x190fce0, fmt=0x13928a0 "/usr/local/var/log/radius/log/radiusd-%{%{Virtual-Ser ver}:-DEFAULT}-%Y%m%d.log", escape=0x7f8551e5f220 <rad_filename_escape>, escape_ ctx=0x0) at src/main/xlat.c:2472 len = <value optimized out> node = 0x1910250 #13 0x00007f8551e651b3 in radius_xlat (out=0x7fffb634c9b0 "0", outlen=<value opt imized out>, request=<value optimized out>, fmt=<value optimized out>, escape=<v alue optimized out>, ctx=<value optimized out>) at src/main/xlat.c:2530 No locals. #14 0x00007f8551e561ad in vradlog_request (type=L_PROXY, lvl=<value optimized ou t>, request=0x190fce0, msg=0x451eb8 "Received response to status check %d (%d in current sequence)", ap=0x7fffb634f290) at src/main/log.c:665 rl = 0x7f8551e56100 <vradlog_request> filename = <value optimized out> fp = 0x0 buffer = "0\000\000\000\060\000\000\000\260\312\064\266\377\177\000\000\ 320\311\064\266\377\177\000\000\354\311\064\266\377\177\000\000\000\000\000\000\ 000\000\000\000\277\313\064\266\377\177\000\000?\363\064\266\001\000\000\000\021 XU78\000\000\000fXU78\000\000\000\300\367Q\001\000\000\000\000@\313\064\266\377\ 177\000\000\022", '\000' <repeats 15 times>, "`vR\001\000\000\000\000@\363\064\2 66\377\177\000\000\350\360D78\000\000\000\060\000\000\000\060\000\000\000 \313\0 64\266\377\177\000\000P\312\064\266\377\177\000\000\335p@U", '\000' <repeats 12 times>, "I\265M78", '\000' <repeats 11 times>"\323, \032G78\000\000\000Յ\346Q\20 5\177\000\000`vR\001\000\000\000\000\247\000\000\000\000\000\000\000\000p\237Q\2 05\177\000\000\022\000\000\000\000\000\000\000\232DG7\000\000\000\000\000\000\00 0\000\377\177", '\000' <repeats 18 times>... p = <value optimized out> extra = 0x7f8551e67749 "" indent = <value optimized out> aq = {{gp_offset = 7, fp_offset = 0, overflow_arg_area = 0x7fffb634f2f0, reg_save_area = 0xd0}} #15 0x00007f8551e56702 in radlog_request (type=<value optimized out>, lvl=<value optimized out>, request=<value optimized out>, msg=<value optimized out>) at sr c/main/log.c:789 ap = {{gp_offset = 32, fp_offset = 48, overflow_arg_area = 0x7fffb634f37 0, reg_save_area = 0x7fffb634f2b0}} #16 0x0000000000430a75 in request_ping (request=0x190fce0, action=<value optimiz ed out>) at src/main/process.c:3243 home = 0x13fc120 buffer = '\000' <repeats 64 times>"\320, \363\064\266\377\177\000\000\02 4\370?\266\377\177\000\000\240\364\064\266\377\177\000\000\000\000\000\000\000\0 00\000\000\360\363\064\266\377\177\000\000\274\370?\266\377\177\000\000\340\374\ 220\001\000\000\000\000@\375\220\001\000\000\000" __FUNCTION__ = "request_ping" #17 0x000000000042d9a5 in request_proxy_reply (packet=0x15278d0) at src/main/pro cess.c:2538 proxy_p = 0x190fd40 request = 0x190fce0 now = {tv_sec = 1430286557, tv_usec = 647413} buffer = '\000' <repeats 48 times>, "\003\362\000\024\020\000\000\000\02 0\000\000\000\205\177\063\222p\316B", '\000' <repeats 13 times>, "-\000\000\000\ 000\000\000\000\001\000\000\000\000\000\000\000\300\367\064\266\377\177\000\000\ 330\321L\001\000\000\000\000@\367\064\266\377\177\000\000\\y\302Q\205\177\000" #18 0x000000000041797f in proxy_socket_recv (listener=0x7f852c001b60) at src/mai n/listen.c:1980 packet = 0x15278d0 buffer = "\335p@U\000\000\000\000s\346\016\000\000\000\000\000\300\367Q\ 001\000\000\000\000\300\367Q\001\000\000\000\000\024\a\000\000\000\000\000\000\2 15\067C\000\000\000\000\000\024\a\000\000\000\000\000\000P\351@\000\000\000\000\ 000\036", '\000' <repeats 15 times>, "158.132.12.191", '\000' <repeats 26 times> "\353, W\000\031?\000\000" #19 0x000000000043017b in event_socket_handler (xel=<value optimized out>, fd=<v alue optimized out>, ctx=<value optimized out>) at src/main/process.c:4432 listener = <value optimized out> #20 0x00007f8551c3067b in fr_event_loop (el=0x14cd1c0) at src/lib/event.c:642 ef = <value optimized out> i = <value optimized out> rcode = 1 when = {tv_sec = 1430286557, tv_usec = 649241} wake = <value optimized out> maxfd = 45 read_fds = {fds_bits = {35184372088832, 0 <repeats 15 times>}} master_fds = {fds_bits = {37517613072384, 0 <repeats 15 times>}} #21 0x0000000000425efc in main (argc=3, argv=<value optimized out>) at src/main/ radiusd.c:581 rcode = 0 status = <value optimized out> argval = <value optimized out> spawn_flag = true write_pid = false display_version = false flag = 0 from_child = {4, 5} autofree = 0x11a9060 A debugging session is active. Inferior 1 [process 28551] will be detached. Quit anyway? (y or n) [answered Y; input not from terminal] -----Original Message----- From: Freeradius-Users [mailto:freeradius-users-bounces+tfklai=hku.hk@lists.freeradius.org] On Behalf Of Lai Fu Keung Sent: Wednesday, April 29, 2015 11:42 AM To: FreeRadius users mailing list Subject: RE: Aborted when seeing Failing proxied request
See "panic_action" in radiusd.conf. You can get panic logs without running "radiusd -X". PLEASE first install the debugging symbols, so that the gdb logs are useful.
You probably want to set:
panic_action = "gdb -silent -x ${raddbdir}/panic.gdb %e %p > ${logdir}/gdb-${name}-%p.log 2>&1"
I've got the gdb log. But how to install the debugging symbols? Fu - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html