Aborted when seeing Failing proxied request
Lai Fu Keung
tfklai at hku.hk
Wed Apr 29 08:27:03 CEST 2015
See if the following gdb log (without debugging symbols) is good enough to trace down the problem.
Fu
--------------
Reading symbols from /usr/local/sbin/radiusd...done.
Attaching to program: /usr/local/sbin/radiusd, process 28551
Reading symbols from /usr/local/lib/libfreeradius-server.so...done.
Loaded symbols for /usr/local/lib/libfreeradius-server.so
Reading symbols from /usr/local/lib/libfreeradius-radius.so...done.
Loaded symbols for /usr/local/lib/libfreeradius-radius.so
Reading symbols from /usr/local/lib/libfreeradius-eap.so...done.
Loaded symbols for /usr/local/lib/libfreeradius-eap.so
Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)..
.done.
Loaded symbols for /usr/lib64/libcrypto.so.10
Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...do
ne.
Loaded symbols for /usr/lib64/libssl.so.10
Reading symbols from /usr/lib64/libtalloc.so.2...(no debugging symbols found)...
done.
Loaded symbols for /usr/lib64/libtalloc.so.2
Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done
.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...don
e.
[New LWP 28556]
[New LWP 28555]
[New LWP 28554]
[New LWP 28553]
[New LWP 28552]
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libreadline.so.6...(no debugging symbols found)...do
ne.
Loaded symbols for /lib64/libreadline.so.6
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found).
..done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)..
.done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...don
e.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...do
ne.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libtinfo.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib64/libtinfo.so.5
Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libfreebl3.so
Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)..
.done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...do
ne.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...don
e.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...d
one.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /usr/local/lib/rlm_realm.so...done.
Loaded symbols for /usr/local/lib/rlm_realm.so
Reading symbols from /usr/local/lib/rlm_chap.so...done.
Loaded symbols for /usr/local/lib/rlm_chap.so
Reading symbols from /usr/local/lib/rlm_ldap.so...done.
Loaded symbols for /usr/local/lib/rlm_ldap.so
Reading symbols from /lib64/libldap-2.4.so.2...(no debugging symbols found)...do
ne.
Loaded symbols for /lib64/libldap-2.4.so.2
Reading symbols from /lib64/liblber-2.4.so.2...(no debugging symbols found)...do
ne.
Loaded symbols for /lib64/liblber-2.4.so.2
Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols found)...d
one.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /usr/lib64/libssl3.so...(no debugging symbols found)...done
.
Loaded symbols for /usr/lib64/libssl3.so
Reading symbols from /usr/lib64/libsmime3.so...(no debugging symbols found)...do
ne.
Loaded symbols for /usr/lib64/libsmime3.so
Reading symbols from /usr/lib64/libnss3.so...(no debugging symbols found)...done
.
Loaded symbols for /usr/lib64/libnss3.so
Reading symbols from /usr/lib64/libnssutil3.so...(no debugging symbols found)...
done.
Loaded symbols for /usr/lib64/libnssutil3.so
Reading symbols from /lib64/libplds4.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libplds4.so
Reading symbols from /lib64/libplc4.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libplc4.so
Reading symbols from /lib64/libnspr4.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnspr4.so
Reading symbols from /usr/local/lib/rlm_exec.so...done.
Loaded symbols for /usr/local/lib/rlm_exec.so
Reading symbols from /usr/local/lib/rlm_utf8.so...done.
Loaded symbols for /usr/local/lib/rlm_utf8.so
Reading symbols from /usr/local/lib/rlm_dhcp.so...done.
Loaded symbols for /usr/local/lib/rlm_dhcp.so
Reading symbols from /usr/local/lib/rlm_logintime.so...done.
Loaded symbols for /usr/local/lib/rlm_logintime.so
Reading symbols from /usr/local/lib/rlm_digest.so...done.
Loaded symbols for /usr/local/lib/rlm_digest.so
Reading symbols from /usr/local/lib/rlm_attr_filter.so...done.
Loaded symbols for /usr/local/lib/rlm_attr_filter.so
Reading symbols from /usr/local/lib/rlm_radutmp.so...done.
Loaded symbols for /usr/local/lib/rlm_radutmp.so
Reading symbols from /usr/local/lib/rlm_files.so...done.
Loaded symbols for /usr/local/lib/rlm_files.so
Reading symbols from /usr/local/lib/rlm_preprocess.so...done.
Loaded symbols for /usr/local/lib/rlm_preprocess.so
Reading symbols from /usr/local/lib/rlm_cache.so...done.
Loaded symbols for /usr/local/lib/rlm_cache.so
Reading symbols from /usr/local/lib/rlm_cache_rbtree.so...done.
Loaded symbols for /usr/local/lib/rlm_cache_rbtree.so
Reading symbols from /usr/local/lib/rlm_passwd.so...done.
Loaded symbols for /usr/local/lib/rlm_passwd.so
Reading symbols from /usr/local/lib/rlm_pap.so...done.
Loaded symbols for /usr/local/lib/rlm_pap.so
Reading symbols from /usr/local/lib/rlm_always.so...done.
Loaded symbols for /usr/local/lib/rlm_always.so
Reading symbols from /usr/local/lib/rlm_eap.so...done.
Loaded symbols for /usr/local/lib/rlm_eap.so
Reading symbols from /usr/local/lib/rlm_eap_md5.so...done.
Loaded symbols for /usr/local/lib/rlm_eap_md5.so
Reading symbols from /usr/local/lib/rlm_eap_gtc.so...done.
Loaded symbols for /usr/local/lib/rlm_eap_gtc.so
Reading symbols from /usr/local/lib/rlm_eap_tls.so...done.
Loaded symbols for /usr/local/lib/rlm_eap_tls.so
Reading symbols from /usr/local/lib/rlm_eap_ttls.so...done.
Loaded symbols for /usr/local/lib/rlm_eap_ttls.so
Reading symbols from /usr/local/lib/rlm_eap_peap.so...done.
Loaded symbols for /usr/local/lib/rlm_eap_peap.so
Reading symbols from /usr/local/lib/rlm_eap_mschapv2.so...done.
Loaded symbols for /usr/local/lib/rlm_eap_mschapv2.so
Reading symbols from /usr/local/lib/rlm_soh.so...done.
Loaded symbols for /usr/local/lib/rlm_soh.so
Reading symbols from /usr/local/lib/rlm_mschap.so...done.
Loaded symbols for /usr/local/lib/rlm_mschap.so
Reading symbols from /usr/local/lib/rlm_perl.so...done.
Loaded symbols for /usr/local/lib/rlm_perl.so
Reading symbols from /usr/lib64/perl5/CORE/libperl.so...(no debugging symbols fo
und)...done.
Loaded symbols for /usr/lib64/perl5/CORE/libperl.so
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libutil.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libutil.so.1
Reading symbols from /usr/lib64/perl5/auto/Data/Dumper/Dumper.so...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/perl5/auto/Data/Dumper/Dumper.so
Reading symbols from /usr/local/lib/rlm_detail.so...done.
Loaded symbols for /usr/local/lib/rlm_detail.so
Reading symbols from /usr/local/lib/rlm_replicate.so...done.
Loaded symbols for /usr/local/lib/rlm_replicate.so
Reading symbols from /usr/local/lib/rlm_expiration.so...done.
Loaded symbols for /usr/local/lib/rlm_expiration.so
Reading symbols from /usr/local/lib/rlm_dynamic_clients.so...done.
Loaded symbols for /usr/local/lib/rlm_dynamic_clients.so
Reading symbols from /usr/local/lib/rlm_unpack.so...done.
Loaded symbols for /usr/local/lib/rlm_unpack.so
Reading symbols from /usr/local/lib/rlm_expr.so...done.
Loaded symbols for /usr/local/lib/rlm_expr.so
Reading symbols from /usr/local/lib/rlm_unix.so...done.
Loaded symbols for /usr/local/lib/rlm_unix.so
Reading symbols from /usr/local/lib/rlm_linelog.so...done.
Loaded symbols for /usr/local/lib/rlm_linelog.so
0x00000038374ac65d in waitpid () from /lib64/libc.so.6
No symbol table info available.
No symbol table info available.
Thread 6 (Thread 0x7f85468a6700 (LWP 28552)):
#0 0x000000383780d930 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x000000000042ba2c in request_handler_thread (arg=0x18072d0) at src/main/thr
eads.c:621
self = 0x18072d0
#2 0x00000038378079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3 0x00000038374e88fd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 5 (Thread 0x7f8545ea5700 (LWP 28553)):
#0 0x000000383780d930 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x000000000042ba2c in request_handler_thread (arg=0x18090c0) at src/main/thr
eads.c:621
self = 0x18090c0
#2 0x00000038378079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3 0x00000038374e88fd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 4 (Thread 0x7f85454a4700 (LWP 28554)):
#0 0x000000383780d930 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x000000000042ba2c in request_handler_thread (arg=0x1813450) at src/main/thr
eads.c:621
self = 0x1813450
#2 0x00000038378079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3 0x00000038374e88fd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 3 (Thread 0x7f8544aa3700 (LWP 28555)):
#0 0x000000383780d930 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x000000000042ba2c in request_handler_thread (arg=0x18135e0) at src/main/thr
eads.c:621
self = 0x18135e0
#2 0x00000038378079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3 0x00000038374e88fd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 2 (Thread 0x7f85440a2700 (LWP 28556)):
#0 0x000000383780d930 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x000000000042ba2c in request_handler_thread (arg=0x1813770) at src/main/thr
eads.c:621
self = 0x1813770
#2 0x00000038378079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3 0x00000038374e88fd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 1 (Thread 0x7f85519dd7e0 (LWP 28551)):
#0 0x00000038374ac65d in waitpid () from /lib64/libc.so.6
No symbol table info available.
#1 0x000000383743e609 in do_system () from /lib64/libc.so.6
No symbol table info available.
#2 0x000000383743e940 in system () from /lib64/libc.so.6
No symbol table info available.
#3 0x00007f8551c15469 in fr_fault (sig=11) at src/lib/debug.c:728
disable = true
cmd = "gdb -silent -x /usr/local/etc/raddb/panic.gdb /usr/local/sbin/rad
iusd 28551 > /usr/local/var/log/radius/gdb-radiusd-28551.log 2>&1", '\000' <repe
ats 401 times>
out = 0x7fffb634bfd9 ".log 2>&1"
left = 411
ret = <value optimized out>
p = 0x7f8551e427f3 ".log 2>&1"
q = 0x0
code = <value optimized out>
#4 <signal handler called>
No symbol table info available.
#5 _fr_cursor_init (cursor=0x7fffb634c6a0, vp=0x78) at src/lib/cursor.c:81
No locals.
#6 0x00007f8551e5bdd4 in tmpl_cursor_init (err=0x7fffb634c708, cursor=0x7fffb63
4c6a0, request=0x190fce0, vpt=0x1910540) at src/main/tmpl.c:1713
vps = 0x78
vp = 0x0
#7 0x00007f8551e640c1 in xlat_getvp (ctx=0x1910a90, request=0x190fce0, node=0x1
910510, escape=0x7f8551e5f220 <rad_filename_escape>, escape_ctx=0x0, lvl=0) at s
rc/main/xlat.c:1796
vp = 0x0
virtual = 0x0
packet = 0x0
dv = <value optimized out>
ret = 0x0
err = 0
cursor = {first = 0x78, found = 0x0, last = 0x0, current = 0x0, next = 0
x0}
quote = 0 '\000'
return_null = true
#8 xlat_aprint (ctx=0x1910a90, request=0x190fce0, node=0x1910510, escape=0x7f85
51e5f220 <rad_filename_escape>, escape_ctx=0x0, lvl=0) at src/main/xlat.c:2169
rcode = <value optimized out>
str = 0x0
child = <value optimized out>
p = <value optimized out>
#9 0x00007f8551e63e69 in xlat_aprint (ctx=0x1910a90, request=0x190fce0, node=0x
19103b0, escape=0x7f8551e5f220 <rad_filename_escape>, escape_ctx=0x0, lvl=0) at
src/main/xlat.c:2270
rcode = <value optimized out>
str = 0x0
child = <value optimized out>
p = <value optimized out>
#10 0x00007f8551e64c68 in xlat_process (out=0x7fffb634c8b8, request=0x190fce0, h
ead=<value optimized out>, escape=0x7f8551e5f220 <rad_filename_escape>, escape_c
tx=0x0) at src/main/xlat.c:2354
i = <value optimized out>
list = 7
total = <value optimized out>
array = 0x1910a90
answer = <value optimized out>
node = <value optimized out>
#11 0x00007f8551e64e24 in xlat_expand_struct (out=0x7fffb634c968, outlen=10240,
request=<value optimized out>, node=<value optimized out>, escape=<value optimiz
ed out>, escape_ctx=<value optimized out>) at src/main/xlat.c:2408
buff = 0x0
len = <value optimized out>
#12 0x00007f8551e6506b in xlat_expand (out=0x7fffb634c968, outlen=10240, request
=0x190fce0, fmt=0x13928a0 "/usr/local/var/log/radius/log/radiusd-%{%{Virtual-Ser
ver}:-DEFAULT}-%Y%m%d.log", escape=0x7f8551e5f220 <rad_filename_escape>, escape_
ctx=0x0) at src/main/xlat.c:2472
len = <value optimized out>
node = 0x1910250
#13 0x00007f8551e651b3 in radius_xlat (out=0x7fffb634c9b0 "0", outlen=<value opt
imized out>, request=<value optimized out>, fmt=<value optimized out>, escape=<v
alue optimized out>, ctx=<value optimized out>) at src/main/xlat.c:2530
No locals.
#14 0x00007f8551e561ad in vradlog_request (type=L_PROXY, lvl=<value optimized ou
t>, request=0x190fce0, msg=0x451eb8 "Received response to status check %d (%d in
current sequence)", ap=0x7fffb634f290) at src/main/log.c:665
rl = 0x7f8551e56100 <vradlog_request>
filename = <value optimized out>
fp = 0x0
buffer = "0\000\000\000\060\000\000\000\260\312\064\266\377\177\000\000\
320\311\064\266\377\177\000\000\354\311\064\266\377\177\000\000\000\000\000\000\
000\000\000\000\277\313\064\266\377\177\000\000?\363\064\266\001\000\000\000\021
XU78\000\000\000fXU78\000\000\000\300\367Q\001\000\000\000\000@\313\064\266\377\
177\000\000\022", '\000' <repeats 15 times>, "`vR\001\000\000\000\000@\363\064\2
66\377\177\000\000\350\360D78\000\000\000\060\000\000\000\060\000\000\000 \313\0
64\266\377\177\000\000P\312\064\266\377\177\000\000\335p at U", '\000' <repeats 12
times>, "I\265M78", '\000' <repeats 11 times>"\323, \032G78\000\000\000Յ\346Q\20
5\177\000\000`vR\001\000\000\000\000\247\000\000\000\000\000\000\000\000p\237Q\2
05\177\000\000\022\000\000\000\000\000\000\000\232DG7\000\000\000\000\000\000\00
0\000\377\177", '\000' <repeats 18 times>...
p = <value optimized out>
extra = 0x7f8551e67749 ""
indent = <value optimized out>
aq = {{gp_offset = 7, fp_offset = 0, overflow_arg_area = 0x7fffb634f2f0,
reg_save_area = 0xd0}}
#15 0x00007f8551e56702 in radlog_request (type=<value optimized out>, lvl=<value
optimized out>, request=<value optimized out>, msg=<value optimized out>) at sr
c/main/log.c:789
ap = {{gp_offset = 32, fp_offset = 48, overflow_arg_area = 0x7fffb634f37
0, reg_save_area = 0x7fffb634f2b0}}
#16 0x0000000000430a75 in request_ping (request=0x190fce0, action=<value optimiz
ed out>) at src/main/process.c:3243
home = 0x13fc120
buffer = '\000' <repeats 64 times>"\320, \363\064\266\377\177\000\000\02
4\370?\266\377\177\000\000\240\364\064\266\377\177\000\000\000\000\000\000\000\0
00\000\000\360\363\064\266\377\177\000\000\274\370?\266\377\177\000\000\340\374\
220\001\000\000\000\000@\375\220\001\000\000\000"
__FUNCTION__ = "request_ping"
#17 0x000000000042d9a5 in request_proxy_reply (packet=0x15278d0) at src/main/pro
cess.c:2538
proxy_p = 0x190fd40
request = 0x190fce0
now = {tv_sec = 1430286557, tv_usec = 647413}
buffer = '\000' <repeats 48 times>, "\003\362\000\024\020\000\000\000\02
0\000\000\000\205\177\063\222p\316B", '\000' <repeats 13 times>, "-\000\000\000\
000\000\000\000\001\000\000\000\000\000\000\000\300\367\064\266\377\177\000\000\
330\321L\001\000\000\000\000@\367\064\266\377\177\000\000\\y\302Q\205\177\000"
#18 0x000000000041797f in proxy_socket_recv (listener=0x7f852c001b60) at src/mai
n/listen.c:1980
packet = 0x15278d0
buffer = "\335p at U\000\000\000\000s\346\016\000\000\000\000\000\300\367Q\
001\000\000\000\000\300\367Q\001\000\000\000\000\024\a\000\000\000\000\000\000\2
15\067C\000\000\000\000\000\024\a\000\000\000\000\000\000P\351@\000\000\000\000\
000\036", '\000' <repeats 15 times>, "158.132.12.191", '\000' <repeats 26 times>
"\353, W\000\031?\000\000"
#19 0x000000000043017b in event_socket_handler (xel=<value optimized out>, fd=<v
alue optimized out>, ctx=<value optimized out>) at src/main/process.c:4432
listener = <value optimized out>
#20 0x00007f8551c3067b in fr_event_loop (el=0x14cd1c0) at src/lib/event.c:642
ef = <value optimized out>
i = <value optimized out>
rcode = 1
when = {tv_sec = 1430286557, tv_usec = 649241}
wake = <value optimized out>
maxfd = 45
read_fds = {fds_bits = {35184372088832, 0 <repeats 15 times>}}
master_fds = {fds_bits = {37517613072384, 0 <repeats 15 times>}}
#21 0x0000000000425efc in main (argc=3, argv=<value optimized out>) at src/main/
radiusd.c:581
rcode = 0
status = <value optimized out>
argval = <value optimized out>
spawn_flag = true
write_pid = false
display_version = false
flag = 0
from_child = {4, 5}
autofree = 0x11a9060
A debugging session is active.
Inferior 1 [process 28551] will be detached.
Quit anyway? (y or n) [answered Y; input not from terminal]
-----Original Message-----
From: Freeradius-Users [mailto:freeradius-users-bounces+tfklai=hku.hk at lists.freeradius.org] On Behalf Of Lai Fu Keung
Sent: Wednesday, April 29, 2015 11:42 AM
To: FreeRadius users mailing list
Subject: RE: Aborted when seeing Failing proxied request
> See "panic_action" in radiusd.conf. You can get panic logs without running "radiusd -X".
> PLEASE first install the debugging symbols, so that the gdb logs are useful.
>
> You probably want to set:
>
> panic_action = "gdb -silent -x ${raddbdir}/panic.gdb %e %p > ${logdir}/gdb-${name}-%p.log 2>&1"
I've got the gdb log. But how to install the debugging symbols?
Fu
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
More information about the Freeradius-Users
mailing list