Stack trace of crash with FR 3.0.10

[Sebastian Hagedorn]

--On 18. November 2015 um 09:13:46 -0500 Alan DeKok 
<aland at deployingradius.com> wrote:

>> we've been running FR 3.0.10 in production for about a month now. We
>> have four production servers on RHEL 6. Every few days one of the
>> radiusd processes quits without an error message. We have now enabled
>> the panic action and installed the debuginfo RPMs. Last night we got the
>> first usable gbd.log. Here's the relevant part:
>
>   Thanks.
>>
>> # 22 0x000000396dc15dbc in fr_pair_value_strsteal (vp=0x7f8b48732dc0,
>> # src=0x7f8b4874f5e0
>> # "leap:session-key=\200hl\240:l\236\255t\256\247\366Z\315%f\233\220q~\3
>> # 43-\207VΟ\334\366N%i\372p\274Q\242\243\232ީ/\257\362\315\376f\344\06
>> # 7>0\356\262\v\233\343\240\344pH\213\177") at
>> # /usr/src/debug/freeradius-server-3.0.10/src/lib/pair.c:2125
>>       q = <value optimized out>
>
>   LEAP?  Don't use that.  It's insecure.

Actually we don't use LEAP and were surprised by that message. The session 
in question was using EAP-TTLS.

>> As far as we have been able to discern, previous crashes didn't all
>> occur in post-proxy.
>
>   OK.  So it's something else.
>
>> The same user authenticated successfully before, so it also doesn't seem
>> to related to that specific user. I will post future backtraces if they
>> offer new evidence. Please let me know if additional info would be
>> helpful.
>
>   Any other back traces would help.
>
>   This might be hard to track down without local tests...

I'm willing to test anything you suggest. Meanwhile here's a new back trace:

Thread 1 (Thread 0x7f57f27b87e0 (LWP 28274)):
#0  0x0000003a1aaac70d in waitpid () from /lib64/libc.so.6
No symbol table info available.
#1  0x0000003a1aa3e609 in do_system () from /lib64/libc.so.6
No symbol table info available.
#2  0x0000003a1aa3e940 in system () from /lib64/libc.so.6
No symbol table info available.
#3  0x000000396dc0c15c in fr_fault (sig=<value optimized out>) at 
/usr/src/debug/freeradius-server-3.0.10/src/lib/debug.c:725
        disable = true
        cmd = "gdb -silent -x /etc/raddb/panic.gdb /usr/sbin/radiusd 28274 
2>&1 | tee /var/log/radius/gdb-radiusd-28274.log", '\000' <repeats 423 
times>
        out = 0x7fff54dafcd8 ".log"
        left = 428
        ret = <value optimized out>
        p = 0x396de3d962 ".log"
        q = 0x0
        code = <value optimized out>
#4  <signal handler called>
No symbol table info available.
#5  0x0000003a20202d19 in ?? () from /usr/lib64/libtalloc.so.2
No symbol table info available.
#6  0x0000003a2020244f in ?? () from /usr/lib64/libtalloc.so.2
No symbol table info available.
#7  0x0000003a20202323 in ?? () from /usr/lib64/libtalloc.so.2
No symbol table info available.
#8  0x0000003a20202323 in ?? () from /usr/lib64/libtalloc.so.2
No symbol table info available.
#9  0x0000000000431257 in request_done (request=0x2bd3de0, action=2) at 
/usr/src/debug/freeradius-server-3.0.10/src/main/process.c:806
        now = {tv_sec = 40078192, tv_usec = 45947520}
        when = {tv_sec = 140734617027872, tv_usec = 40077872}
#10 0x0000000000432c31 in request_receive (ctx=0x2bd1a80, 
listener=0x2638a30, packet=0x2bd1ae0, client=0x263e340, fun=0x40efb0 
<rad_authenticate>) at 
/usr/src/debug/freeradius-server-3.0.10/src/main/process.c:1656
        child_state = <value optimized out>
        count = <value optimized out>
        packet_p = <value optimized out>
        request = <value optimized out>
        now = {tv_sec = 1447842529, tv_usec = 339180}
        sock = 0x2638b70
#11 0x0000000000419751 in auth_socket_recv (listener=0x2638a30) at 
/usr/src/debug/freeradius-server-3.0.10/src/main/listen.c:1568
        rcode = <value optimized out>
        code = 1
        src_port = <value optimized out>
        packet = <value optimized out>
        fun = 0x40efb0 <rad_authenticate>
        client = 0x263e340
        src_ipaddr = {af = 2, ipaddr = {ip4addr = {s_addr = 36312236}, 
ip6addr = {__in6_u = {__u6_addr8 = "\254\024*\002", '\000' <repeats 11 
times>, __u6_addr16 = {5292, 554, 0, 0, 0, 0, 0, 0}, __u6_addr32 = 
{36312236, 0, 0, 0}}}}, prefix = 32 ' ', scope = 0}
        ctx = 0x2bd1a80
#12 0x0000000000430fa0 in event_socket_handler (xel=<value optimized out>, 
fd=<value optimized out>, ctx=<value optimized out>) at 
/usr/src/debug/freeradius-server-3.0.10/src/main/process.c:4556
        listener = <value optimized out>
#13 0x000000396dc28cfb in fr_event_loop (el=0x2535930) at 
/usr/src/debug/freeradius-server-3.0.10/src/lib/event.c:641
        ef = <value optimized out>
        i = <value optimized out>
        rcode = 1
        when = {tv_sec = 1447842529, tv_usec = 345425}
        wake = <value optimized out>
        maxfd = 31
        read_fds = {fds_bits = {4194304, 0 <repeats 15 times>}}
        master_fds = {fds_bits = {4294049680, 0 <repeats 15 times>}}
#14 0x0000000000427204 in main (argc=<value optimized out>, argv=<value 
optimized out>) at 
/usr/src/debug/freeradius-server-3.0.10/src/main/radiusd.c:579
        rcode = 0
        status = <value optimized out>
        argval = <value optimized out>
        spawn_flag = true
        display_version = false
        flag = 0
        from_child = {4, 5}
        state = 0x66bb40
        autofree = 0x2264060
A debugging session is active.
-- 
    .:.Sebastian Hagedorn - Weyertal 121 (Gebäude 133), Zimmer 2.02.:.
                 .:.Regionales Rechenzentrum (RRZK).:.
   .:.Universität zu Köln / Cologne University - ✆ +49-221-470-89578.:.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 191 bytes
Desc: not available
URL: <http://lists.freeradius.org/pipermail/freeradius-users/attachments/20151118/8f9b21a9/attachment.sig>