Crashes FreeRadius 3.0.11 using rlm_rest/libcurl + rlm_cache_rbtree

[RANJALAHY RASOLOFOMANANA, Bija]

Hi Arran,

1/ For the crash, here is the backtrace :

(gdb) bt full
#0  0x00000034b680f5db in raise (sig=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/pt-raise.c:42
        resultvar = 0
        pid = <value optimized out>
#1  0x00007f33611716c6 in skgesigOSCrash () from /opt/oracle/instantclient_11_2/libclntsh.so.11.1
No symbol table info available.
#2  0x00007f3361422f79 in kpeDbgSignalHandler () from /opt/oracle/instantclient_11_2/libclntsh.so.11.1
No symbol table info available.
#3  0x00007f33611718d6 in skgesig_sigactionHandler () from /opt/oracle/instantclient_11_2/libclntsh.so.11.1
No symbol table info available.
#4  <signal handler called>
No symbol table info available.
#5  0x00000034b6032625 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
        resultvar = 0
        pid = <value optimized out>
        selftid = 6947
#6  0x00000034b6033e05 in abort () at abort.c:92
        save_stage = 2
        act = {__sigaction_handler = {sa_handler = 0x7f32cfa73f98, sa_sigaction = 0x7f32cfa73f98}, sa_mask = {__val = {139856208936832, 139858672564600, 16,
              226393152297, 1, 226391833519, 5, 226393155952, 3, 139856208936830, 2, 226393152323, 1, 226393159054, 3, 139856208936838}}, sa_flags = 10,
          sa_restorer = 0x34b6157592}
        sigs = {__val = {32, 0 <repeats 15 times>}}
#7  0x00000034b6070537 in __libc_message (do_abort=2, fmt=0x34b6158780 "*** glibc detected *** %s: %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:198
        ap = {{gp_offset = 40, fp_offset = 48, overflow_arg_area = 0x7f32cfa74900, reg_save_area = 0x7f32cfa74810}}
        ap_copy = {{gp_offset = 16, fp_offset = 48, overflow_arg_area = 0x7f32cfa74900, reg_save_area = 0x7f32cfa74810}}
        fd = 2
        on_2 = <value optimized out>
        list = <value optimized out>
        nlist = <value optimized out>
        cp = <value optimized out>
        written = <value optimized out>
#8  0x00000034b6075e66 in malloc_printerr (action=3, str=0x34b6158aa8 "free(): invalid next size (fast)", ptr=<value optimized out>) at malloc.c:6336
        buf = "00007f32740756a0"
        cp = <value optimized out>
#9  0x00000034b60789b3 in _int_free (av=0x7f3274000020, p=0x7f3274075690, have_lock=0) at malloc.c:4832
        size = <value optimized out>
        fb = <value optimized out>
        nextchunk = <value optimized out>
        nextsize = <value optimized out>
        nextinuse = <value optimized out>
        prevsize = <value optimized out>
        bck = <value optimized out>
        fwd = <value optimized out>
        errstr = <value optimized out>
        locked = <value optimized out>
#10 0x00007f3359d519ad in ?? ()
No symbol table info available.
#11 0x00007f32741628f0 in ?? ()
No symbol table info available.
#12 0x00007f3359d515fb in ?? ()
No symbol table info available.
#13 0x00007f3274162860 in ?? ()
No symbol table info available.
#14 0x00007f3359d54e26 in ?? ()
No symbol table info available.
#15 0x0000000000000000 in ?? ()
No symbol table info available.



Here is what we got when trying to read the malloc chunk on which the abort is triggered :

(gdb) f 9
#9  0x00000034b60789b3 in _int_free (av=0x7f3274000020, p=0x7f3274075690, have_lock=0) at malloc.c:4832
4832          malloc_printerr (check_action, errstr, chunk2mem(p));
(gdb) p *p
$3 = {prev_size = 352, size = 52, fd = 0x7f3274075540, bk = 0x800000006d, fd_nextsize = 0x0, bk_nextsize = 0x0}
(gdb) x/4x 0x7f3274075690
0x7f3274075690: 0x00000160      0x00000000      0x00000034      0x00000000
(gdb) x/14x 0x7f32740756a0
0x7f32740756a0: 0x74075540      0x00007f32      0x0000006d      0x00000080
0x7f32740756b0: 0x00000000      0x00000000      0x00000000      0x00000000
0x7f32740756c0: 0x00000000      0x00000000      0x35303933      0x00363438
0x7f32740756d0: 0x38373630      0x39393230



2/ For the freeze, we'll check if we can install gdb on the live production environment.

Thanks again !


-----Message d'origine-----
De : Freeradius-Users [mailto:freeradius-users-bounces+bija.ranjalahy=capgemini.com at lists.freeradius.org] De la part de Arran Cudbard-Bell
Envoyé : mardi 22 août 2017 09:45
À : FreeRadius users mailing list
Objet : Re: Crashes FreeRadius 3.0.11 using rlm_rest/libcurl + rlm_cache_rbtree


> On 22 Aug 2017, at 15:37, RANJALAHY RASOLOFOMANANA, Bija <bija.ranjalahy at capgemini.com> wrote:
> 
> Thanks for your help.
> 
> Here is some update:
> 
> - We upgraded the version of FreeRADIUS to 3.0.15 version.
> - We still have the 2 bugs (crash and freeze).
> - We deactivated the rlm_cache_rbtree module. So obviously, the problem isn't there.
> 
> We built in developer mode. We have much information but still can't figure out where the problem comes from.
> Maybe because some debuginfo packages are still missing. We couldn't find the right versions of some debuginfo packages for redhat (json-c-debuginfo, libxml2-debuginfo…).
> 

> We are still working on that core dumps messages but maybe someone has any idea that may help ?
> Anyone having issues using rlm_rest or libcurl ?

For the freeze attach using gdb once the process has frozen.  Or just run it under gdb if it freezes consistently.

Then use the command:

	thread apply all bt

To get a backtrace from all the threads.

For the crash, run under gdb and post the backtrace.

-Arran


This message contains information that may be privileged or confidential and is the property of the Capgemini Group. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message.