Crashes FreeRadius 3.0.11 using rlm_rest/libcurl + rlm_cache_rbtree
Hi all, We are frequently facing a crash of our freeradius server on live production environment (8 times in 2 months). The crash can't be reproduced on our test environment so we need help. Sometimes, we have a core dump (case 1- the server crashes), sometimes, we don't (case 2- the server freezes). Unfortunately, the core dumps don't provide useful informations: Here are examples of the core dumps : core 1: #0 0x00000034b680f5db in ?? () #1 0x00007fa718d066c6 in ?? () #2 0x0000000000000000 in ?? () core 2: #0 0x00000034b680f5db in ?? () #1 0x00007f169f94e6c6 in ?? () #2 0x0000000000000000 in ?? () Freeradius version : 3.0.11 Context : our server sends REST request to a http server to retrieve data which are then stored in a cache during 1 hour. The problem started when we added this feature. Leads : 1- The server logs all http requests that are sent. Somehow, the line log is truncated when the problem appears. Maybe, it is just a coincidence but we are suspecting this part. We use the rlm_rest module + libcurl to make our requests. Have you encountered a problem using those module/library ? We tried to upgrade libcurl from 7.50.3 to 7.54.0 but this doesn't fix the problem. 2- In the case 2 (the server freezes), we've got the following message in our log files: Mon Jun 26 22:00:31 2017 : Error: (10180900) Ignoring duplicate packet from client xxx port 21665 - ID: 164 due to unfinished request in component authorize module <our cache module> Our cache module is based on the rlm_cache_rbtree module: cache <cache name> { driver = "rlm_cache_rbtree" Have you encountered a problem using rlm_cache_rbtree module ? Thanks for your help. This message contains information that may be privileged or confidential and is the property of the Capgemini Group. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message.
On Jun 28, 2017, at 7:09 AM, RANJALAHY RASOLOFOMANANA, Bija <bija.ranjalahy@capgemini.com> wrote:
Hi all,
We are frequently facing a crash of our freeradius server on live production environment (8 times in 2 months). The crash can't be reproduced on our test environment so we need help. Sometimes, we have a core dump (case 1- the server crashes), sometimes, we don't (case 2- the server freezes). Unfortunately, the core dumps don't provide useful informations: Here are examples of the core dumps : core 1: #0 0x00000034b680f5db in ?? () #1 0x00007fa718d066c6 in ?? () #2 0x0000000000000000 in ?? ()
core 2: #0 0x00000034b680f5db in ?? () #1 0x00007f169f94e6c6 in ?? () #2 0x0000000000000000 in ?? ()
Freeradius version : 3.0.11 Context : our server sends REST request to a http server to retrieve data which are then stored in a cache during 1 hour. The problem started when we added this feature.
Leads : 1- The server logs all http requests that are sent. Somehow, the line log is truncated when the problem appears. Maybe, it is just a coincidence but we are suspecting this part. We use the rlm_rest module + libcurl to make our requests.
Have you encountered a problem using those module/library ? We tried to upgrade libcurl from 7.50.3 to 7.54.0 but this doesn't fix the problem.
2- In the case 2 (the server freezes), we've got the following message in our log files: Mon Jun 26 22:00:31 2017 : Error: (10180900) Ignoring duplicate packet from client xxx port 21665 - ID: 164 due to unfinished request in component authorize module <our cache module>
Our cache module is based on the rlm_cache_rbtree module: cache <cache name> { driver = "rlm_cache_rbtree"
Have you encountered a problem using rlm_cache_rbtree module ?
Sounds like a double locking issue in rlm_cache_rbtree. Upgrade to 3.0.14. If that doesn't work install the debuginfo packages for libcurl, and build with server --with-developer so the debug symbols don't get stripped. -Arran
Thanks for your help. Here is some update: - We upgraded the version of FreeRADIUS to 3.0.15 version. - We still have the 2 bugs (crash and freeze). - We deactivated the rlm_cache_rbtree module. So obviously, the problem isn't there. We built in developer mode. We have much information but still can't figure out where the problem comes from. Maybe because some debuginfo packages are still missing. We couldn't find the right versions of some debuginfo packages for redhat (json-c-debuginfo, libxml2-debuginfo...). We are still working on that core dumps messages but maybe someone has any idea that may help ? Anyone having issues using rlm_rest or libcurl ? Thanks ! -----Message d'origine----- De : Freeradius-Users [mailto:freeradius-users-bounces+bija.ranjalahy=capgemini.com@lists.freeradius.org] De la part de Arran Cudbard-Bell Envoyé : mercredi 28 juin 2017 15:06 À : FreeRadius users mailing list Objet : Re: Crashes FreeRadius 3.0.11 using rlm_rest/libcurl + rlm_cache_rbtree
On Jun 28, 2017, at 7:09 AM, RANJALAHY RASOLOFOMANANA, Bija <bija.ranjalahy@capgemini.com> wrote:
Hi all,
We are frequently facing a crash of our freeradius server on live production environment (8 times in 2 months). The crash can't be reproduced on our test environment so we need help. Sometimes, we have a core dump (case 1- the server crashes), sometimes, we don't (case 2- the server freezes). Unfortunately, the core dumps don't provide useful informations: Here are examples of the core dumps : core 1: #0 0x00000034b680f5db in ?? () #1 0x00007fa718d066c6 in ?? () #2 0x0000000000000000 in ?? ()
core 2: #0 0x00000034b680f5db in ?? () #1 0x00007f169f94e6c6 in ?? () #2 0x0000000000000000 in ?? ()
Freeradius version : 3.0.11 Context : our server sends REST request to a http server to retrieve data which are then stored in a cache during 1 hour. The problem started when we added this feature.
Leads : 1- The server logs all http requests that are sent. Somehow, the line log is truncated when the problem appears. Maybe, it is just a coincidence but we are suspecting this part. We use the rlm_rest module + libcurl to make our requests.
Have you encountered a problem using those module/library ? We tried to upgrade libcurl from 7.50.3 to 7.54.0 but this doesn't fix the problem.
2- In the case 2 (the server freezes), we've got the following message in our log files: Mon Jun 26 22:00:31 2017 : Error: (10180900) Ignoring duplicate packet from client xxx port 21665 - ID: 164 due to unfinished request in component authorize module <our cache module>
Our cache module is based on the rlm_cache_rbtree module: cache <cache name> { driver = "rlm_cache_rbtree"
Have you encountered a problem using rlm_cache_rbtree module ?
Sounds like a double locking issue in rlm_cache_rbtree. Upgrade to 3.0.14. If that doesn't work install the debuginfo packages for libcurl, and build with server --with-developer so the debug symbols don't get stripped. -Arran This message contains information that may be privileged or confidential and is the property of the Capgemini Group. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message.
On 22 Aug 2017, at 15:37, RANJALAHY RASOLOFOMANANA, Bija <bija.ranjalahy@capgemini.com> wrote:
Thanks for your help.
Here is some update:
- We upgraded the version of FreeRADIUS to 3.0.15 version. - We still have the 2 bugs (crash and freeze). - We deactivated the rlm_cache_rbtree module. So obviously, the problem isn't there.
We built in developer mode. We have much information but still can't figure out where the problem comes from. Maybe because some debuginfo packages are still missing. We couldn't find the right versions of some debuginfo packages for redhat (json-c-debuginfo, libxml2-debuginfo…).
We are still working on that core dumps messages but maybe someone has any idea that may help ? Anyone having issues using rlm_rest or libcurl ?
For the freeze attach using gdb once the process has frozen. Or just run it under gdb if it freezes consistently. Then use the command: thread apply all bt To get a backtrace from all the threads. For the crash, run under gdb and post the backtrace. -Arran
Hi Arran, 1/ For the crash, here is the backtrace : (gdb) bt full #0 0x00000034b680f5db in raise (sig=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/pt-raise.c:42 resultvar = 0 pid = <value optimized out> #1 0x00007f33611716c6 in skgesigOSCrash () from /opt/oracle/instantclient_11_2/libclntsh.so.11.1 No symbol table info available. #2 0x00007f3361422f79 in kpeDbgSignalHandler () from /opt/oracle/instantclient_11_2/libclntsh.so.11.1 No symbol table info available. #3 0x00007f33611718d6 in skgesig_sigactionHandler () from /opt/oracle/instantclient_11_2/libclntsh.so.11.1 No symbol table info available. #4 <signal handler called> No symbol table info available. #5 0x00000034b6032625 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 resultvar = 0 pid = <value optimized out> selftid = 6947 #6 0x00000034b6033e05 in abort () at abort.c:92 save_stage = 2 act = {__sigaction_handler = {sa_handler = 0x7f32cfa73f98, sa_sigaction = 0x7f32cfa73f98}, sa_mask = {__val = {139856208936832, 139858672564600, 16, 226393152297, 1, 226391833519, 5, 226393155952, 3, 139856208936830, 2, 226393152323, 1, 226393159054, 3, 139856208936838}}, sa_flags = 10, sa_restorer = 0x34b6157592} sigs = {__val = {32, 0 <repeats 15 times>}} #7 0x00000034b6070537 in __libc_message (do_abort=2, fmt=0x34b6158780 "*** glibc detected *** %s: %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:198 ap = {{gp_offset = 40, fp_offset = 48, overflow_arg_area = 0x7f32cfa74900, reg_save_area = 0x7f32cfa74810}} ap_copy = {{gp_offset = 16, fp_offset = 48, overflow_arg_area = 0x7f32cfa74900, reg_save_area = 0x7f32cfa74810}} fd = 2 on_2 = <value optimized out> list = <value optimized out> nlist = <value optimized out> cp = <value optimized out> written = <value optimized out> #8 0x00000034b6075e66 in malloc_printerr (action=3, str=0x34b6158aa8 "free(): invalid next size (fast)", ptr=<value optimized out>) at malloc.c:6336 buf = "00007f32740756a0" cp = <value optimized out> #9 0x00000034b60789b3 in _int_free (av=0x7f3274000020, p=0x7f3274075690, have_lock=0) at malloc.c:4832 size = <value optimized out> fb = <value optimized out> nextchunk = <value optimized out> nextsize = <value optimized out> nextinuse = <value optimized out> prevsize = <value optimized out> bck = <value optimized out> fwd = <value optimized out> errstr = <value optimized out> locked = <value optimized out> #10 0x00007f3359d519ad in ?? () No symbol table info available. #11 0x00007f32741628f0 in ?? () No symbol table info available. #12 0x00007f3359d515fb in ?? () No symbol table info available. #13 0x00007f3274162860 in ?? () No symbol table info available. #14 0x00007f3359d54e26 in ?? () No symbol table info available. #15 0x0000000000000000 in ?? () No symbol table info available. Here is what we got when trying to read the malloc chunk on which the abort is triggered : (gdb) f 9 #9 0x00000034b60789b3 in _int_free (av=0x7f3274000020, p=0x7f3274075690, have_lock=0) at malloc.c:4832 4832 malloc_printerr (check_action, errstr, chunk2mem(p)); (gdb) p *p $3 = {prev_size = 352, size = 52, fd = 0x7f3274075540, bk = 0x800000006d, fd_nextsize = 0x0, bk_nextsize = 0x0} (gdb) x/4x 0x7f3274075690 0x7f3274075690: 0x00000160 0x00000000 0x00000034 0x00000000 (gdb) x/14x 0x7f32740756a0 0x7f32740756a0: 0x74075540 0x00007f32 0x0000006d 0x00000080 0x7f32740756b0: 0x00000000 0x00000000 0x00000000 0x00000000 0x7f32740756c0: 0x00000000 0x00000000 0x35303933 0x00363438 0x7f32740756d0: 0x38373630 0x39393230 2/ For the freeze, we'll check if we can install gdb on the live production environment. Thanks again ! -----Message d'origine----- De : Freeradius-Users [mailto:freeradius-users-bounces+bija.ranjalahy=capgemini.com@lists.freeradius.org] De la part de Arran Cudbard-Bell Envoyé : mardi 22 août 2017 09:45 À : FreeRadius users mailing list Objet : Re: Crashes FreeRadius 3.0.11 using rlm_rest/libcurl + rlm_cache_rbtree
On 22 Aug 2017, at 15:37, RANJALAHY RASOLOFOMANANA, Bija <bija.ranjalahy@capgemini.com> wrote:
Thanks for your help.
Here is some update:
- We upgraded the version of FreeRADIUS to 3.0.15 version. - We still have the 2 bugs (crash and freeze). - We deactivated the rlm_cache_rbtree module. So obviously, the problem isn't there.
We built in developer mode. We have much information but still can't figure out where the problem comes from. Maybe because some debuginfo packages are still missing. We couldn't find the right versions of some debuginfo packages for redhat (json-c-debuginfo, libxml2-debuginfo…).
We are still working on that core dumps messages but maybe someone has any idea that may help ? Anyone having issues using rlm_rest or libcurl ?
For the freeze attach using gdb once the process has frozen. Or just run it under gdb if it freezes consistently. Then use the command: thread apply all bt To get a backtrace from all the threads. For the crash, run under gdb and post the backtrace. -Arran This message contains information that may be privileged or confidential and is the property of the Capgemini Group. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message.
On Aug 22, 2017, at 9:44 AM, RANJALAHY RASOLOFOMANANA, Bija <bija.ranjalahy@capgemini.com> wrote:
Hi Arran,
1/ For the crash, here is the backtrace :
Re-build the server with debugging symbols. See doc/bugs for details. A stack trace full of opaque hex values and no function names doesn't let us help you. Alan DeKok.
Thanks Alan. The backtrace that I sent yesterday comes from a server built with debugging symbols. Here is an extract of our makefile: # to build in developer mode: OPT_ENABLE_DEV = --enable-developer ./configure --prefix=$(FREERADIUS_HOME) \ --with-openssl=no \ --with-talloc-include-dir=$(TALLOC_HOME)/include \ --with-talloc-lib-dir=$(TALLOC_HOME)/lib \ --with-oracle-include-dir=$(ORACLE_HOME)/sdk/include \ --with-oracle-lib-dir=$(ORACLE_HOME) \ --with-libcurl=$(CURL_HOME) \ $(OPT_ENABLE_DEV) Is there something else we should do ? Thanks ! -----Message d'origine----- De : Freeradius-Users [mailto:freeradius-users-bounces+bija.ranjalahy=capgemini.com@lists.freeradius.org] De la part de Alan DeKok Envoyé : mercredi 23 août 2017 00:00 À : FreeRadius users mailing list Objet : Re: Crashes FreeRadius 3.0.11 using rlm_rest/libcurl + rlm_cache_rbtree On Aug 22, 2017, at 9:44 AM, RANJALAHY RASOLOFOMANANA, Bija <bija.ranjalahy@capgemini.com> wrote:
Hi Arran,
1/ For the crash, here is the backtrace :
Re-build the server with debugging symbols. See doc/bugs for details. A stack trace full of opaque hex values and no function names doesn't let us help you. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html This message contains information that may be privileged or confidential and is the property of the Capgemini Group. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message.
On Aug 23, 2017, at 4:45 AM, RANJALAHY RASOLOFOMANANA, Bija <bija.ranjalahy@capgemini.com> wrote:
The backtrace that I sent yesterday comes from a server built with debugging symbols.
If it had debugging symbols, the backtrace would show them. It didn't have debugging symbols.
Here is an extract of our makefile:
# to build in developer mode: OPT_ENABLE_DEV = --enable-developer
./configure --prefix=$(FREERADIUS_HOME) \ --with-openssl=no \ --with-talloc-include-dir=$(TALLOC_HOME)/include \ --with-talloc-lib-dir=$(TALLOC_HOME)/lib \ --with-oracle-include-dir=$(ORACLE_HOME)/sdk/include \ --with-oracle-lib-dir=$(ORACLE_HOME) \ --with-libcurl=$(CURL_HOME) \ $(OPT_ENABLE_DEV)
Is there something else we should do ?
That should work. But I have no idea what else you're doing. Following the doc/bugs link should work. Alan DeKok.
On 23 Aug 2017, at 16:45, RANJALAHY RASOLOFOMANANA, Bija <bija.ranjalahy@capgemini.com> wrote:
Thanks Alan.
The backtrace that I sent yesterday comes from a server built with debugging symbols. Here is an extract of our makefile:
# to build in developer mode: OPT_ENABLE_DEV = --enable-developer
./configure --prefix=$(FREERADIUS_HOME) \ --with-openssl=no \ --with-talloc-include-dir=$(TALLOC_HOME)/include \ --with-talloc-lib-dir=$(TALLOC_HOME)/lib \ --with-oracle-include-dir=$(ORACLE_HOME)/sdk/include \ --with-oracle-lib-dir=$(ORACLE_HOME) \ --with-libcurl=$(CURL_HOME) \ $(OPT_ENABLE_DEV)
Is there something else we should do ?
Look for output when gdb starts about it ignoring debugging symbols for some reason… -Arran
Hello, Thanks for all your suggestions on this. I'm working with Bija. We've finally found out what was going on. The issue is an overflow of allocated memory in rlm_rest module, triggering a server abort later on. See the issue I've opened on GitHub for the details (and a proposed fix): https://github.com/FreeRADIUS/freeradius-server/issues/2051 We had a CRC mismatch issue with our debuginfo packages, which is why gdb refused to use them. This is weird as these are the official debuginfo, matching the exact version of the libraries. Anyhow, we fixed this using objcopy, as follows (example for libpthread): # backup the library - it will be modified cp /lib64/libpthread-2.12.so /lib64/libpthread-2.12.so.ORIG # then modify library: remove .gnu_debuglink and add it again (with correct CRC) objcopy -R .gnu_debuglink --add-gnu-debuglink="/usr/lib/debug/lib64/libpthread-2.12.so.debug" /lib64/libpthread-2.12.so Regards, Nicolas.
Thanks for your help.
Here is some update:
- We upgraded the version of FreeRADIUS to 3.0.15 version. - We still have the 2 bugs (crash and freeze). - We deactivated the rlm_cache_rbtree module. So obviously, the problem isn't there.
We built in developer mode. We have much information but still can't figure out where the problem comes from. Maybe because some debuginfo packages are still missing. We couldn't find the right versions of some debuginfo packages for redhat (json-c-debuginfo, libxml2-debuginfo...).
We are still working on that core dumps messages but maybe someone has any idea that may help ? Anyone having issues using rlm_rest or libcurl ?
Thanks !
-----Message d'origine----- De : Freeradius-Users [mailto:freeradius-users-bounces+bija.ranjalahy=capgemini.com@lists.freeradius.org] De la part de Arran Cudbard-Bell Envoyé : mercredi 28 juin 2017 15:06 À : FreeRadius users mailing list Objet : Re: Crashes FreeRadius 3.0.11 using rlm_rest/libcurl + rlm_cache_rbtree
On Jun 28, 2017, at 7:09 AM, RANJALAHY RASOLOFOMANANA, Bija <bija.ranjalahy@capgemini.com> wrote:
Hi all,
We are frequently facing a crash of our freeradius server on live production environment (8 times in 2 months). The crash can't be reproduced on our test environment so we need help. Sometimes, we have a core dump (case 1- the server crashes), sometimes, we don't (case 2- the server freezes).
This message contains information that may be privileged or confidential and is the property of the Capgemini Group. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message.
On Sep 1, 2017, at 4:43 AM, Chaigneau, Nicolas <nicolas.chaigneau@capgemini.com> wrote:
We've finally found out what was going on. The issue is an overflow of allocated memory in rlm_rest module, triggering a server abort later on. See the issue I've opened on GitHub for the details (and a proposed fix):
Thanks. I've pushed a different patch which should fix the issue.
We had a CRC mismatch issue with our debuginfo packages, which is why gdb refused to use them. This is weird as these are the official debuginfo, matching the exact version of the libraries. Anyhow, we fixed this using objcopy, as follows (example for libpthread):
Weird... Alan DeKok.
first step is to try 3.0.14 as the bug/issue may have already been fixed in the past year. if it STILL exhibits issues, you need to read the bugs document , follow the steps in that to get relevant GDB callbacks etc. alan On 28 June 2017 at 12:09, RANJALAHY RASOLOFOMANANA, Bija <bija.ranjalahy@capgemini.com> wrote:
Hi all,
We are frequently facing a crash of our freeradius server on live production environment (8 times in 2 months). The crash can't be reproduced on our test environment so we need help. Sometimes, we have a core dump (case 1- the server crashes), sometimes, we don't (case 2- the server freezes). Unfortunately, the core dumps don't provide useful informations: Here are examples of the core dumps : core 1: #0 0x00000034b680f5db in ?? () #1 0x00007fa718d066c6 in ?? () #2 0x0000000000000000 in ?? ()
core 2: #0 0x00000034b680f5db in ?? () #1 0x00007f169f94e6c6 in ?? () #2 0x0000000000000000 in ?? ()
Freeradius version : 3.0.11 Context : our server sends REST request to a http server to retrieve data which are then stored in a cache during 1 hour. The problem started when we added this feature.
Leads : 1- The server logs all http requests that are sent. Somehow, the line log is truncated when the problem appears. Maybe, it is just a coincidence but we are suspecting this part. We use the rlm_rest module + libcurl to make our requests.
Have you encountered a problem using those module/library ? We tried to upgrade libcurl from 7.50.3 to 7.54.0 but this doesn't fix the problem.
2- In the case 2 (the server freezes), we've got the following message in our log files: Mon Jun 26 22:00:31 2017 : Error: (10180900) Ignoring duplicate packet from client xxx port 21665 - ID: 164 due to unfinished request in component authorize module <our cache module>
Our cache module is based on the rlm_cache_rbtree module: cache <cache name> { driver = "rlm_cache_rbtree"
Have you encountered a problem using rlm_cache_rbtree module ?
Thanks for your help. This message contains information that may be privileged or confidential and is the property of the Capgemini Group. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
participants (5)
-
Alan Buxey -
Alan DeKok -
Arran Cudbard-Bell -
Chaigneau, Nicolas -
RANJALAHY RASOLOFOMANANA, Bija