Freeradius crashes with SIGABRT
Hi, our radius proxy crashes regularily since we switched from version 2 to 3. First we ran freeradius version 3.0.15 (package from Suse Linux Enterprise 12.4), then version 3.0.17 and 3.0.18 (compiled with rpmbuild) with no difference. Our present configuration is like this: radius@radius-dev1:~> uname -a Linux radius-dev1 4.12.14-95.6-default #1 SMP Thu Jan 17 06:04:39 UTC 2019 (6af4ef8) x86_64 x86_64 x86_64 GNU/Linux radius@radius-dev1:~> /usr/sbin/radiusd -v radiusd: FreeRADIUS Version 3.0.18, for host x86_64-suse-linux-gnu, built on Mar 4 2019 at 00:00 FreeRADIUS Version 3.0.18 I run the radius-daemon inside of gdb with dumps enabled: Starting program: /usr/sbin/radiusd -f -d /etc/raddb ... *** Error in `/usr/sbin/radiusd': double free or corruption (fasttop): 0x00007fffe8009b40 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x7411f)[0x7ffff60ba11f] /lib64/libc.so.6(+0x79596)[0x7ffff60bf596] /lib64/libc.so.6(+0x7a3db)[0x7ffff60c03db] /usr/lib64/libtalloc.so.2(+0x300d)[0x7ffff6e7a00d] /usr/lib64/libtalloc.so.2(+0x2d87)[0x7ffff6e79d87] /usr/lib64/libtalloc.so.2(+0x2d87)[0x7ffff6e79d87] /usr/sbin/radiusd(+0x37ae7)[0x55555558bae7] /usr/sbin/radiusd(+0x3a14d)[0x55555558e14d] /usr/lib64/freeradius/libfreeradius-radius.so(fr_event_run+0x74)[0x7ffff79928d4] /usr/lib64/freeradius/libfreeradius-radius.so(fr_event_loop+0x2c9)[0x7ffff7992ec9] /usr/sbin/radiusd(main+0x706)[0x555555569f06] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7ffff6066725] /usr/sbin/radiusd(_start+0x29)[0x55555556a239] Thread 1 "radiusd" received signal SIGABRT, Aborted. 0x00007ffff607af97 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55 55 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) (gdb) bt #0 0x00007ffff607af97 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55 #1 0x00007ffff607c36a in __GI_abort () at abort.c:78 #2 0x00007ffff60ba124 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7ffff61b3f48 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175 #3 0x00007ffff60bf596 in malloc_printerr (action=3, str=0x7ffff61b40a0 "double free or corruption (fasttop)", ptr=<optimized out>, ar_ptr=<optimized out>) at malloc.c:5018 #4 0x00007ffff60c03db in _int_free (av=0x7fffe8000020, p=<optimized out>, have_lock=0) at malloc.c:3872 #5 0x00007ffff6e7a00d in _tc_free_internal (tc=0x7fffe8009b40, location=0x5555555b02ea "src/main/process.c:604") at ../talloc.c:1148 #6 0x00007ffff6e79d87 in _tc_free_children_internal (location=0x5555555b02ea "src/main/process.c:604", ptr=0x7fffe801cb80, tc=0x7fffe801cb20) at ../talloc.c:1593 #7 _tc_free_internal (location=0x5555555b02ea "src/main/process.c:604", tc=0x7fffe801cb20) at ../talloc.c:1104 #8 _tc_free_children_internal (location=0x5555555b02ea "src/main/process.c:604", ptr=0x555555c80a60, tc=0x555555c80a00) at ../talloc.c:1593 #9 _tc_free_internal (tc=0x555555c80a00, location=0x5555555b02ea "src/main/process.c:604") at ../talloc.c:1104 #10 0x00007ffff6e79d87 in _tc_free_children_internal (location=0x5555555b02ea "src/main/process.c:604", ptr=0x555555c808b0, tc=0x555555c80850) at ../talloc.c:1593 #11 _tc_free_internal (location=0x5555555b02ea "src/main/process.c:604", tc=0x555555c80850) at ../talloc.c:1104 #12 _tc_free_children_internal (location=0x5555555b02ea "src/main/process.c:604", ptr=0x555555c80690, tc=0x555555c80630) at ../talloc.c:1593 #13 _tc_free_internal (tc=0x555555c80630, location=0x5555555b02ea "src/main/process.c:604") at ../talloc.c:1104 #14 0x000055555558bae7 in request_done (request=request@entry=0x555555c808b0, action=<optimized out>, action@entry=2) at src/main/process.c:899 #15 0x000055555558e14d in request_cleanup_delay (request=0x555555c808b0, action=<optimized out>) at src/main/process.c:1212 #16 0x00007ffff79928d4 in fr_event_run (el=el@entry=0x555555bd60b0, when=when@entry=0x7fffffffe350) at src/lib/event.c:309 #17 0x00007ffff7992ec9 in fr_event_loop (el=0x555555bd60b0) at src/lib/event.c:632 #18 0x0000555555569f06 in main (argc=<optimized out>, argv=<optimized out>) at src/main/radiusd.c:611 The crash occures always within a few hours, sometimes soon after startup. Version 2.x with a similar setup runs stable. I didn't provide the output of -X because it contains sensible data, but I could provide cleaned logs and configs, if needed. Thanks and greetings, Daniel
On Mar 6, 2019, at 8:40 AM, Daniel Feuchtinger <daniel.feuchtinger@lrz.de> wrote:
our radius proxy crashes regularily since we switched from version 2 to 3. First we ran freeradius version 3.0.15 (package from Suse Linux Enterprise 12.4), then version 3.0.17 and 3.0.18 (compiled with rpmbuild) with no difference.
Hmm... I haven't seen that before.
The crash occures always within a few hours, sometimes soon after startup.
Is there any way to narrow down the configuration to something simple? That way I may be able to reproduce it. Also, what are you doing? It must be something unusual, because I haven't seen this before. The basic functionality is pretty well tested, and doesn't have this issue. Alan DeKok.
Am 06.03.19 um 15:14 schrieb Alan DeKok:
On Mar 6, 2019, at 8:40 AM, Daniel Feuchtinger <daniel.feuchtinger@lrz.de> wrote:
The crash occures always within a few hours, sometimes soon after startup.
Is there any way to narrow down the configuration to something simple? That way I may be able to reproduce it. That could be difficult, I try to figure something out.
Also, what are you doing? It must be something unusual, because I haven't seen this before. The basic functionality is pretty well tested, and doesn't have this issue.
We are proxying a lot of radius messages on this server: grep home_server proxy.conf|wc -l 101 grep client clients.conf|wc -l 131 Most of the traffic should be eduroam authentication. The server does load balancing with 3 servers that do the real authentication for us (and crash too from time to time, but that's another story), talks to a radsecproxy for external eduroam users and a ton of small other radius servers owned by university institutes. I'll check, if I'm able to reduce the configuration without impact on the users. Daniel
Daniel, If you do not mind, please register that crash in https://github.com/FreeRADIUS/freeradius-server/issues/new following the placeholders. then, it can be useful share with us the output of "freeradius -XC" -- Jorge Pereira On Wed, Mar 6, 2019 at 12:52 PM Daniel Feuchtinger <daniel.feuchtinger@lrz.de> wrote:
Am 06.03.19 um 15:14 schrieb Alan DeKok:
On Mar 6, 2019, at 8:40 AM, Daniel Feuchtinger <daniel.feuchtinger@lrz.de> wrote:
The crash occures always within a few hours, sometimes soon after startup.
Is there any way to narrow down the configuration to something simple? That way I may be able to reproduce it. That could be difficult, I try to figure something out.
Also, what are you doing? It must be something unusual, because I haven't seen this before. The basic functionality is pretty well tested, and doesn't have this issue.
We are proxying a lot of radius messages on this server: grep home_server proxy.conf|wc -l 101 grep client clients.conf|wc -l 131
Most of the traffic should be eduroam authentication. The server does load balancing with 3 servers that do the real authentication for us (and crash too from time to time, but that's another story), talks to a radsecproxy for external eduroam users and a ton of small other radius servers owned by university institutes.
I'll check, if I'm able to reduce the configuration without impact on the users.
Daniel
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On Mar 6, 2019, at 10:52 AM, Daniel Feuchtinger <daniel.feuchtinger@lrz.de> wrote:
We are proxying a lot of radius messages on this server: grep home_server proxy.conf|wc -l 101 grep client clients.conf|wc -l 131
That shouldn't matter. The clients and home servers are placed into data structure, and never modified. Are you executing other programs? Sending CoA packets?
Most of the traffic should be eduroam authentication. The server does load balancing with 3 servers that do the real authentication for us (and crash too from time to time, but that's another story), talks to a radsecproxy for external eduroam users and a ton of small other radius servers owned by university institutes.
I'll check, if I'm able to reduce the configuration without impact on the users.
Another alternative is to use LLVM. If you install the LLVM RPM, you can then build FreeRADIUS with some more options. 1) install LLVM 2) grab the 3.0.18 source (or the source from GitHub) 3) do: CC=clang ./configure --prefix=/opt/freeradius ... args ... I think the name of the compiler on Suse is "clang". If not, you'll have to find that out You're best to put this build into a unique directory, so it's easy to get rid of it later. 4) Edit "Make.inc", and look for the line "CFLAGS = ..." add this: -fsanitize=address -fno-omit-frame-pointer -fno-optimize-sibling-calls -fsanitize-address-use-after-scope 5) run "make; make install" You can then run that version with some extra environment variables: export ASAN_SYMBOLIZER_PATH="/usr/local/opt/llvm/bin/llvm-symbolizer" You'll have to find out where that executable is located... export ASAN_OPTIONS="malloc_context_size=50 detect_leaks=1 symbolize=1" Then run FreeRADIUS: /usr/freeradius/sbin/radiusd -f -d /etc/raddb With some luck, the address sanitizer may find the issue. Note that the server will run much slower than normal, but that's likely fine. These issues are very, very, difficult to track down without tools like the address sanitizer. We're using that in the v4 / master branch, and it's a huge help. Alan DeKok.
On 3/6/19 11:17 PM, Alan DeKok wrote:
Another alternative is to use LLVM. If you install the LLVM RPM, you can then build FreeRADIUS with some more options.
1) install LLVM 2) grab the 3.0.18 source (or the source from GitHub) 3) do: CC=clang ./configure --prefix=/opt/freeradius ... args ...
I think the name of the compiler on Suse is "clang". If not, you'll have to find that out
I don't believe clang actually is part of SLES-12. clang is in openSUSE distributions only. One could also try using valgrind and with origin tracking, it should also indicate where the pointer was freed. valgrind --track-origins=yes /usr/sbin/radiusd -f -d /etc/raddb Valgrind is available for SLE-12, and latest valgrind (just 1 RPM) can also be downloaded from https://software.opensuse.org//download.html?project=devel%3Atools&package=v... - Adam
I changed to a debian test machine with stretch running version 3.0.18, compiled with clang (more on that in a minute). The crashes look the same to me. Am 07.03.19 um 15:48 schrieb Adam Majer:
On 3/6/19 11:17 PM, Alan DeKok wrote:
One could also try using valgrind and with origin tracking, it should also indicate where the pointer was freed.
valgrind --track-origins=yes /usr/sbin/radiusd -f -d /etc/raddb
That gives a lot of errors without any crash, starting with: ==8960== Memcheck, a memory error detector ==8960== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al. ==8960== Using Valgrind-3.12.0.SVN and LibVEX; rerun with -h for copyright info ==8960== Command: /usr/sbin/freeradius -f -d /etc/freeradius/3.0/ ==8960== ==8961== ==8961== HEAP SUMMARY: ==8961== in use at exit: 314 bytes in 3 blocks ==8961== total heap usage: 5 allocs, 2 frees, 466 bytes allocated ==8961== ==8961== LEAK SUMMARY: ==8961== definitely lost: 0 bytes in 0 blocks ==8961== indirectly lost: 0 bytes in 0 blocks ==8961== possibly lost: 314 bytes in 3 blocks ==8961== still reachable: 0 bytes in 0 blocks ==8961== suppressed: 0 bytes in 0 blocks ==8961== Rerun with --leak-check=full to see details of leaked memory ==8961== ==8961== For counts of detected and suppressed errors, rerun with: -v ==8961== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) ==8960== Thread 4: ==8960== Conditional jump or move depends on uninitialised value(s) ==8960== at 0x403908D: ??? ==8960== by 0xD3E80EF: ??? ==8960== Uninitialised value was created by a heap allocation ==8960== at 0x4C2BBAF: malloc (vg_replace_malloc.c:299) ==8960== by 0x5BBBF9D: talloc_pool (in /usr/lib/x86_64-linux-gnu/libtalloc.so.2.1.8) ==8960== by 0x41B630: auth_socket_recv (listen.c:1553) ==8960== by 0x436DBD: event_socket_handler (process.c:4850) ==8960== by 0x5092647: fr_event_loop (in /usr/lib/freeradius/libfreeradius-radius.so) ==8960== by 0x427643: main (radiusd.c:596) ==8960== ==8960== Conditional jump or move depends on uninitialised value(s) ==8960== at 0x4048D0A: ??? ==8960== by 0xD3E80EF: ??? ==8960== Uninitialised value was created by a heap allocation ==8960== at 0x4C2BBAF: malloc (vg_replace_malloc.c:299) ==8960== by 0x5BBBF9D: talloc_pool (in /usr/lib/x86_64-linux-gnu/libtalloc.so.2.1.8) ==8960== by 0x41B630: auth_socket_recv (listen.c:1553) ==8960== by 0x436DBD: event_socket_handler (process.c:4850) ==8960== by 0x5092647: fr_event_loop (in /usr/lib/freeradius/libfreeradius-radius.so) ==8960== by 0x427643: main (radiusd.c:596) and ending with ==8960== Thread 4: ==8960== Conditional jump or move depends on uninitialised value(s) ==8960== at 0x40390AF: ??? ==8960== by 0x15EA0F2F: ??? ==8960== Uninitialised value was created by a heap allocation ==8960== at 0x4C2BBAF: malloc (vg_replace_malloc.c:299) ==8960== by 0x5BBBF9D: talloc_pool (in /usr/lib/x86_64-linux-gnu/libtalloc.so.2.1.8) ==8960== by 0x41B630: auth_socket_recv (listen.c:1553) ==8960== by 0x436DBD: event_socket_handler (process.c:4850) ==8960== by 0x5092647: fr_event_loop (in /usr/lib/freeradius/libfreeradius-radius.so) ==8960== by 0x427643: main (radiusd.c:596) ==8960== ==8960== Conditional jump or move depends on uninitialised value(s) ==8960== at 0x4048725: ??? ==8960== by 0x15EA0F2F: ??? ==8960== Uninitialised value was created by a heap allocation ==8960== at 0x4C2BBAF: malloc (vg_replace_malloc.c:299) ==8960== by 0x5BBBF9D: talloc_pool (in /usr/lib/x86_64-linux-gnu/libtalloc.so.2.1.8) ==8960== by 0x41B630: auth_socket_recv (listen.c:1553) ==8960== by 0x436DBD: event_socket_handler (process.c:4850) ==8960== by 0x5092647: fr_event_loop (in /usr/lib/freeradius/libfreeradius-radius.so) ==8960== by 0x427643: main (radiusd.c:596) ==8960== ==8960== Conditional jump or move depends on uninitialised value(s) ==8960== at 0x404844C: ??? ==8960== by 0x15EA0F2F: ??? ==8960== Uninitialised value was created by a heap allocation ==8960== at 0x4C2BBAF: malloc (vg_replace_malloc.c:299) ==8960== by 0x5BBBF9D: talloc_pool (in /usr/lib/x86_64-linux-gnu/libtalloc.so.2.1.8) ==8960== by 0x41B630: auth_socket_recv (listen.c:1553) ==8960== by 0x436DBD: event_socket_handler (process.c:4850) ==8960== by 0x5092647: fr_event_loop (in /usr/lib/freeradius/libfreeradius-radius.so) ==8960== by 0x427643: main (radiusd.c:596) ==8960== ==8960== ==8960== More than 1000 different errors detected. I'm not reporting any more. ==8960== Final error counts will be inaccurate. Go fix your program! ==8960== Rerun with --error-limit=no to disable this cutoff. Note ==8960== that errors may occur in your program without prior warning from ==8960== Valgrind, because errors are no longer being displayed. ==8960== freeradius still running fine at this point. Is it worth to investigate those errors? Daniel
On Mar 15, 2019, at 10:41 AM, Daniel Feuchtinger <daniel.feuchtinger@lrz.de> wrote:
I changed to a debian test machine with stretch running version 3.0.18, compiled with clang (more on that in a minute). The crashes look the same to me.
:(
valgrind --track-origins=yes /usr/sbin/radiusd -f -d /etc/raddb
That gives a lot of errors without any crash, starting with: ... freeradius still running fine at this point. Is it worth to investigate those errors?
No. Those are buried inside of malloc(), and have nothing to do with us. Alan DeKok.
Am 06.03.19 um 23:17 schrieb Alan DeKok:
Are you executing other programs? Sending CoA packets? Yes, some perl scripts. No CoA as far as I know.
Another alternative is to use LLVM. If you install the LLVM RPM, you can then build FreeRADIUS with some more options. I changed the setup to debian because llvm packages and dbg-Repos are more accessible.
1) install LLVM 2) grab the 3.0.18 source (or the source from GitHub) 3) do: CC=clang ./configure --prefix=/opt/freeradius ... args ...
We did that and succeeded so far without the CFLAGS. This version crashes too.
4) Edit "Make.inc", and look for the line "CFLAGS = ..." add this:
-fsanitize=address -fno-omit-frame-pointer -fno-optimize-sibling-calls -fsanitize-address-use-after-scope We get compile errors with those flags in some modules
using Debian Stretch and Debian Buster (ippool, rlm_mschap, rml_eap, sorry for the verbosity, it's just rlm_mschap): /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbencrypt.o: in function `main': ./src/modules/rlm_mschap/smbencrypt.c:63: undefined reference to `__asan_option_detect_stack_use_after_return' /usr/bin/ld: ./src/modules/rlm_mschap/smbencrypt.c:63: undefined reference to `__asan_stack_malloc_5' /usr/bin/ld: ./src/modules/rlm_mschap/smbencrypt.c:63: undefined reference to `__asan_set_shadow_f8' /usr/bin/ld: ./src/modules/rlm_mschap/smbencrypt.c:63: undefined reference to `__asan_set_shadow_f8' /usr/bin/ld: ./src/modules/rlm_mschap/smbencrypt.c:65: undefined reference to `__asan_set_shadow_00' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbencrypt.o: in function `ntpwdhash': ./src/modules/rlm_mschap/smbencrypt.c:52: undefined reference to `__asan_set_shadow_00' /usr/bin/ld: ./src/modules/rlm_mschap/smbencrypt.c:60: undefined reference to `__asan_set_shadow_f8' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbencrypt.o: in function `main': ./src/modules/rlm_mschap/smbencrypt.c:76: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbencrypt.c:84: undefined reference to `__asan_set_shadow_f8' /usr/bin/ld: ./src/modules/rlm_mschap/smbencrypt.c:83: undefined reference to `__asan_stack_free_5' /usr/bin/ld: ./src/modules/rlm_mschap/smbencrypt.c:83: undefined reference to `__asan_set_shadow_00' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbencrypt.o: in function `tohex': ./src/modules/rlm_mschap/smbencrypt.c:43: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbencrypt.c:43: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbencrypt.c:43: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbencrypt.c:44: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbencrypt.c:44: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbencrypt.c:43: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbencrypt.c:43: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbencrypt.c:43: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbencrypt.c:44: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbencrypt.c:44: undefined reference to `__asan_report_store1' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbencrypt.o: in function `main': ./src/modules/rlm_mschap/smbencrypt.c:74: undefined reference to `__asan_report_load8' /usr/bin/ld: ./src/modules/rlm_mschap/smbencrypt.c:70: undefined reference to `__asan_report_load8' /usr/bin/ld: ./src/modules/rlm_mschap/smbencrypt.c:71: undefined reference to `__asan_report_load8' /usr/bin/ld: ./src/modules/rlm_mschap/smbencrypt.c:72: undefined reference to `__asan_report_load8' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbencrypt.o: in function `tohex': ./src/modules/rlm_mschap/smbencrypt.c:46: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbencrypt.c:46: undefined reference to `__asan_report_store1' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbencrypt.o: in function `main': ./src/modules/rlm_mschap/smbencrypt.c:76: undefined reference to `__asan_report_load1' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbencrypt.o: in function `ntpwdhash': ./src/modules/rlm_mschap/smbencrypt.c:56: undefined reference to `__asan_report_store1' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbencrypt.o: in function `asan.module_ctor': ./src/modules/rlm_mschap/smbencrypt.c:(.text+0x742): undefined reference to `__asan_init' /usr/bin/ld: ./src/modules/rlm_mschap/smbencrypt.c:(.text+0x747): undefined reference to `__asan_version_mismatch_check_v8' /usr/bin/ld: ./src/modules/rlm_mschap/smbencrypt.c:(.text+0x758): undefined reference to `__asan_register_globals' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbencrypt.o: in function `asan.module_dtor': ./src/modules/rlm_mschap/smbencrypt.c:(.text+0x76e): undefined reference to `__asan_unregister_globals' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `smbhash': ./src/modules/rlm_mschap/smbdes.c:287: undefined reference to `__asan_option_detect_stack_use_after_return' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:287: undefined reference to `__asan_stack_malloc_6' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:287: undefined reference to `__asan_set_shadow_f8' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:299: undefined reference to `__asan_memset' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:298: undefined reference to `__asan_report_store1' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `dohash': ./src/modules/rlm_mschap/smbdes.c:196: undefined reference to `__asan_set_shadow_00' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `permute': ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_store1' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `dohash': ./src/modules/rlm_mschap/smbdes.c:204: undefined reference to `__asan_memcpy' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:206: undefined reference to `__asan_memcpy' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `lshift': ./src/modules/rlm_mschap/smbdes.c:171: undefined reference to `__asan_memcpy' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:171: undefined reference to `__asan_memcpy' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `concat': ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_store1' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `permute': ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_store1' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `dohash': ./src/modules/rlm_mschap/smbdes.c:219: undefined reference to `__asan_memcpy' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:220: undefined reference to `__asan_memcpy' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:237: undefined reference to `__asan_memcpy' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:237: undefined reference to `__asan_memcpy' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:237: undefined reference to `__asan_memcpy' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o:./src/modules/rlm_mschap/smbdes.c:237: more undefined references to `__asan_memcpy' follow /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `dohash': ./src/modules/rlm_mschap/smbdes.c:251: undefined reference to `__asan_report_store4' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `permute': ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_store1' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `dohash': ./src/modules/rlm_mschap/smbdes.c:266: undefined reference to `__asan_set_shadow_f8' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `smbhash': ./src/modules/rlm_mschap/smbdes.c:310: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:312: undefined reference to `__asan_stack_free_6' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:312: undefined reference to `__asan_set_shadow_00' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `str_to_key': ./src/modules/rlm_mschap/smbdes.c:272: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:273: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:274: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:275: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:276: undefined reference to `__asan_report_load1' LINK build/lib/rlm_expr.la /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o:./src/modules/rlm_mschap/smbdes.c:277: more undefined references to `__asan_report_load1' follow /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `str_to_key': ./src/modules/rlm_mschap/smbdes.c:281: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:281: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:281: undefined reference to `__asan_report_store1' /usr/bin/ldbuild/make/jlibtool --silent --mode=link clang -o build/lib/rlm_expr.la -rpath /usr/lib/freeradius -Wl,-z,relro -Wl,-z,now build/objs/src/modules/rlm_expr/rlm_expr.lo build/objs/src/modules/rlm_expr/paircmp.lo : ./src/modules/rlm_mschap/smbdes.c:281: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:281: undefined reference to `__asan_report_store1' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o:./src/modules/rlm_mschap/smbdes.c:281: more undefined references to `__asan_report_store1' follow /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `smbhash': ./src/modules/rlm_mschap/smbdes.c:305: undefined reference to `__asan_report_store_n' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:305: undefined reference to `__asan_report_store_n' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `lshift': ./src/modules/rlm_mschap/smbdes.c:169: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:169: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:169: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:169: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:169: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:169: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:169: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:169: undefined reference to `__asan_report_store1' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `permute': ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_store1' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `dohash': ./src/modules/rlm_mschap/smbdes.c:241: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:241: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:243: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:243: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:243: undefined reference to `__asan_report_load1' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o:./src/modules/rlm_mschap/smbdes.c:(.text+0x357e): more undefined references to `__asan_report_load1' follow /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `dohash': ./src/modules/rlm_mschap/smbdes.c:246: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:246: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:246: undefined reference to `__asan_report_store1' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `permute': ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_store1' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `dohash': ./src/modules/rlm_mschap/smbdes.c:243: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:246: undefined reference to `__asan_report_store1' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `permute': ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_store1' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `concat': ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_load16' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_store16' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_store16' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `xor': ./src/modules/rlm_mschap/smbdes.c:186: undefined reference to `__asan_report_load16' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:186: undefined reference to `__asan_report_load16' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:186: undefined reference to `__asan_report_load16' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:186: undefined reference to `__asan_report_load16' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:186: undefined reference to `__asan_report_load16' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:186: undefined reference to `__asan_report_store16' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:186: undefined reference to `__asan_report_load16' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:186: undefined reference to `__asan_report_load16' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:186: undefined reference to `__asan_report_load16' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:186: undefined reference to `__asan_report_store16' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:186: undefined reference to `__asan_report_load16' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:186: undefined reference to `__asan_report_store16' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:186: undefined reference to `__asan_report_store16' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:186: undefined reference to `__asan_report_load16' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:186: undefined reference to `__asan_report_store16' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `smbhash': ./src/modules/rlm_mschap/smbdes.c:297: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:297: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:298: undefined reference to `__asan_report_load1' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `permute': ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `dohash': ./src/modules/rlm_mschap/smbdes.c:209: undefined reference to `__asan_report_load1' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `concat': ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_load_n' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_load_n' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_load1' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `permute': ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `dohash': ./src/modules/rlm_mschap/smbdes.c:251: undefined reference to `__asan_report_store4' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:251: undefined reference to `__asan_report_load_n' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:251: undefined reference to `__asan_report_store4' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:251: undefined reference to `__asan_report_store4' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:251: undefined reference to `__asan_report_load_n' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:251: undefined reference to `__asan_report_store4' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:251: undefined reference to `__asan_report_load4' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:251: undefined reference to `__asan_report_store4' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:251: undefined reference to `__asan_report_load_n' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:251: undefined reference to `__asan_report_load_n' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:251: undefined reference to `__asan_report_store4' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:251: undefined reference to `__asan_report_load4' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:251: undefined reference to `__asan_report_store4' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:251: undefined reference to `__asan_report_load_n' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:251: undefined reference to `__asan_report_load_n' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:251: undefined reference to `__asan_report_load4' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:251: undefined reference to `__asan_report_load_n' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:251: undefined reference to `__asan_report_load4' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:251: undefined reference to `__asan_report_load_n' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `permute': ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_store1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:161: undefined reference to `__asan_report_load1' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `smbhash': ./src/modules/rlm_mschap/smbdes.c:309: undefined reference to `__asan_report_load1' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `concat': ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_load16' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_store16' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_load16' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:177: undefined reference to `__asan_report_store16' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_load16' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_store16' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_load16' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:179: undefined reference to `__asan_report_store16' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `smbdes_lmpwdhash': ./src/modules/rlm_mschap/smbdes.c:319: undefined reference to `__asan_option_detect_stack_use_after_return' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:319: undefined reference to `__asan_stack_malloc_0' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:324: undefined reference to `__asan_memset' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `toupper': /usr/include/ctype.h:215: undefined reference to `__asan_report_load4' /usr/bin/ld: /usr/include/ctype.h:215: undefined reference to `__asan_report_load8' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `smbdes_lmpwdhash': ./src/modules/rlm_mschap/smbdes.c:325: undefined reference to `__asan_report_load1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:326: undefined reference to `__asan_report_store1' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `smbdes_mschap': ./src/modules/rlm_mschap/smbdes.c:340: undefined reference to `__asan_option_detect_stack_use_after_return' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:340: undefined reference to `__asan_stack_malloc_1' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:344: undefined reference to `__asan_memset' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:344: undefined reference to `__asan_memcpy' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `asan.module_ctor': ./src/modules/rlm_mschap/smbdes.c:(.text+0x3d42): undefined reference to `__asan_init' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:(.text+0x3d47): undefined reference to `__asan_version_mismatch_check_v8' /usr/bin/ld: ./src/modules/rlm_mschap/smbdes.c:(.text+0x3d58): undefined reference to `__asan_register_globals' /usr/bin/ld: build/objs/src/modules/rlm_mschap/smbdes.o: in function `asan.module_dtor': ./src/modules/rlm_mschap/smbdes.c:(.text+0x3d6e): undefined reference to `__asan_unregister_globals' clang: error: linker command failed with exit code 1 (use -v to see invocation) make[2]: *** [scripts/boiler.mk:630: build/bin/smbencrypt] Error 1 make[2]: *** Waiting for unfinished jobs.... make[2]: Leaving directory '/<<PKGBUILDDIR>>' make[1]: *** [debian/rules:23: override_dh_auto_install] Error 2 make[1]: Leaving directory '/<<PKGBUILDDIR>>' make: *** [debian/rules:94: binary] Error 2 dpkg-buildpackage: error: fakeroot debian/rules binary subprocess returned exit status 2
On Mar 15, 2019, at 11:19 AM, Daniel Feuchtinger <daniel.feuchtinger@lrz.de> wrote:
-fsanitize=address -fno-omit-frame-pointer -fno-optimize-sibling-calls -fsanitize-address-use-after-scope We get compile errors with those flags in some modules using Debian Stretch and Debian Buster (ippool, rlm_mschap, rml_eap, sorry for the verbosity, it's just rlm_mschap):
/usr/bin/ld: build/objs/src/modules/rlm_mschap/smbencrypt.o: in function `main': ./src/modules/rlm_mschap/smbencrypt.c:63: undefined reference to `__asan_option_detect_stack_use_after_return' /usr/bin/ld: ./src/modules/rlm_mschap/smbencrypt.c:63: undefined reference to `__asan_stack_malloc_5' /usr/bin/ld: ./src/modules/rlm_mschap/smbencrypt.c:63: undefined reference to `__asan_set_shadow_f8'
It's not linking to the address sanitizer libraries. Edit the top-level Make.inc. There's an entry: LIBS = ... Add "-lasan" to the end, without the quotes. It should build. Alan DeKok.
Hi, finally I found the time to build the recent freeradius with address sanitizer support. I built with gcc, that works fine, no clang needed. RTLD_DEEPBIND in src/main/modules.c seems to break asan-support? https://github.com/google/sanitizers/issues/611 The changelog says, trouble with oracle was the cause for RTLD_DEEPBIND, would it be possible to couple RTLD_DEEPBIND via configure to oracle support? The debian package has no oracle support, so I used this patch: --- freeradius-3.0.20+dfsg.orig/src/main/modules.c +++ freeradius-3.0.20+dfsg/src/main/modules.c @@ -168,10 +168,11 @@ fr_dlhandle fr_dlopenext(char const *nam } else #endif flags |= RTLD_LOCAL; +/* #ifdef RTLD_DEEPBIND flags |= RTLD_DEEPBIND; #endif - +*/ The first run crashed after a few minutes (see AddressSanitizer output below), a second run with higher max_requests still runs without a crash for a few hours now. I guess there shouldn't be a crash with double-free, even if max_requests is to low? Here's the asan-Log: ================================================================= ==115708==ERROR: AddressSanitizer: attempting double-free on 0x60b0001bbab0 in thread T0: #0 0x7f7ee2477fb0 in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe8fb0) #1 0x7f7ee1e825d2 in _tc_free_internal ../talloc.c:1201 #2 0x7f7ee1e8243f in _tc_free_children_internal ../talloc.c:1646 #3 0x7f7ee1e8243f in _tc_free_internal ../talloc.c:1163 #4 0x7f7ee1e8243f in _tc_free_children_internal ../talloc.c:1646 #5 0x7f7ee1e8243f in _tc_free_internal ../talloc.c:1163 #6 0x7f7ee1e8243f in _tc_free_children_internal ../talloc.c:1646 #7 0x7f7ee1e8243f in _tc_free_internal ../talloc.c:1163 #8 0x7f7ee1e7d347 in _tc_free_children_internal ../talloc.c:1646 #9 0x7f7ee1e7d347 in _tc_free_internal ../talloc.c:1163 #10 0x7f7ee1e7d347 in _talloc_free_internal ../talloc.c:1227 #11 0x7f7ee1e7d347 in _talloc_free ../talloc.c:1769 #12 0x5594dc01a3cd in request_free src/main/process.c:602 #13 0x5594dc01a3cd in request_free src/main/process.c:587 #14 0x5594dc0206fc in request_done src/main/process.c:897 #15 0x5594dc024ac0 in request_receive src/main/process.c:1765 #16 0x5594dbfe7c3b in auth_socket_recv src/main/listen.c:1597 #17 0x5594dc01a775 in event_socket_handler src/main/process.c:4869 #18 0x7f7ee22b1988 in fr_event_loop src/lib/event.c:649 #19 0x5594dc031734 in radius_event_process src/main/process.c:5954 #20 0x5594dbfc7d33 in main src/main/radiusd.c:626 #21 0x7f7ee197209a in __libc_start_main ../csu/libc-start.c:308 #22 0x5594dbfc8979 in _start (/usr/sbin/freeradius+0x57979) 0x60b0001bbab0 is located 0 bytes inside of 111-byte region [0x60b0001bbab0,0x60b0001bbb1f) freed by thread T0 here: ==115708==AddressSanitizer CHECK failed: ../../../../src/libsanitizer/sanitizer_common/sanitizer_stackdepotbase.h:140 "((id & (((u32)-1) >> kReservedBits))) == ((id))" (0x6a15fa73, 0xea15fa73) #0 0x7f7ee2482fa5 (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xf3fa5) #1 0x7f7ee249df39 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10ef39) #2 0x7f7ee24982af (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x1092af) #3 0x7f7ee23bc1ec (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x2d1ec) #4 0x7f7ee23bd60d (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x2e60d) #5 0x7f7ee23bdc2f (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x2ec2f) #6 0x7f7ee247fee3 (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xf0ee3) #7 0x7f7ee23bacc4 (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x2bcc4) #8 0x7f7ee2477f8a in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe8f8a) #9 0x7f7ee1e825d2 in _tc_free_internal ../talloc.c:1201 #10 0x7f7ee1e8243f in _tc_free_children_internal ../talloc.c:1646 #11 0x7f7ee1e8243f in _tc_free_internal ../talloc.c:1163 #12 0x7f7ee1e8243f in _tc_free_children_internal ../talloc.c:1646 #13 0x7f7ee1e8243f in _tc_free_internal ../talloc.c:1163 #14 0x7f7ee1e8243f in _tc_free_children_internal ../talloc.c:1646 #15 0x7f7ee1e8243f in _tc_free_internal ../talloc.c:1163 #16 0x7f7ee1e7d347 in _tc_free_children_internal ../talloc.c:1646 #17 0x7f7ee1e7d347 in _tc_free_internal ../talloc.c:1163 #18 0x7f7ee1e7d347 in _talloc_free_internal ../talloc.c:1227 #19 0x7f7ee1e7d347 in _talloc_free ../talloc.c:1769 #20 0x5594dc01a3cd in request_free src/main/process.c:602 #21 0x5594dc01a3cd in request_free src/main/process.c:587 #22 0x5594dc0206fc in request_done src/main/process.c:897 #23 0x5594dc024ac0 in request_receive src/main/process.c:1765 #24 0x5594dbfe7c3b in auth_socket_recv src/main/listen.c:1597 #25 0x5594dc01a775 in event_socket_handler src/main/process.c:4869 #26 0x7f7ee22b1988 in fr_event_loop src/lib/event.c:649 #27 0x5594dc031734 in radius_event_process src/main/process.c:5954 #28 0x5594dbfc7d33 in main src/main/radiusd.c:626 #29 0x7f7ee197209a in __libc_start_main ../csu/libc-start.c:308 #30 0x5594dbfc8979 in _start (/usr/sbin/freeradius+0x57979) Am 06.03.19 um 23:17 schrieb Alan DeKok:
Another alternative is to use LLVM. If you install the LLVM RPM, you can then build FreeRADIUS with some more options.
1) install LLVM 2) grab the 3.0.18 source (or the source from GitHub) 3) do: CC=clang ./configure --prefix=/opt/freeradius ... args ...
I think the name of the compiler on Suse is "clang". If not, you'll have to find that out
You're best to put this build into a unique directory, so it's easy to get rid of it later.
4) Edit "Make.inc", and look for the line "CFLAGS = ..." add this:
-fsanitize=address -fno-omit-frame-pointer -fno-optimize-sibling-calls -fsanitize-address-use-after-scope
5) run "make; make install"
You can then run that version with some extra environment variables:
export ASAN_SYMBOLIZER_PATH="/usr/local/opt/llvm/bin/llvm-symbolizer"
You'll have to find out where that executable is located...
export ASAN_OPTIONS="malloc_context_size=50 detect_leaks=1 symbolize=1"
Then run FreeRADIUS: /usr/freeradius/sbin/radiusd -f -d /etc/raddb
With some luck, the address sanitizer may find the issue. Note that the server will run much slower than normal, but that's likely fine.
These issues are very, very, difficult to track down without tools like the address sanitizer. We're using that in the v4 / master branch, and it's a huge help.
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
-- Daniel Feuchtinger Leibniz-Rechenzentrum Boltzmannstraße 1 D-85748 Garching b. München Tel. 089 35831 8820
Am 04.12.19 um 13:05 schrieb Daniel Feuchtinger:
The first run crashed after a few minutes (see AddressSanitizer output below), a second run with higher max_requests still runs without a crash for a few hours now. I guess there shouldn't be a crash with double-free, even if max_requests is to low? The issue is not related to max_requests, but maybe to threading? With -s the server run stable for some hours, without -s it crashes after minutes.
The asan output looks the same as before.
================================================================= ==115708==ERROR: AddressSanitizer: attempting double-free on 0x60b0001bbab0 in thread T0: #0 0x7f7ee2477fb0 in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe8fb0) #1 0x7f7ee1e825d2 in _tc_free_internal ../talloc.c:1201 #2 0x7f7ee1e8243f in _tc_free_children_internal ../talloc.c:1646 #3 0x7f7ee1e8243f in _tc_free_internal ../talloc.c:1163 #4 0x7f7ee1e8243f in _tc_free_children_internal ../talloc.c:1646 #5 0x7f7ee1e8243f in _tc_free_internal ../talloc.c:1163 #6 0x7f7ee1e8243f in _tc_free_children_internal ../talloc.c:1646 #7 0x7f7ee1e8243f in _tc_free_internal ../talloc.c:1163 #8 0x7f7ee1e7d347 in _tc_free_children_internal ../talloc.c:1646 #9 0x7f7ee1e7d347 in _tc_free_internal ../talloc.c:1163 #10 0x7f7ee1e7d347 in _talloc_free_internal ../talloc.c:1227 #11 0x7f7ee1e7d347 in _talloc_free ../talloc.c:1769 #12 0x5594dc01a3cd in request_free src/main/process.c:602 #13 0x5594dc01a3cd in request_free src/main/process.c:587 #14 0x5594dc0206fc in request_done src/main/process.c:897 #15 0x5594dc024ac0 in request_receive src/main/process.c:1765 #16 0x5594dbfe7c3b in auth_socket_recv src/main/listen.c:1597 #17 0x5594dc01a775 in event_socket_handler src/main/process.c:4869 #18 0x7f7ee22b1988 in fr_event_loop src/lib/event.c:649 #19 0x5594dc031734 in radius_event_process src/main/process.c:5954 #20 0x5594dbfc7d33 in main src/main/radiusd.c:626 #21 0x7f7ee197209a in __libc_start_main ../csu/libc-start.c:308 #22 0x5594dbfc8979 in _start (/usr/sbin/freeradius+0x57979)
0x60b0001bbab0 is located 0 bytes inside of 111-byte region [0x60b0001bbab0,0x60b0001bbb1f) freed by thread T0 here: ==115708==AddressSanitizer CHECK failed: ../../../../src/libsanitizer/sanitizer_common/sanitizer_stackdepotbase.h:140 "((id & (((u32)-1) >> kReservedBits))) == ((id))" (0x6a15fa73, 0xea15fa73) #0 0x7f7ee2482fa5 (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xf3fa5) #1 0x7f7ee249df39 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10ef39) #2 0x7f7ee24982af (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x1092af) #3 0x7f7ee23bc1ec (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x2d1ec) #4 0x7f7ee23bd60d (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x2e60d) #5 0x7f7ee23bdc2f (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x2ec2f) #6 0x7f7ee247fee3 (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xf0ee3) #7 0x7f7ee23bacc4 (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x2bcc4) #8 0x7f7ee2477f8a in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe8f8a) #9 0x7f7ee1e825d2 in _tc_free_internal ../talloc.c:1201 #10 0x7f7ee1e8243f in _tc_free_children_internal ../talloc.c:1646 #11 0x7f7ee1e8243f in _tc_free_internal ../talloc.c:1163 #12 0x7f7ee1e8243f in _tc_free_children_internal ../talloc.c:1646 #13 0x7f7ee1e8243f in _tc_free_internal ../talloc.c:1163 #14 0x7f7ee1e8243f in _tc_free_children_internal ../talloc.c:1646 #15 0x7f7ee1e8243f in _tc_free_internal ../talloc.c:1163 #16 0x7f7ee1e7d347 in _tc_free_children_internal ../talloc.c:1646 #17 0x7f7ee1e7d347 in _tc_free_internal ../talloc.c:1163 #18 0x7f7ee1e7d347 in _talloc_free_internal ../talloc.c:1227 #19 0x7f7ee1e7d347 in _talloc_free ../talloc.c:1769 #20 0x5594dc01a3cd in request_free src/main/process.c:602 #21 0x5594dc01a3cd in request_free src/main/process.c:587 #22 0x5594dc0206fc in request_done src/main/process.c:897 #23 0x5594dc024ac0 in request_receive src/main/process.c:1765 #24 0x5594dbfe7c3b in auth_socket_recv src/main/listen.c:1597 #25 0x5594dc01a775 in event_socket_handler src/main/process.c:4869 #26 0x7f7ee22b1988 in fr_event_loop src/lib/event.c:649 #27 0x5594dc031734 in radius_event_process src/main/process.c:5954 #28 0x5594dbfc7d33 in main src/main/radiusd.c:626 #29 0x7f7ee197209a in __libc_start_main ../csu/libc-start.c:308 #30 0x5594dbfc8979 in _start (/usr/sbin/freeradius+0x57979)
Am 06.03.19 um 23:17 schrieb Alan DeKok:
Another alternative is to use LLVM. If you install the LLVM RPM, you can then build FreeRADIUS with some more options.
1) install LLVM 2) grab the 3.0.18 source (or the source from GitHub) 3) do: CC=clang ./configure --prefix=/opt/freeradius ... args ...
I think the name of the compiler on Suse is "clang". If not, you'll have to find that out
You're best to put this build into a unique directory, so it's easy to get rid of it later.
4) Edit "Make.inc", and look for the line "CFLAGS = ..." add this:
-fsanitize=address -fno-omit-frame-pointer -fno-optimize-sibling-calls -fsanitize-address-use-after-scope
5) run "make; make install"
You can then run that version with some extra environment variables:
export ASAN_SYMBOLIZER_PATH="/usr/local/opt/llvm/bin/llvm-symbolizer"
You'll have to find out where that executable is located...
export ASAN_OPTIONS="malloc_context_size=50 detect_leaks=1 symbolize=1"
Then run FreeRADIUS: /usr/freeradius/sbin/radiusd -f -d /etc/raddb
With some luck, the address sanitizer may find the issue. Note that the server will run much slower than normal, but that's likely fine.
These issues are very, very, difficult to track down without tools like the address sanitizer. We're using that in the v4 / master branch, and it's a huge help.
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
-- Daniel Feuchtinger Leibniz-Rechenzentrum Boltzmannstraße 1 D-85748 Garching b. München Tel. 089 35831 8820
On Dec 4, 2019, at 8:24 AM, Daniel Feuchtinger <daniel.feuchtinger@lrz.de> wrote:
Am 04.12.19 um 13:05 schrieb Daniel Feuchtinger:
The first run crashed after a few minutes (see AddressSanitizer output below), a second run with higher max_requests still runs without a crash for a few hours now. I guess there shouldn't be a crash with double-free, even if max_requests is to low? The issue is not related to max_requests, but maybe to threading? With -s the server run stable for some hours, without -s it crashes after minutes.
The issue seems to be that the server is slow, and is receiving conflicting packets. i.e. the NAS sends packet A, some time later gives up, and then send packet B. But the server is still running packet A. Likely because it's blocked in a database. So it tries to cancel packet A. But for some reason that cancellation is delayed. During that delay, the NAS retransmits packet B. And the server tries to cancel request A again. And then things go boom. It's not clear to me *why* this is happening. The issue is buried deep inside of some fairly complex state machine work in the server. I've pushed a patch which should help. It updates the main "check for duplicate / conflicting packet" code. When the *first* conflicting packet B comes in, packet A is now removed entirely from the conflicting packet list. That means when the next packet B comes in, any lookup will find B not A, and there's no double free. One reason this has been so hard to track down is that it doesn't occur in most situations. It looks like it happens only when (1) the back-end DB is very slow, and (2) the NAS retransmits very aggressively. Please test the latest code from GitHub: https://github.com/FreeRADIUS/freeradius-server.git If that works, it should be the definitive fix. Alan DeKok.
Thanks a lot for the help. Am 04.12.19 um 15:14 schrieb Alan DeKok:
On Dec 4, 2019, at 8:24 AM, Daniel Feuchtinger <daniel.feuchtinger@lrz.de> wrote:
The issue seems to be that the server is slow, and is receiving conflicting packets. i.e. the NAS sends packet A, some time later gives up, and then send packet B.
But the server is still running packet A. Likely because it's blocked in a database.
It's mainly a Proxy, only a few perl / ldap skripts to determine a group on a small part of the NASes.
So it tries to cancel packet A. But for some reason that cancellation is delayed. During that delay, the NAS retransmits packet B. And the server tries to cancel request A again. And then things go boom.
It's not clear to me *why* this is happening. The issue is buried deep inside of some fairly complex state machine work in the server.
I've pushed a patch which should help. It updates the main "check for duplicate / conflicting packet" code. When the *first* conflicting packet B comes in, packet A is now removed entirely from the conflicting packet list. That means when the next packet B comes in, any lookup will find B not A, and there's no double free.
One reason this has been so hard to track down is that it doesn't occur in most situations. It looks like it happens only when (1) the back-end DB is very slow, and (2) the NAS retransmits very aggressively.
Please test the latest code from GitHub:
I tried the Patch in commit 36656cc39f52dbc79f8b373d313281144eef205b with the 3.0.20 code, that resulted in a lot of memory leaks (109 of them within a few minutes): ================================================================= ==33264==ERROR: LeakSanitizer: detected memory leaks Direct leak of 3168 byte(s) in 36 object(s) allocated from: #0 0x7efc8022b330 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe9330) #1 0x7efc800f9ad8 in rad_malloc src/main/util.c:528 #2 0x55fc419d8aef in server_pool_alloc src/main/realms.c:1016 #3 0x55fc419d9016 in old_server_add src/main/realms.c:1628 #4 0x55fc419d9d84 in old_realm_config src/main/realms.c:1706 #5 0x55fc419e0404 in realm_add src/main/realms.c:2005 #6 0x55fc419e0404 in realms_init src/main/realms.c:2223 #7 0x55fc41997bb1 in main_config_init src/main/mainconfig.c:1141 #8 0x55fc4196e5cb in main src/main/radiusd.c:341 #9 0x7efc7f72509a in __libc_start_main ../csu/libc-start.c:308 SUMMARY: AddressSanitizer: 3168 byte(s) leaked in 36 allocation(s). The server did not crash but it stopped logging and probably working while producing the leaks. I'll try again with the current git code tomorrow.
If that works, it should be the definitive fix.
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On Dec 4, 2019, at 11:31 AM, Daniel Feuchtinger <daniel.feuchtinger@lrz.de> wrote:
I tried the Patch in commit 36656cc39f52dbc79f8b373d313281144eef205b with the 3.0.20 code, that resulted in a lot of memory leaks (109 of them within a few minutes):
That patch shouldn't cause memory leaks.
================================================================= ==33264==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 3168 byte(s) in 36 object(s) allocated from: #0 0x7efc8022b330 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe9330) #1 0x7efc800f9ad8 in rad_malloc src/main/util.c:528 #2 0x55fc419d8aef in server_pool_alloc src/main/realms.c:1016
The patch definitely won't cause that memory leak.
The server did not crash but it stopped logging and probably working while producing the leaks.
I'll try again with the current git code tomorrow.
OK. Alan DeKok.
Am 04.12.19 um 17:35 schrieb Alan DeKok:
On Dec 4, 2019, at 11:31 AM, Daniel Feuchtinger <daniel.feuchtinger@lrz.de> wrote:
I tried the Patch in commit 36656cc39f52dbc79f8b373d313281144eef205b with the 3.0.20 code, that resulted in a lot of memory leaks (109 of them within a few minutes):
That patch shouldn't cause memory leaks. You're right, that was the systemd trying to restart, running the config check, sorry about the confusion. freeradius -Cm causes that memory leak and lets the config checkfail, freeradius -C does not.
The crash problem remains the same, some code lines shifted in process.c by 10 after the patch: ================================================================= ==36658==ERROR: AddressSanitizer: attempting double-free on 0x60b0054c1fd0 in thread T0: #0 0x7f5cc0a0ffb0 in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe8fb0) #1 0x7f5cc041a5d2 in _tc_free_internal ../talloc.c:1201 #2 0x7f5cc041a43f in _tc_free_children_internal ../talloc.c:1646 #3 0x7f5cc041a43f in _tc_free_internal ../talloc.c:1163 #4 0x7f5cc041a43f in _tc_free_children_internal ../talloc.c:1646 #5 0x7f5cc041a43f in _tc_free_internal ../talloc.c:1163 #6 0x7f5cc0415347 in _tc_free_children_internal ../talloc.c:1646 #7 0x7f5cc0415347 in _tc_free_internal ../talloc.c:1163 #8 0x7f5cc0415347 in _talloc_free_internal ../talloc.c:1227 #9 0x7f5cc0415347 in _talloc_free ../talloc.c:1769 #10 0x55abd09923cd in request_free src/main/process.c:602 #11 0x55abd09923cd in request_free src/main/process.c:587 #12 0x55abd09986fc in request_done src/main/process.c:897 #13 0x55abd099cb56 in request_receive src/main/process.c:1775 #14 0x55abd095fc3b in auth_socket_recv src/main/listen.c:1597 #15 0x55abd0992775 in event_socket_handler src/main/process.c:4879 #16 0x7f5cc0849988 in fr_event_loop src/lib/event.c:649 #17 0x55abd09a97b4 in radius_event_process src/main/process.c:5964 #18 0x55abd093fd33 in main src/main/radiusd.c:626 #19 0x7f5cbff0a09a in __libc_start_main ../csu/libc-start.c:308 #20 0x55abd0940979 in _start (/usr/sbin/freeradius+0x57979) I'll try the current code tomorrow, but it may have the same results. -- Daniel Feuchtinger Leibniz-Rechenzentrum Boltzmannstraße 1 D-85748 Garching b. München Tel. 089 35831 8820
participants (4)
-
Adam Majer -
Alan DeKok -
Daniel Feuchtinger -
Jorge Pereira