received signal SIGSEGV, Segmentation fault. malloc_consolidate (av=0x7ffff5e2de80) at malloc.c:5196
Hello, I'm testing kerberos authentication with the workaround from yesterday (https://github.com/FreeRADIUS/freeradius-server/commit/ea887a2ab7308a6f612da...). After 2-3 authentication tests radiusd get a SIGSEGV. OS: CentOS 6.5 x86_64 FR built today from v3.0.x branch BT: rad_recv: Access-Request packet from host 127.0.0.1 port 55082, id=14, length=103 User-Name = 'hachmer' User-Password = 'pw' NAS-IP-Address = 127.0.0.1 NAS-Port = 0 Message-Authenticator = 0x64454c20cfa4534882ca622de7e32c0e (2) # Executing section authorize from file /etc/raddb/sites-enabled/default (2) authorize { (2) filter_username filter_username { (2) ? if (User-Name != "%{tolower:%{User-Name}}") Program received signal SIGSEGV, Segmentation fault. malloc_consolidate (av=0x7ffff5e2de80) at malloc.c:5196 5196 nextsize = chunksize(nextchunk); Missing separate debuginfos, use: debuginfo-install ncurses-libs-5.7-3.20090208.el6.x86_64 pcre-7.8-6.el6.x86_64 sqlite-3.6.20-1.el6.x86_64 (gdb) bt #0 malloc_consolidate (av=0x7ffff5e2de80) at malloc.c:5196 #1 0x00007ffff5b17405 in _int_malloc (av=0x7ffff5e2de80, bytes=<value optimized out>) at malloc.c:4406 #2 0x00007ffff5b18991 in __libc_malloc (bytes=1104) at malloc.c:3664 #3 0x00007ffff715063f in __talloc (ctx=<value optimized out>, el_size=<value optimized out>, count=<value optimized out>, name=<value optimized out>) at ../talloc.c:560 #4 _talloc_named_const (ctx=<value optimized out>, el_size=<value optimized out>, count=<value optimized out>, name=<value optimized out>) at ../talloc.c:669 #5 _talloc_array (ctx=<value optimized out>, el_size=<value optimized out>, count=<value optimized out>, name=<value optimized out>) at ../talloc.c:2217 #6 0x00007ffff7bcf0f4 in xlat_aprint (ctx=0xa16a40, request=0xa16a40, node=0x913af0, escape=0, escape_ctx=0x0, lvl=0) at src/main/xlat.c:1895 #7 0x00007ffff7bcfcd1 in xlat_process (out=0x7fffffffc1b8, request=0xa16a40, head=<value optimized out>, escape=0, escape_ctx=0x0) at src/main/xlat.c:1976 #8 0x00007ffff7bcff8c in xlat_expand (out=0x7fffffffc2c0, outlen=0, request=0xa16a40, fmt=0x81c0c0 "%{tolower:%{User-Name}}", escape=0, escape_ctx=0x0) at src/main/xlat.c:2068 #9 0x00007ffff7bcb7e8 in radius_expand_tmpl (out=0x7fffffffc2c0, request=0xa16a40, vpt=<value optimized out>) at src/main/evaluate.c:108 #10 0x00007ffff7bcbfd8 in radius_evaluate_map (request=0xa16a40, modreturn=<value optimized out>, depth=<value optimized out>, c=0x81bd30) at src/main/evaluate.c:424 #11 0x00007ffff7bcc5b1 in radius_evaluate_cond (request=0xa16a40, modreturn=10, depth=0, c=<value optimized out>) at src/main/evaluate.c:597 #12 0x000000000041ed3b in modcall_recurse (request=0xa16a40, component=RLM_COMPONENT_AUTZ, depth=2, entry=0x7fffffffd3d0) at src/main/modcall.c:480 #13 0x00000000004201d4 in modcall_child (request=<value optimized out>, component=RLM_COMPONENT_AUTZ, depth=2, entry=0x7fffffffd3b8, c=0xa026a0, result=0x7fffffffc9ec) at src/main/modcall.c:414 #14 0x000000000041eb5f in modcall_recurse (request=0xa16a40, component=RLM_COMPONENT_AUTZ, depth=1, entry=0x7fffffffd3b8) at src/main/modcall.c:780 #15 0x00000000004201d4 in modcall_child (request=<value optimized out>, component=RLM_COMPONENT_AUTZ, depth=1, entry=0x7fffffffd3a0, c=0xa02620, result=0x7fffffffcf4c) at src/main/modcall.c:414 #16 0x000000000041eb5f in modcall_recurse (request=0xa16a40, component=RLM_COMPONENT_AUTZ, depth=0, entry=0x7fffffffd3a0) at src/main/modcall.c:780 #17 0x00000000004200fd in modcall (component=<value optimized out>, c=<value optimized out>, request=<value optimized out>) at src/main/modcall.c:1036 #18 0x000000000041af93 in indexed_modcall (comp=RLM_COMPONENT_AUTZ, idx=0, request=0xa16a40) at src/main/modules.c:747 #19 0x000000000040cc42 in rad_authenticate (request=0xa16a40) at src/main/auth.c:409 #20 0x000000000042b41c in request_running (request=0xa16a40, action=1) at src/main/process.c:1210 #21 0x000000000042af4d in request_queue_or_run (request=0xa16a40, process=0x42b2e0 <request_running>) at src/main/process.c:850 #22 0x000000000042d2b0 in request_receive (listener=0xa131f0, packet=0x8fe710, client=0x8110a0, fun=0x40c9b0 <rad_authenticate>) at src/main/process.c:1409 #23 0x00000000004158b3 in auth_socket_recv (listener=0xa131f0) at src/main/listen.c:1509 #24 0x000000000042777b in event_socket_handler (xel=<value optimized out>, fd=<value optimized out>, ctx=0xa131f0) at src/main/process.c:3494 #25 0x00007ffff79abcbb in fr_event_loop (el=0xa05770) at src/lib/event.c:417 #26 0x0000000000420b14 in main (argc=<value optimized out>, argv=<value optimized out>) at src/main/radiusd.c:479 Thanks in advance, Tobias Hachmer
On 11 Dec 2013, at 14:00, Hachmer, Tobias <Tobias.Hachmer@stadt-frankfurt.de> wrote:
Hello,
I’m testing kerberos authentication with the workaround from yesterday (https://github.com/FreeRADIUS/freeradius-server/commit/ea887a2ab7308a6f612da...). After 2-3 authentication tests radiusd get a SIGSEGV.
OS: CentOS 6.5 x86_64 FR built today from v3.0.x branch
BT: rad_recv: Access-Request packet from host 127.0.0.1 port 55082, id=14, length=103 User-Name = ‘hachmer' User-Password = 'pw' NAS-IP-Address = 127.0.0.1 NAS-Port = 0 Message-Authenticator = 0x64454c20cfa4534882ca622de7e32c0e (2) # Executing section authorize from file /etc/raddb/sites-enabled/default (2) authorize { (2) filter_username filter_username { (2) ? if (User-Name != "%{tolower:%{User-Name}}")
*sigh* probably memory corruption could you run it under valgrind and see if there's any useful info? https://github.com/FreeRADIUS/freeradius-server/blob/master/doc/bugs (see section 5). I'd do it but valgrind is completely broken under OSX currently. Also when you build your RPMs could you make sure to pass --enable-developer to turn off optimization so we don't get all those <value optimized out> messages. -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS Development Team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
Hello Arran, thanks for the answer. -----Ursprüngliche Nachricht----- Von: freeradius-users-bounces+tobias.hachmer=stadt-frankfurt.de@lists.freeradius.org [mailto:freeradius-users-bounces+tobias.hachmer=stadt-frankfurt.de@lists.freeradius.org] Im Auftrag von Arran Cudbard-Bell Gesendet: Mittwoch, 11. Dezember 2013 15:26 An: FreeRadius users mailing list Betreff: Re: received signal SIGSEGV, Segmentation fault. malloc_consolidate (av=0x7ffff5e2de80) at malloc.c:5196
*sigh* probably memory corruption could you run it under valgrind and see if there's any useful info? Complete output of valgrind is about 10000 lines. It's hard to me to find out which lines are the useful ones but I give it a try. See at the end of this post.
Also when you build your RPMs could you make sure to pass --enable-developer to turn off optimization so we don't get all those <value optimized out> messages. I have built a new package with this configure parameter but there are still optimized out messages.
This SIGSEGV signal only comes if the Auth-Type is Kerberos. Running radius and there are only requests which perform ldap auths all is running fine. Again the BT: rad_recv: Access-Request packet from host 127.0.0.1 port 57314, id=37, length=103 User-Name = 'hachmer' User-Password = 'pass' NAS-IP-Address = 127.0.0.1 NAS-Port = 0 Message-Authenticator = 0x73022b83754542e2627e6cda51d690d8 (2) # Executing section authorize from file /etc/raddb/sites-enabled/default (2) authorize { (2) filter_username filter_username { (2) ? if (User-Name != "%{tolower:%{User-Name}}") Program received signal SIGSEGV, Segmentation fault. 0x00007ffff5b15285 in malloc_consolidate () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install freeradius-3.0.1-6.el6.x86_64 (gdb) bt #0 0x00007ffff5b15285 in malloc_consolidate () from /lib64/libc.so.6 #1 0x00007ffff5b18405 in _int_malloc () from /lib64/libc.so.6 #2 0x00007ffff5b19991 in malloc () from /lib64/libc.so.6 #3 0x00007ffff715163f in _talloc_array () from /usr/lib64/libtalloc.so.2 #4 0x00007ffff7bceb25 in xlat_aprint (ctx=0xa35b80, request=0xa35b80, node=0xa39dc0, escape=0, escape_ctx=0x0, lvl=0) at src/main/xlat.c:1895 #5 0x00007ffff7bcf6f1 in xlat_process (out=0x7fffffffc508, request=0xa35b80, head=<value optimized out>, escape=0, escape_ctx=0x0) at src/main/xlat.c:1976 #6 0x00007ffff7bcf9ac in xlat_expand (out=0x7fffffffc690, outlen=0, request=0xa35b80, fmt=0x82dff0 "%{tolower:%{User-Name}}", escape=0, escape_ctx=0x0) at src/main/xlat.c:2068 #7 0x00007ffff7bcb328 in radius_expand_tmpl (out=0x7fffffffc690, request=0xa35b80, vpt=<value optimized out>) at src/main/evaluate.c:108 #8 0x00007ffff7bcb9d8 in radius_evaluate_map (request=0xa35b80, modreturn=<value optimized out>, depth=<value optimized out>, c=0x82dc60) at src/main/evaluate.c:424 #9 0x00007ffff7bcc131 in radius_evaluate_cond (request=0xa35b80, modreturn=10, depth=0, c=<value optimized out>) at src/main/evaluate.c:597 #10 0x000000000041e56b in modcall_recurse (request=0xa35b80, component=RLM_COMPONENT_AUTZ, depth=2, entry=0x7fffffffd800) at src/main/modcall.c:480 #11 0x000000000041f9f4 in modcall_child (request=<value optimized out>, component=RLM_COMPONENT_AUTZ, depth=2, entry=0x7fffffffd7e8, c=0xa1adb0, result=0x7fffffffd1dc) at src/main/modcall.c:414 #12 0x000000000041e3a7 in modcall_recurse (request=0xa35b80, component=RLM_COMPONENT_AUTZ, depth=1, entry=0x7fffffffd7e8) at src/main/modcall.c:780 #13 0x000000000041f9f4 in modcall_child (request=<value optimized out>, component=RLM_COMPONENT_AUTZ, depth=1, entry=0x7fffffffd7d0, c=0xa1ad30, result=0x7fffffffd78c) at src/main/modcall.c:414 #14 0x000000000041e3a7 in modcall_recurse (request=0xa35b80, component=RLM_COMPONENT_AUTZ, depth=0, entry=0x7fffffffd7d0) at src/main/modcall.c:780 #15 0x000000000041f91d in modcall (component=<value optimized out>, c=<value optimized out>, request=<value optimized out>) at src/main/modcall.c:1036 #16 0x000000000041a8a3 in indexed_modcall (comp=RLM_COMPONENT_AUTZ, idx=0, request=0xa35b80) at src/main/modules.c:747 #17 0x000000000040cb52 in rad_authenticate (request=0xa35b80) at src/main/auth.c:409 #18 0x000000000042a96c in request_running (request=0xa35b80, action=1) at src/main/process.c:1211 #19 0x000000000042a4dd in request_queue_or_run (request=0xa35b80, process=0x42a830 <request_running>) at src/main/process.c:851 #20 0x000000000042c7c1 in request_receive (listener=0xa337a0, packet=0x8827e0, client=0x81a370, fun=0x40c8f0 <rad_authenticate>) at src/main/process.c:1414 #21 0x0000000000417624 in auth_socket_recv (listener=0xa337a0) at src/main/listen.c:1509 #22 0x0000000000426e6a in event_socket_handler (xel=<value optimized out>, fd=<value optimized out>, ctx=0xa337a0) at src/main/process.c:3499 #23 0x00007ffff79ac35b in fr_event_loop (el=0xa1de60) at src/lib/event.c:417 #24 0x0000000000420294 in main (argc=<value optimized out>, argv=<value optimized out>) at src/main/radiusd.c:479 Valgrind: ==1937== Memcheck, a memory error detector ==1937== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al. ==1937== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info ==1937== Command: radiusd -Xm ==1937== radiusd: FreeRADIUS Version 3.0.1 (git #fafc5f6), for host x86_64-redhat-linux-gnu, built on Dec 12 2013 at 07:10:43 ... Sending Access-Accept of id 164 from 127.0.0.1 port 1812 to 127.0.0.1 port 40209 Idle-Timeout = 300 Session-Timeout = 3600 Xylan-Access-Priv = Xylan-Read-Priv Xylan-Access-Priv = Xylan-Write-Priv Xylan-Access-Priv = Xylan-Admin-Priv Xylan-Acce-Priv-F-W1 = 0xffffffff Xylan-Acce-Priv-F-W2 = 0xffffffff Xylan-Nms-Group = 'Default' NS-Admin-Privilege = Root-Admin NS-NSM-User-Domain-Name = 'global' NS-NSM-User-Role-Mapping = 'global:System Administrator' Aruba-User-Role = 'root' Aruba-Priv-Admin-User = 1 Juniper-Junosspace-Profile = 'readwrite' Service-Type = Administrative-User ==1937== Syscall param sendmsg(msg.msg_control) points to uninitialised byte(s) ==1937== at 0x6750D30: __sendmsg_nocancel (in /lib64/libpthread-2.12.so) ==1937== by 0x506A18C: sendfromto (udpfromto.c:391) ==1937== by 0x50672AD: rad_send (radius.c:283) ==1937== by 0x412E1D: auth_socket_send (listen.c:1271) ==1937== by 0x42AA3E: request_running (process.c:1154) ==1937== by 0x42A4DC: request_queue_or_run (process.c:851) ==1937== by 0x42C7C0: request_receive (process.c:1414) ==1937== by 0x417623: auth_socket_recv (listen.c:1509) ==1937== by 0x426E69: event_socket_handler (process.c:3499) ==1937== by 0x506F35A: fr_event_loop (event.c:417) ==1937== by 0x420293: main (radiusd.c:479) ==1937== Address 0x7fefffc4c is on thread 1's stack ==1937== (0) Finished request 0. ... (1) krb5 : Using client principal "hachmer@EXAMPLE.COM" ==1937== Invalid read of size 1 ==1937== at 0x6E2112C: vfprintf (in /lib64/libc-2.12.so) ==1937== by 0x6E486D9: vasprintf (in /lib64/libc-2.12.so) ==1937== by 0x6E28647: asprintf (in /lib64/libc-2.12.so) ==1937== by 0x7839591: k5_plugin_register_dyn (in /lib64/libkrb5.so.3.3) ==1937== by 0x783AD97: ??? (in /lib64/libkrb5.so.3.3) ==1937== by 0x783BCF9: ??? (in /lib64/libkrb5.so.3.3) ==1937== by 0x782AFC5: ??? (in /lib64/libkrb5.so.3.3) ==1937== by 0x782B908: krb5_init_creds_init (in /lib64/libkrb5.so.3.3) ==1937== by 0x782D806: ??? (in /lib64/libkrb5.so.3.3) ==1937== by 0x782F6A7: krb5_get_init_creds_password (in /lib64/libkrb5.so.3.3) ==1937== by 0xD02771E: krb5_auth (rlm_krb5.c:518) ==1937== by 0x41E709: modcall_recurse (modcall.c:311) ==1937== Address 0x908a9b0 is 0 bytes inside a block of size 24 free'd ==1937== at 0x4C273F0: free (vg_replace_malloc.c:446) ==1937== by 0x7830B2D: krb5_free_context (in /lib64/libkrb5.so.3.3) ==1937== by 0xD0274BD: krb5_auth (rlm_krb5.c:569) ==1937== by 0x41E709: modcall_recurse (modcall.c:311) ==1937== by 0x41F9F3: modcall_child (modcall.c:414) ==1937== by 0x41E3A6: modcall_recurse (modcall.c:780) ==1937== by 0x41F91C: modcall (modcall.c:1036) ==1937== by 0x41A8A2: indexed_modcall (modules.c:747) ==1937== by 0x40CFD0: rad_authenticate (auth.c:252) ==1937== by 0x42A96B: request_running (process.c:1211) ==1937== by 0x42A4DC: request_queue_or_run (process.c:851) ==1937== by 0x42C7C0: request_receive (process.c:1414) ==1937== ==1937== Invalid read of size 1 ==1937== at 0x4C2B008: mempcpy (mc_replace_strmem.c:1267) ==1937== by 0x6E4D95D: _IO_default_xsputn (in /lib64/libc-2.12.so) ==1937== by 0x6E2148F: vfprintf (in /lib64/libc-2.12.so) ==1937== by 0x6E486D9: vasprintf (in /lib64/libc-2.12.so) ==1937== by 0x6E28647: asprintf (in /lib64/libc-2.12.so) ==1937== by 0x7839591: k5_plugin_register_dyn (in /lib64/libkrb5.so.3.3) ==1937== by 0x783AD97: ??? (in /lib64/libkrb5.so.3.3) ==1937== by 0x783BCF9: ??? (in /lib64/libkrb5.so.3.3) ==1937== by 0x782AFC5: ??? (in /lib64/libkrb5.so.3.3) ==1937== by 0x782B908: krb5_init_creds_init (in /lib64/libkrb5.so.3.3) ==1937== by 0x782D806: ??? (in /lib64/libkrb5.so.3.3) ==1937== by 0x782F6A7: krb5_get_init_creds_password (in /lib64/libkrb5.so.3.3) ==1937== Address 0x908a9c6 is 22 bytes inside a block of size 24 free'd ==1937== at 0x4C273F0: free (vg_replace_malloc.c:446) ==1937== by 0x7830B2D: krb5_free_context (in /lib64/libkrb5.so.3.3) ==1937== by 0xD0274BD: krb5_auth (rlm_krb5.c:569) ==1937== by 0x41E709: modcall_recurse (modcall.c:311) ==1937== by 0x41F9F3: modcall_child (modcall.c:414) ==1937== by 0x41E3A6: modcall_recurse (modcall.c:780) ==1937== by 0x41F91C: modcall (modcall.c:1036) ==1937== by 0x41A8A2: indexed_modcall (modules.c:747) ==1937== by 0x40CFD0: rad_authenticate (auth.c:252) ==1937== by 0x42A96B: request_running (process.c:1211) ==1937== by 0x42A4DC: request_queue_or_run (process.c:851) ==1937== by 0x42C7C0: request_receive (process.c:1414) ==1937== ==1937== Invalid read of size 1 ==1937== at 0x4C2B01B: mempcpy (mc_replace_strmem.c:1267) ==1937== by 0x6E4D95D: _IO_default_xsputn (in /lib64/libc-2.12.so) ==1937== by 0x6E2148F: vfprintf (in /lib64/libc-2.12.so) ==1937== by 0x6E486D9: vasprintf (in /lib64/libc-2.12.so) ==1937== by 0x6E28647: asprintf (in /lib64/libc-2.12.so) ==1937== by 0x7839591: k5_plugin_register_dyn (in /lib64/libkrb5.so.3.3) ==1937== by 0x783AD97: ??? (in /lib64/libkrb5.so.3.3) ==1937== by 0x783BCF9: ??? (in /lib64/libkrb5.so.3.3) ==1937== by 0x782AFC5: ??? (in /lib64/libkrb5.so.3.3) ==1937== by 0x782B908: krb5_init_creds_init (in /lib64/libkrb5.so.3.3) ==1937== by 0x782D806: ??? (in /lib64/libkrb5.so.3.3) ==1937== by 0x782F6A7: krb5_get_init_creds_password (in /lib64/libkrb5.so.3.3) ==1937== Address 0x908a9c4 is 20 bytes inside a block of size 24 free'd ==1937== at 0x4C273F0: free (vg_replace_malloc.c:446) ==1937== by 0x7830B2D: krb5_free_context (in /lib64/libkrb5.so.3.3) ==1937== by 0xD0274BD: krb5_auth (rlm_krb5.c:569) ==1937== by 0x41E709: modcall_recurse (modcall.c:311) ==1937== by 0x41F9F3: modcall_child (modcall.c:414) ==1937== by 0x41E3A6: modcall_recurse (modcall.c:780) ==1937== by 0x41F91C: modcall (modcall.c:1036) ==1937== by 0x41A8A2: indexed_modcall (modules.c:747) ==1937== by 0x40CFD0: rad_authenticate (auth.c:252) ==1937== by 0x42A96B: request_running (process.c:1211) ==1937== by 0x42A4DC: request_queue_or_run (process.c:851) ==1937== by 0x42C7C0: request_receive (process.c:1414) ==1937== (1) krb5 : Successfully retrieved and decrypted TGT ==1937== Invalid free() / delete / delete[] / realloc() ==1937== at 0x4C273F0: free (vg_replace_malloc.c:446) ==1937== by 0x7830B2D: krb5_free_context (in /lib64/libkrb5.so.3.3) ==1937== by 0xD0274BD: krb5_auth (rlm_krb5.c:569) ==1937== by 0x41E709: modcall_recurse (modcall.c:311) ==1937== by 0x41F9F3: modcall_child (modcall.c:414) ==1937== by 0x41E3A6: modcall_recurse (modcall.c:780) ==1937== by 0x41F91C: modcall (modcall.c:1036) ==1937== by 0x41A8A2: indexed_modcall (modules.c:747) ==1937== by 0x40CFD0: rad_authenticate (auth.c:252) ==1937== by 0x42A96B: request_running (process.c:1211) ==1937== by 0x42A4DC: request_queue_or_run (process.c:851) ==1937== by 0x42C7C0: request_receive (process.c:1414) ==1937== Address 0x908a9b0 is 0 bytes inside a block of size 24 free'd ==1937== at 0x4C273F0: free (vg_replace_malloc.c:446) ==1937== by 0x7830B2D: krb5_free_context (in /lib64/libkrb5.so.3.3) ==1937== by 0xD0274BD: krb5_auth (rlm_krb5.c:569) ==1937== by 0x41E709: modcall_recurse (modcall.c:311) ==1937== by 0x41F9F3: modcall_child (modcall.c:414) ==1937== by 0x41E3A6: modcall_recurse (modcall.c:780) ==1937== by 0x41F91C: modcall (modcall.c:1036) ==1937== by 0x41A8A2: indexed_modcall (modules.c:747) ==1937== by 0x40CFD0: rad_authenticate (auth.c:252) ==1937== by 0x42A96B: request_running (process.c:1211) ==1937== by 0x42A4DC: request_queue_or_run (process.c:851) ==1937== by 0x42C7C0: request_receive (process.c:1414) ==1937== (1) [krb5] = ok ... (4) krb5 : Using client principal "hachmer@EXAMPLE.COM" ==1937== Invalid read of size 1 ==1937== at 0x4C2B050: mempcpy (mc_replace_strmem.c:1267) ==1937== by 0x6E4D95D: _IO_default_xsputn (in /lib64/libc-2.12.so) ==1937== by 0x6E2148F: vfprintf (in /lib64/libc-2.12.so) ==1937== by 0x6E486D9: vasprintf (in /lib64/libc-2.12.so) ==1937== by 0x6E28647: asprintf (in /lib64/libc-2.12.so) ==1937== by 0x7839591: k5_plugin_register_dyn (in /lib64/libkrb5.so.3.3) ==1937== by 0x783AD97: ??? (in /lib64/libkrb5.so.3.3) ==1937== by 0x783BCF9: ??? (in /lib64/libkrb5.so.3.3) ==1937== by 0x782AFC5: ??? (in /lib64/libkrb5.so.3.3) ==1937== by 0x782B908: krb5_init_creds_init (in /lib64/libkrb5.so.3.3) ==1937== by 0x782D806: ??? (in /lib64/libkrb5.so.3.3) ==1937== by 0x782F6A7: krb5_get_init_creds_password (in /lib64/libkrb5.so.3.3) ==1937== Address 0x908a9b0 is 0 bytes inside a block of size 24 free'd ==1937== at 0x4C273F0: free (vg_replace_malloc.c:446) ==1937== by 0x7830B2D: krb5_free_context (in /lib64/libkrb5.so.3.3) ==1937== by 0xD0274BD: krb5_auth (rlm_krb5.c:569) ==1937== by 0x41E709: modcall_recurse (modcall.c:311) ==1937== by 0x41F9F3: modcall_child (modcall.c:414) ==1937== by 0x41E3A6: modcall_recurse (modcall.c:780) ==1937== by 0x41F91C: modcall (modcall.c:1036) ==1937== by 0x41A8A2: indexed_modcall (modules.c:747) ==1937== by 0x40CFD0: rad_authenticate (auth.c:252) ==1937== by 0x42A96B: request_running (process.c:1211) ==1937== by 0x42A4DC: request_queue_or_run (process.c:851) ==1937== by 0x42C7C0: request_receive (process.c:1414) ==1937== ==1937== Invalid read of size 1 ==1937== at 0x4C2B063: mempcpy (mc_replace_strmem.c:1267) ==1937== by 0x6E4D95D: _IO_default_xsputn (in /lib64/libc-2.12.so) ==1937== by 0x6E2148F: vfprintf (in /lib64/libc-2.12.so) ==1937== by 0x6E486D9: vasprintf (in /lib64/libc-2.12.so) ==1937== by 0x6E28647: asprintf (in /lib64/libc-2.12.so) ==1937== by 0x7839591: k5_plugin_register_dyn (in /lib64/libkrb5.so.3.3) ==1937== by 0x783AD97: ??? (in /lib64/libkrb5.so.3.3) ==1937== by 0x783BCF9: ??? (in /lib64/libkrb5.so.3.3) ==1937== by 0x782AFC5: ??? (in /lib64/libkrb5.so.3.3) ==1937== by 0x782B908: krb5_init_creds_init (in /lib64/libkrb5.so.3.3) ==1937== by 0x782D806: ??? (in /lib64/libkrb5.so.3.3) ==1937== by 0x782F6A7: krb5_get_init_creds_password (in /lib64/libkrb5.so.3.3) ==1937== Address 0x908a9b2 is 2 bytes inside a block of size 24 free'd ==1937== at 0x4C273F0: free (vg_replace_malloc.c:446) ==1937== by 0x7830B2D: krb5_free_context (in /lib64/libkrb5.so.3.3) ==1937== by 0xD0274BD: krb5_auth (rlm_krb5.c:569) ==1937== by 0x41E709: modcall_recurse (modcall.c:311) ==1937== by 0x41F9F3: modcall_child (modcall.c:414) ==1937== by 0x41E3A6: modcall_recurse (modcall.c:780) ==1937== by 0x41F91C: modcall (modcall.c:1036) ==1937== by 0x41A8A2: indexed_modcall (modules.c:747) ==1937== by 0x40CFD0: rad_authenticate (auth.c:252) ==1937== by 0x42A96B: request_running (process.c:1211) ==1937== by 0x42A4DC: request_queue_or_run (process.c:851) ==1937== by 0x42C7C0: request_receive (process.c:1414) ==1937== (4) krb5 : Successfully retrieved and decrypted TGT (4) [krb5] = ok ... ^CSignalled to terminate Exiting normally. (7) Cleaning up request packet ID 43 with timestamp +181 ==1937== Invalid read of size 8 ==1937== at 0x5067C58: rbtree_walk (rbtree.c:716) ==1937== by 0x42D2B7: radius_event_free (process.c:4204) ==1937== by 0x4202FC: main (radiusd.c:519) ==1937== Address 0xfeb6f98 is 8 bytes inside a block of size 40 free'd ==1937== at 0x4C273F0: free (vg_replace_malloc.c:446) ==1937== by 0x5067B19: rbtree_delete_internal (rbtree.c:481) ==1937== by 0x506E024: fr_packet_list_yank (packet.c:554) ==1937== by 0x429499: request_done (process.c:500) ==1937== by 0x42A023: request_hash_cb (process.c:4165) ==1937== by 0x5067C87: rbtree_walk (rbtree.c:700) ==1937== by 0x42D2B7: radius_event_free (process.c:4204) ==1937== by 0x4202FC: main (radiusd.c:519) ==1937== ==1937== Invalid read of size 8 ==1937== at 0x5067CB9: rbtree_walk (rbtree.c:720) ==1937== by 0x42D2B7: radius_event_free (process.c:4204) ==1937== by 0x4202FC: main (radiusd.c:519) ==1937== Address 0xfeb6fa0 is 16 bytes inside a block of size 40 free'd ==1937== at 0x4C273F0: free (vg_replace_malloc.c:446) ==1937== by 0x5067B19: rbtree_delete_internal (rbtree.c:481) ==1937== by 0x506E024: fr_packet_list_yank (packet.c:554) ==1937== by 0x429499: request_done (process.c:500) ==1937== by 0x42A023: request_hash_cb (process.c:4165) ==1937== by 0x5067C87: rbtree_walk (rbtree.c:700) ==1937== by 0x42D2B7: radius_event_free (process.c:4204) ==1937== by 0x4202FC: main (radiusd.c:519) ==1937== ==1937== Invalid free() / delete / delete[] / realloc() ==1937== at 0x4C273F0: free (vg_replace_malloc.c:446) ==1937== by 0x7830B2D: krb5_free_context (in /lib64/libkrb5.so.3.3) ==1937== by 0xD027993: krb5_detach (rlm_krb5.c:117) ==1937== by 0x58B95A3: ??? (in /usr/lib64/libtalloc.so.2.0.7) ==1937== by 0x58B9322: ??? (in /usr/lib64/libtalloc.so.2.0.7) ==1937== by 0x5067E7C: FreeWalker (rbtree.c:76) ==1937== by 0x5067E5C: FreeWalker (rbtree.c:73) ==1937== by 0x5067E5C: FreeWalker (rbtree.c:73) ==1937== by 0x5067E5C: FreeWalker (rbtree.c:73) ==1937== by 0x5067E6D: FreeWalker (rbtree.c:74) ==1937== by 0x5067E6D: FreeWalker (rbtree.c:74) ==1937== by 0x5067E5C: FreeWalker (rbtree.c:73) ==1937== Address 0x908a9b0 is 0 bytes inside a block of size 24 free'd ==1937== at 0x4C273F0: free (vg_replace_malloc.c:446) ==1937== by 0x7830B2D: krb5_free_context (in /lib64/libkrb5.so.3.3) ==1937== by 0xD0274BD: krb5_auth (rlm_krb5.c:569) ==1937== by 0x41E709: modcall_recurse (modcall.c:311) ==1937== by 0x41F9F3: modcall_child (modcall.c:414) ==1937== by 0x41E3A6: modcall_recurse (modcall.c:780) ==1937== by 0x41F91C: modcall (modcall.c:1036) ==1937== by 0x41A8A2: indexed_modcall (modules.c:747) ==1937== by 0x40CFD0: rad_authenticate (auth.c:252) ==1937== by 0x42A96B: request_running (process.c:1211) ==1937== by 0x42A4DC: request_queue_or_run (process.c:851) ==1937== by 0x42C7C0: request_receive (process.c:1414) ==1937== rlm_ldap (ldap): Removing connection pool rlm_ldap (ldap): Closing connection (8) rlm_ldap (ldap): Closing connection (7) rlm_ldap (ldap): Closing connection (6) rlm_ldap (ldap): Closing connection (4) ASSERT FAILED src/main/connection.c[451]: this->in_use == false ==1937== ==1937== HEAP SUMMARY: ==1937== in use at exit: 2,757,485 bytes in 36,430 blocks ==1937== total heap usage: 296,722 allocs, 260,300 frees, 34,697,818 bytes allocated ==1937== ... ==1937== 117 (40 direct, 77 indirect) bytes in 1 blocks are definitely lost in loss record 665 of 1,396 ==1937== at 0x4C279EE: malloc (vg_replace_malloc.c:270) ==1937== by 0x7821C8A: krb5_build_principal_alloc_va (in /lib64/libkrb5.so.3.3) ==1937== by 0x7821DBE: krb5_build_principal (in /lib64/libkrb5.so.3.3) ==1937== by 0x785BD14: krb5_sname_to_principal (in /lib64/libkrb5.so.3.3) ==1937== by 0xD027A6E: krb5_instantiate (rlm_krb5.c:195) ==1937== by 0x41B0ED: find_module_instance (modules.c:620) ==1937== by 0x41C393: setup_modules (modules.c:1608) ==1937== by 0x419BC9: read_mainconfig (mainconfig.c:944) ==1937== by 0x41FFE1: main (radiusd.c:315) ... ==1937== LEAK SUMMARY: ==1937== definitely lost: 67 bytes in 2 blocks ==1937== indirectly lost: 77 bytes in 4 blocks ==1937== possibly lost: 651,755 bytes in 5,191 blocks ==1937== still reachable: 2,105,586 bytes in 31,233 blocks ==1937== suppressed: 0 bytes in 0 blocks ==1937== Reachable blocks (those to which a pointer was found) are not shown. ==1937== To see them, rerun with: --leak-check=full --show-reachable=yes ==1937== ==1937== For counts of detected and suppressed errors, rerun with: -v ==1937== Use --track-origins=yes to see where uninitialised values come from ==1937== ERROR SUMMARY: 993 errors from 640 contexts (suppressed: 144 from 9) ==1937== could not unlink /tmp/vgdb-pipe-from-vgdb-to-1937-by-root-on-radius-test ==1937== could not unlink /tmp/vgdb-pipe-to-vgdb-from-1937-by-root-on-radius-test ==1937== could not unlink /tmp/vgdb-pipe-shared-mem-vgdb-1937-by-root-on-radius-test Aborted Regards, Tobias Hachmer
thanks for the answer.
Nice assert! But that's not really related to the issue :(
*sigh* probably memory corruption could you run it under valgrind and see if there's any useful info? Complete output of valgrind is about 10000 lines. It's hard to me to find out which lines are the useful ones but I give it a try. See at the end of this post.
Could you post the full valgrind output anyway, just in case there is something else, either send it as an attachment to my email or put it in a gist.
Also when you build your RPMs could you make sure to pass --enable-developer to turn off optimization so we don't get all those <value optimized out> messages. I have built a new package with this configure parameter but there are still optimized out messages.
Weird, try: CFLAGS='-g3 -O0' ./configure <args>
(1) krb5 : Successfully retrieved and decrypted TGT ==1937== Invalid free() / delete / delete[] / realloc() ==1937== at 0x4C273F0: free (vg_replace_malloc.c:446) ==1937== by 0x7830B2D: krb5_free_context (in /lib64/libkrb5.so.3.3) ==1937== by 0xD0274BD: krb5_auth (rlm_krb5.c:569) ==1937== by 0x41E709: modcall_recurse (modcall.c:311) ==1937== by 0x41F9F3: modcall_child (modcall.c:414) ==1937== by 0x41E3A6: modcall_recurse (modcall.c:780) ==1937== by 0x41F91C: modcall (modcall.c:1036) ==1937== by 0x41A8A2: indexed_modcall (modules.c:747) ==1937== by 0x40CFD0: rad_authenticate (auth.c:252) ==1937== by 0x42A96B: request_running (process.c:1211) ==1937== by 0x42A4DC: request_queue_or_run (process.c:851) ==1937== by 0x42C7C0: request_receive (process.c:1414) ==1937== Address 0x908a9b0 is 0 bytes inside a block of size 24 free'd ==1937== at 0x4C273F0: free (vg_replace_malloc.c:446) ==1937== by 0x7830B2D: krb5_free_context (in /lib64/libkrb5.so.3.3) ==1937== by 0xD0274BD: krb5_auth (rlm_krb5.c:569) ==1937== by 0x41E709: modcall_recurse (modcall.c:311) ==1937== by 0x41F9F3: modcall_child (modcall.c:414) ==1937== by 0x41E3A6: modcall_recurse (modcall.c:780) ==1937== by 0x41F91C: modcall (modcall.c:1036) ==1937== by 0x41A8A2: indexed_modcall (modules.c:747) ==1937== by 0x40CFD0: rad_authenticate (auth.c:252) ==1937== by 0x42A96B: request_running (process.c:1211) ==1937== by 0x42A4DC: request_queue_or_run (process.c:851) ==1937== by 0x42C7C0: request_receive (process.c:1414) ==1937==
Hm, double free, and you see it again later when the destructor is freeing the main krb5 context. ==1937== Invalid free() / delete / delete[] / realloc() ==1937== at 0x4C273F0: free (vg_replace_malloc.c:446) ==1937== by 0x7830B2D: krb5_free_context (in /lib64/libkrb5.so.3.3) ==1937== by 0xD027993: krb5_detach (rlm_krb5.c:117) ==1937== by 0x58B95A3: ??? (in /usr/lib64/libtalloc.so.2.0.7) ==1937== by 0x58B9322: ??? (in /usr/lib64/libtalloc.so.2.0.7) ==1937== by 0x5067E7C: FreeWalker (rbtree.c:76) ==1937== by 0x5067E5C: FreeWalker (rbtree.c:73) ==1937== by 0x5067E5C: FreeWalker (rbtree.c:73) ==1937== by 0x5067E5C: FreeWalker (rbtree.c:73) ==1937== by 0x5067E6D: FreeWalker (rbtree.c:74) ==1937== by 0x5067E6D: FreeWalker (rbtree.c:74) ==1937== by 0x5067E5C: FreeWalker (rbtree.c:73) ==1937== Address 0x908a9b0 is 0 bytes inside a block of size 24 free'd ==1937== at 0x4C273F0: free (vg_replace_malloc.c:446) ==1937== by 0x7830B2D: krb5_free_context (in /lib64/libkrb5.so.3.3) ==1937== by 0xD0274BD: krb5_auth (rlm_krb5.c:569) ==1937== by 0x41E709: modcall_recurse (modcall.c:311) ==1937== by 0x41F9F3: modcall_child (modcall.c:414) ==1937== by 0x41E3A6: modcall_recurse (modcall.c:780) ==1937== by 0x41F91C: modcall (modcall.c:1036) ==1937== by 0x41A8A2: indexed_modcall (modules.c:747) ==1937== by 0x40CFD0: rad_authenticate (auth.c:252) ==1937== by 0x42A96B: request_running (process.c:1211) ==1937== by 0x42A4DC: request_queue_or_run (process.c:851) ==1937== by 0x42C7C0: request_receive (process.c:1414) ==1937== Interestingly and importantly if you look at the calls on detach, the block of memory being freed twice, was originally freed by a request! My guess is krb5_copy_context isn't duplicating a bit of memory it needs to. Could you comment out krb5_free_context(context) at line 448 in rlm_krb5.c? Rebuild, and run a few requests through it. My guess is it won't SEGV anymore. It shouldn't be that hard to figure out what's not being duplicated if that's really the problem. -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS Development Team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
-----Ursprüngliche Nachricht----- Von: freeradius-users-bounces+tobias.hachmer=stadt-frankfurt.de@lists.freeradius.org [mailto:freeradius-users-bounces+tobias.hachmer=stadt-frankfurt.de@lists.freeradius.org] Im Auftrag von Arran Cudbard-Bell Gesendet: Donnerstag, 12. Dezember 2013 14:04 An: FreeRadius users mailing list Betreff: Re: received signal SIGSEGV, Segmentation fault. malloc_consolidate (av=0x7ffff5e2de80) at malloc.c:5196
thanks for the answer. Nice assert! But that's not really related to the issue :( That thanks?
*sigh* probably memory corruption could you run it under valgrind and see if there's any useful info? Complete output of valgrind is about 10000 lines. It's hard to me to find out which lines are the useful ones but I give it a try. See at the end of this post. Could you post the full valgrind output anyway, just in case there is something else, either send it as an attachment to my email or put it in a gist. Sure, will send this directly to you ...
Also when you build your RPMs could you make sure to pass --enable-developer to turn off optimization so we don't get all those <value optimized out> messages. I have built a new package with this configure parameter but there are still optimized out messages. Weird, try: CFLAGS='-g3 -O0' ./configure <args> That has worked.
Interestingly and importantly if you look at the calls on detach, the block of memory being freed twice, was originally freed by a request! My guess is krb5_copy_context isn't duplicating a bit of memory it needs to. Could you comment out krb5_free_context(context) at line 448 in rlm_krb5.c? Rebuild, and run a few requests through it. Yes, my provided bt and valgrind output is based on
radiusd: FreeRADIUS Version 3.0.1 (git #e86a529), for host x86_64-redhat-linux-gnu, built on Dec 12 2013 at 14:19:30 + the line you mentioned commented out. Regards, Tobias BT: rad_recv: Access-Request packet from host 127.0.0.1 port 42335, id=16, length=103 User-Name = 'hachmer' User-Password = 'pass' NAS-IP-Address = 127.0.0.1 NAS-Port = 0 Message-Authenticator = 0xecaf11b4272d31821075a076004c4808 (4) # Executing section authorize from file /etc/raddb/sites-enabled/default (4) authorize { (4) filter_username filter_username { (4) ? if (User-Name != "%{tolower:%{User-Name}}") Program received signal SIGSEGV, Segmentation fault. 0x00007ffff5b04285 in malloc_consolidate () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install freeradius-3.0.1-9.el6.x86_64 (gdb) bt #0 0x00007ffff5b04285 in malloc_consolidate () from /lib64/libc.so.6 #1 0x00007ffff5b07405 in _int_malloc () from /lib64/libc.so.6 #2 0x00007ffff5b08991 in malloc () from /lib64/libc.so.6 #3 0x00007ffff714063f in _talloc_array () from /usr/lib64/libtalloc.so.2 #4 0x00007ffff7bcedcf in xlat_aprint (ctx=0x89e5f0, request=0x89e5f0, node=0x998810, escape=0, escape_ctx=0x0, lvl=0) at src/main/xlat.c:1895 #5 0x00007ffff7bcf0c7 in xlat_process (out=0x7fffffffc2c0, request=0x89e5f0, head=0x998810, escape=0, escape_ctx=0x0) at src/main/xlat.c:1976 #6 0x00007ffff7bcf462 in xlat_expand (out=0x7fffffffc3f8, outlen=0, request=0x89e5f0, fmt=0x83dff0 "%{tolower:%{User-Name}}", escape=0, escape_ctx=0x0) at src/main/xlat.c:2068 #7 0x00007ffff7bcf5f6 in radius_axlat (out=0x7fffffffc3f8, request=0x89e5f0, fmt=0x83dff0 "%{tolower:%{User-Name}}", escape=0, ctx=0x0) at src/main/xlat.c:2095 #8 0x00007ffff7bc88ec in radius_expand_tmpl (out=0x7fffffffc3f8, request=0x89e5f0, vpt=0x83df60) at src/main/evaluate.c:108 #9 0x00007ffff7bc92bc in radius_evaluate_map (request=0x89e5f0, modreturn=10, depth=0, c=0x83dc60) at src/main/evaluate.c:424 #10 0x00007ffff7bc9800 in radius_evaluate_cond (request=0x89e5f0, modreturn=10, depth=0, c=0x83dc60) at src/main/evaluate.c:597 #11 0x0000000000422929 in modcall_recurse (request=0x89e5f0, component=RLM_COMPONENT_AUTZ, depth=2, entry=0x7fffffffd770) at src/main/modcall.c:480 #12 0x00000000004226f3 in modcall_child (request=0x89e5f0, component=RLM_COMPONENT_AUTZ, depth=2, entry=0x7fffffffd758, c=0xa2ad60, result=0x7fffffffcfd8) at src/main/modcall.c:414 #13 0x0000000000423beb in modcall_recurse (request=0x89e5f0, component=RLM_COMPONENT_AUTZ, depth=1, entry=0x7fffffffd758) at src/main/modcall.c:780 #14 0x00000000004226f3 in modcall_child (request=0x89e5f0, component=RLM_COMPONENT_AUTZ, depth=1, entry=0x7fffffffd740, c=0xa2ace0, result=0x7fffffffd618) at src/main/modcall.c:414 #15 0x0000000000423beb in modcall_recurse (request=0x89e5f0, component=RLM_COMPONENT_AUTZ, depth=0, entry=0x7fffffffd740) at src/main/modcall.c:780 #16 0x00000000004249c0 in modcall (component=RLM_COMPONENT_AUTZ, c=0xa2c6a0, request=0x89e5f0) at src/main/modcall.c:1036 #17 0x000000000042015e in indexed_modcall (comp=RLM_COMPONENT_AUTZ, idx=0, request=0x89e5f0) at src/main/modules.c:747 #18 0x0000000000421dfc in process_authorize (autz_type=0, request=0x89e5f0) at src/main/modules.c:1647 #19 0x000000000040d1a3 in rad_authenticate (request=0x89e5f0) at src/main/auth.c:409 #20 0x0000000000432641 in request_running (request=0x89e5f0, action=1) at src/main/process.c:1211 #21 0x00000000004318a9 in request_queue_or_run (request=0x89e5f0, process=0x432576 <request_running>) at src/main/process.c:851 #22 0x0000000000432d59 in request_receive (listener=0xa43750, packet=0x9368c0, client=0x82a370, fun=0x40cff4 <rad_authenticate>) at src/main/process.c:1414 #23 0x000000000041442f in auth_socket_recv (listener=0xa43750) at src/main/listen.c:1509 #24 0x00000000004386fc in event_socket_handler (xel=0xa2de10, fd=12, ctx=0xa43750) at src/main/process.c:3499 #25 0x00007ffff79a54dc in fr_event_loop (el=0xa2de10) at src/lib/event.c:417 #26 0x00000000004398b7 in radius_event_process () at src/main/process.c:4216 #27 0x0000000000427d1e in main (argc=2, argv=0x7fffffffe678) at src/main/radiusd.c:479
BT: rad_recv: Access-Request packet from host 127.0.0.1 port 42335, id=16, length=103 User-Name = 'hachmer' User-Password = 'pass' NAS-IP-Address = 127.0.0.1 NAS-Port = 0 Message-Authenticator = 0xecaf11b4272d31821075a076004c4808 (4) # Executing section authorize from file /etc/raddb/sites-enabled/default (4) authorize { (4) filter_username filter_username { (4) ? if (User-Name != "%{tolower:%{User-Name}}")
Program received signal SIGSEGV, Segmentation fault. 0x00007ffff5b04285 in malloc_consolidate () from /lib64/libc.so.6
*sigh* It's a double free in libkrb5. They free ctx->plugin_base_dir in krb5_free_context, but don't strdup it in krb5_copy_context. The proper struct is hidden, only the type is exposed FreeRADIUS Development Team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
On 12 Dec 2013, at 18:00, Arran Cudbard-Bell <a.cudbardb@freeradius.org> wrote:
BT: rad_recv: Access-Request packet from host 127.0.0.1 port 42335, id=16, length=103 User-Name = 'hachmer' User-Password = 'pass' NAS-IP-Address = 127.0.0.1 NAS-Port = 0 Message-Authenticator = 0xecaf11b4272d31821075a076004c4808 (4) # Executing section authorize from file /etc/raddb/sites-enabled/default (4) authorize { (4) filter_username filter_username { (4) ? if (User-Name != "%{tolower:%{User-Name}}")
Program received signal SIGSEGV, Segmentation fault. 0x00007ffff5b04285 in malloc_consolidate () from /lib64/libc.so.6
*sigh* It's a double free in libkrb5.
They free ctx->plugin_base_dir in krb5_free_context, but don't strdup it in krb5_copy_context.
The proper struct is hidden, only the type is exposed
Sorry whacked send too early: The proper struct is hidden, only the type is exposed, so there's no way to calculate the offset of that field in the struct, and therefore no way to fix this problem from outside of the ctx_copy function. I'll submit another bug report, and another fix, and then I guess, add some logic to the configure script to automatically disable thread support for MIT kerberos in anything less than the absolute latest library version (provided they merge the patches for the next release). Then for the packages, i'll set the default dependency to Heimdal (after doing some testing with Heimdal to make sure it doesn't have similar issues). If you want a quick fix to get something working you can either patch your version of the kerberos library, adding: if (ctx->plugin_base_dir) nctx->plugin_base_dir = strdup(ctx->plugin_base_dir); Just above the if (ctx->os_context.default_ccname != NULL) { line in src/lib/krb5/krb/copy_ctx.c. Or on the FreeRADIUS side, edit src/modules/rlm_krb5/all.mk and remove -DKRB5_IS_THREAD_SAFE from the SRC_CFLAGS. Or switch to using the Heimdal library. I'm not sure why it worked on Alan Buxey's server, maybe he was using a very old version of the library which didn't have these defects. -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS Development Team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
On 12 Dec 2013, at 18:16, Arran Cudbard-Bell <a.cudbardb@freeradius.org> wrote:
On 12 Dec 2013, at 18:00, Arran Cudbard-Bell <a.cudbardb@freeradius.org> wrote:
BT: rad_recv: Access-Request packet from host 127.0.0.1 port 42335, id=16, length=103 User-Name = 'hachmer' User-Password = 'pass' NAS-IP-Address = 127.0.0.1 NAS-Port = 0 Message-Authenticator = 0xecaf11b4272d31821075a076004c4808 (4) # Executing section authorize from file /etc/raddb/sites-enabled/default (4) authorize { (4) filter_username filter_username { (4) ? if (User-Name != "%{tolower:%{User-Name}}")
Program received signal SIGSEGV, Segmentation fault. 0x00007ffff5b04285 in malloc_consolidate () from /lib64/libc.so.6
*sigh* It's a double free in libkrb5.
They free ctx->plugin_base_dir in krb5_free_context, but don't strdup it in krb5_copy_context.
The proper struct is hidden, only the type is exposed
Sorry whacked send too early:
The proper struct is hidden, only the type is exposed, so there's no way to calculate the offset of that field in the struct, and therefore no way to fix this problem from outside of the ctx_copy function.
I'll submit another bug report
Ok here are the pull requests for MIT krb5, i'll do the configure and packaging scripts tomorrow. https://github.com/krb5/krb5/pull/36 https://github.com/krb5/krb5/pull/37 https://github.com/krb5/krb5/pull/38 -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS Development Team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
-----Ursprüngliche Nachricht----- Von: freeradius-users-bounces+tobias.hachmer=stadt-frankfurt.de@lists.freeradius.org [mailto:freeradius-users-bounces+tobias.hachmer=stadt-frankfurt.de@lists.freeradius.org] Im Auftrag von Arran Cudbard-Bell Gesendet: Freitag, 13. Dezember 2013 01:48 An: FreeRadius users mailing list Betreff: Re: received signal SIGSEGV, Segmentation fault. malloc_consolidate (av=0x7ffff5e2de80) at malloc.c:5196
I'll submit another bug report Ok here are the pull requests for MIT krb5, i'll do the configure and packaging scripts tomorrow. https://github.com/krb5/krb5/pull/36 https://github.com/krb5/krb5/pull/37 https://github.com/krb5/krb5/pull/38
I have patched krb5 on CentOS 6.5 and first tests seems to be fine! CentOS 6.5 uses krb5 1.10.3 and I had to adopt the function names: k5_copy_etypes k5_copy_etypes to krb5int_copy_etypes krb5int_copy_etypes Regards, Tobias
On 13 Dec 2013, at 08:07, Hachmer, Tobias <Tobias.Hachmer@stadt-frankfurt.de> wrote:
-----Ursprüngliche Nachricht----- Von: freeradius-users-bounces+tobias.hachmer=stadt-frankfurt.de@lists.freeradius.org [mailto:freeradius-users-bounces+tobias.hachmer=stadt-frankfurt.de@lists.freeradius.org] Im Auftrag von Arran Cudbard-Bell Gesendet: Freitag, 13. Dezember 2013 01:48 An: FreeRadius users mailing list Betreff: Re: received signal SIGSEGV, Segmentation fault. malloc_consolidate (av=0x7ffff5e2de80) at malloc.c:5196
I'll submit another bug report Ok here are the pull requests for MIT krb5, i'll do the configure and packaging scripts tomorrow. https://github.com/krb5/krb5/pull/36 https://github.com/krb5/krb5/pull/37 https://github.com/krb5/krb5/pull/38
I have patched krb5 on CentOS 6.5 and first tests seems to be fine! CentOS 6.5 uses krb5 1.10.3 and I had to adopt the function names:
k5_copy_etypes k5_copy_etypes
to
krb5int_copy_etypes krb5int_copy_etypes
OK :) Glad it works. I've modified the configure scripts to disable threading on kerberos <= 1.11.4, but it's not perfect due to the wide range of version formats that seem to be in use. Leaving packages for now until I can determine whether it'd break other things on suse/redhat/debian. Thread support shouldn't matter unless you're going to be loading the system quite heavily. If you want to enable it manually, add '-DKRB5_IS_THREAD_SAFE' to all.mk. Thanks for reporting the bugs. -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS Development Team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
-----Ursprüngliche Nachricht----- Von: freeradius-users-bounces+tobias.hachmer=stadt-frankfurt.de@lists.freeradius.org [mailto:freeradius-users-bounces+tobias.hachmer=stadt-frankfurt.de@lists.freeradius.org] Im Auftrag von Arran Cudbard-Bell Gesendet: Donnerstag, 12. Dezember 2013 19:17 An: FreeRadius users mailing list Betreff: Re: received signal SIGSEGV, Segmentation fault. malloc_consolidate (av=0x7ffff5e2de80) at malloc.c:5196
The proper struct is hidden, only the type is exposed, so there's no way to calculate the offset of that field in the struct, and therefore no way to fix this problem from outside of the ctx_copy function. I'll submit another bug report, and another fix, and then I guess, add some logic to the configure script to automatically disable thread support for MIT kerberos in anything less than the absolute latest library version (provided they merge the patches for the next release). Then for the packages, i'll set the default dependency to Heimdal (after doing some testing with Heimdal to make sure it doesn't have similar issues). If you want a quick fix to get something working you can either patch your version of the kerberos library, adding:
if (ctx->plugin_base_dir) nctx->plugin_base_dir = strdup(ctx->plugin_base_dir);
Just above the if (ctx->os_context.default_ccname != NULL) { line in src/lib/krb5/krb/copy_ctx.c. Or on the FreeRADIUS side, edit src/modules/rlm_krb5/all.mk and remove -DKRB5_IS_THREAD_SAFE from the SRC_CFLAGS. Or switch to using the Heimdal library.
Thank you Arran for your investigations! It's fine to know what's the crux of the matter and how to work around this. Regards, Tobias
participants (2)
-
Arran Cudbard-Bell -
Hachmer, Tobias