2.1.12 segfault with ldap module
While setting up freeradius recently, we noticed that it would segfault on a restart after being used. The segfault line would be something like: kernel: radiusd[2608]: segfault at 7feb7471f480 ip 00007feb7471f480 sp 00007fff451ea168 error 14 in libfreeradius-eap-2.1.12.so[7feb74d2b000+9000] At first I thought this would be something in the eap module, but after building a debug build and running it in gdb I got this backtrace: Program received signal SIGSEGV, Segmentation fault. 0x00007ffff3e0f480 in ?? () (gdb) bt #0 0x00007ffff3e0f480 in ?? () #1 0x00007ffff71d9dea in ?? () from /usr/lib64/libcrypto.so.1.0.0 #2 0x00007ffff755a12d in SSL_free () from /usr/lib64/libssl.so.1.0.0 #3 0x00007ffff548b1da in ?? () from /usr/lib64/libldap_r-2.4.so.2 #4 0x00007ffff524df59 in ber_sockbuf_remove_io () from /usr/lib64/liblber-2.4.so.2 #5 0x00007ffff524dfed in ber_int_sb_destroy () from /usr/lib64/liblber-2.4.so.2 #6 0x00007ffff524e06c in ber_sockbuf_free () from /usr/lib64/liblber-2.4.so.2 #7 0x00007ffff546fb2c in ldap_ld_free () from /usr/lib64/libldap_r-2.4.so.2 #8 0x00007ffff56a50c0 in ldap_detach (instance=<value optimized out>) at rlm_ldap.c:2588 #9 0x00000000004186ce in module_instance_free (data=<value optimized out>) at modules.c:380 #10 0x00007ffff7bcee4d in FreeWalker (tree=0x796a50, X=0x799670) at rbtree.c:63 #11 0x00007ffff7bcee8e in rbtree_free (tree=0x9b14c0) at rbtree.c:74 #12 0x0000000000419450 in detach_modules () at modules.c:428 #13 0x000000000041d13c in main (argc=<value optimized out>, argv=<value optimized out>) at radiusd.c:456 After seeing that it looks like it's actually the ldap module causing problems, I commented out all of my bits that use the ldap module and it doesn't segfault on shutdown. I also tried it with ldap but without TLS enabled and it shut down clean still. Has anyone seen this before? Any solutions? For the time being, I guess I'll run the ldap module without TLS enabled. Segfaults, even benign ones, seem to get everyone antsy...
Brian De Wolf wrote:
After seeing that it looks like it's actually the ldap module causing problems, I commented out all of my bits that use the ldap module and it doesn't segfault on shutdown. I also tried it with ldap but without TLS enabled and it shut down clean still.
OK... then it's clearly LDAP.
Has anyone seen this before? Any solutions? For the time being, I guess I'll run the ldap module without TLS enabled. Segfaults, even benign ones, seem to get everyone antsy...
From a quick inspection, the cause is likely that a connection is still in use by the module, but in another thread. When it releases a connection that's still in use, bad things happen. I've committed a fix which avoids the problem. The side effect is that the server exits while LDAP connections are still open. It's imperfect, but there are few better choices. Alan DeKok.
On Tue, 21 Feb 2012 01:09:19 -0800 Alan DeKok <aland@deployingradius.com> wrote:
From a quick inspection, the cause is likely that a connection is still in use by the module, but in another thread. When it releases a connection that's still in use, bad things happen.
Would that still be possible if I was starting with -t? This is in a test environment, so I'm certain there weren't any clients being processed at the time of restart.
I've committed a fix which avoids the problem. The side effect is that the server exits while LDAP connections are still open. It's imperfect, but there are few better choices.
Hmm, I pulled the two relevant patches but I still get the segfault: (gdb) run -t -f -d /etc/raddb Starting program: /tmp/radiusd/sbin/radiusd -t -f -d /etc/raddb [Thread debugging using libthread_db enabled] [New Thread 0x7ffff2eeb700 (LWP 1219)] [Thread 0x7ffff2eeb700 (LWP 1219) exited] Program received signal SIGTERM, Terminated. 0x00007ffff6694f13 in select () from /lib64/libc.so.6 (gdb) continue Continuing. Program received signal SIGSEGV, Segmentation fault. 0x00007ffff3e0f480 in ?? () (gdb) bt #0 0x00007ffff3e0f480 in ?? () #1 0x00007ffff71d9dea in ?? () from /usr/lib64/libcrypto.so.1.0.0 #2 0x00007ffff755a12d in SSL_free () from /usr/lib64/libssl.so.1.0.0 #3 0x00007ffff548b1da in ?? () from /usr/lib64/libldap_r-2.4.so.2 #4 0x00007ffff524df59 in ber_sockbuf_remove_io () from /usr/lib64/liblber-2.4.so.2 #5 0x00007ffff524dfed in ber_int_sb_destroy () from /usr/lib64/liblber-2.4.so.2 #6 0x00007ffff524e06c in ber_sockbuf_free () from /usr/lib64/liblber-2.4.so.2 #7 0x00007ffff546fb2c in ldap_ld_free () from /usr/lib64/libldap_r-2.4.so.2 #8 0x00007ffff56a50eb in ldap_detach (instance=<value optimized out>) at rlm_ldap.c:2590 #9 0x00000000004186fe in module_instance_free (data=<value optimized out>) at modules.c:383 #10 0x00007ffff7bcee4d in FreeWalker (tree=0x7973e0, X=0x79a000) at rbtree.c:63 #11 0x00007ffff7bcee8e in rbtree_free (tree=0x9b1e10) at rbtree.c:74 #12 0x0000000000419480 in detach_modules () at modules.c:431 #13 0x000000000041d17c in main (argc=<value optimized out>, argv=<value optimized out>) at radiusd.c:456 Interestingly, I tried running it with -X and it fails elsewhere (I didn't try this before the patches so it might be the same, sorry): (gdb) run -X -t -f -d /etc/raddb ... Program received signal SIGTERM, Terminated. 0x00007ffff6694f13 in select () from /lib64/libc.so.6 (gdb) continue Continuing. Polling for detail file /var/log/radius/radacct/forward/detail-* Detail listener /var/log/radius/radacct/forward/detail-* state unopened signalled 0 waiting 0.800610 sec Waking up in 0.8 seconds. Signalled to terminate Exiting normally. Program received signal SIGSEGV, Segmentation fault. rbtree_find (tree=0x7bfbc0, Data=0x9c1ac0) at rbtree.c:479 479 int result = tree->Compare(Data, Current->Data); (gdb) bt #0 rbtree_find (tree=0x7bfbc0, Data=0x9c1ac0) at rbtree.c:479 #1 0x00007ffff7bcf0f9 in rbtree_deletebydata (tree=0x7bfbc0, data=0x9c1ac0) at rbtree.c:457 #2 0x00007ffff462a894 in eap_handler_free (inst=0x7a0030, handler=0x9c1ac0) at mem.c:154 #3 0x00007ffff462afc4 in eaplist_free (inst=0x7a0030) at mem.c:266 #4 0x00007ffff4628cbf in eap_detach (instance=<value optimized out>) at rlm_eap.c:66 #5 0x00000000004186fe in module_instance_free (data=<value optimized out>) at modules.c:383 #6 0x00007ffff7bcee4d in FreeWalker (tree=0x7972d0, X=0x7bc660) at rbtree.c:63 #7 0x00007ffff7bcee2d in FreeWalker (tree=0x7972d0, X=0x799ef0) at rbtree.c:60 #8 0x00007ffff7bcee8e in rbtree_free (tree=0x7bfbc0) at rbtree.c:74 #9 0x0000000000419480 in detach_modules () at modules.c:431 #10 0x000000000041d17c in main (argc=<value optimized out>, argv=<value optimized out>) at radiusd.c:456 I tried it again with "run -X -t -f -d /etc/raddb" and it gave a backtrace that pointed at LDAP again, but it was also able to shut down all of the rlm_sqsl_mysql handles before segfaulting. Is there any other output/debug stuff that would be helpful?
participants (2)
-
Alan DeKok -
Brian De Wolf