Segmentation fault with LDAP authentication and rate control?
Hi all, We're using 2.2.0 on RHEL 6.2 using LDAPS as authentication backend. The servers are running well over 2 years until recently. The server fails with SIGSEGV or SIGABRT whenever there is 'burst' of authentication requests, say over 100 requests in the same second. In the SIGSEGV case, coredump shows: --------------------- cut here ---------------------------- Core was generated by `/usr/local/sbin/radiusd -d /usr/local/etc/raddb'. Program terminated with signal 11, Segmentation fault. #0 0x00007f402afa9b89 in ldap_connect (instance=0xd24cf0, dn=0x7f3f98022818 "uid=1234576,dc=cuhk,dc=edu,dc=hk", password=0x..... "hideme", auth=1, result=0x7f401d780fac, err=0x0) at rlm_ldap.c:2410 2410 if (strcmp(TLS_DEFAULT_VERIFY, inst->tls_require_cert ) != 0 ) { Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-13.el6.x86_64 glibc-2.12-1.107.el6_4.5.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-10.el6_4.3.x86_64 libcom_err-1.41.12-11.el6.x86_64 libgcc-4.4.6-3.el6.x86_64 libselinux-2.0.94-5.2.el6.x86_64 mysql-libs-5.1.69-1.el6_4.x86_64 nspr-4.9.5-2.el6_4.x86_64 nss-3.14.3-4.el6_4.x86_64 nss-softokn-3.14.3-3.el6_4.x86_64 nss-softokn-freebl-3.14.3-3.el6_4.x86_64 nss-util-3.14.3-3.el6_4.x86_64 openldap-2.4.23-32.el6_4.1.x86_64 openssl-1.0.0-27.el6_4.2.x86_64 sqlite-3.6.20-1.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007f402afa9b89 in ldap_connect (instance=0xd24cf0, dn=0x7f3f98022818 " uid=1234576,dc=cuhk,dc=edu,dc=hk", password=0x..... "hideme", auth=1, result=0x7f401d780fac, err=0x0) at rlm_ldap.c:2410 #1 0x00007f402afacecf in ldap_authenticate (instance=0xd24cf0, request=0xd8bfe0) at rlm_ldap.c:1938 #2 0x000000000041c742 in call_modsingle (component=0, c=<value optimized out>, request=<value optimized out>) at modcall.c:304 #3 modcall (component=0, c=<value optimized out>, request=<value optimized out>) at modcall.c:686 #4 0x0000000000418c83 in indexed_modcall (comp=0, idx=7980103, request=0xd8bfe0) at modules.c:740 #5 0x000000000040941c in rad_check_password (request=0xd8bfe0) at auth.c:382 #6 rad_authenticate (request=0xd8bfe0) at auth.c:667 #7 0x0000000000428365 in radius_handle_request (request=0xd8bfe0, fun=0x408ae0 <rad_authenticate>) at event.c:3784 #8 0x00000000004204cb in request_handler_thread (arg=0xd8f700) at threads.c:537 #9 0x0000003a12c07851 in start_thread () from /lib64/libpthread.so.0 #10 0x0000003a124e894d in clone () from /lib64/libc.so.6 (gdb) frame 0 #0 0x00007f402afa9b89 in ldap_connect (instance=0xd24cf0, dn=0x7f3f98022818 " uid=1234576,dc=cuhk,dc=edu,dc=hk", password=0x..... "hideme", auth=1, result=0x7f401d780fac, err=0x0) at rlm_ldap.c:2410 2410 if (strcmp(TLS_DEFAULT_VERIFY, inst->tls_require_cert ) != 0 ) { (gdb) info locals inst = 0xd24cf0 ld = 0x7f3f980238d0 msgid = <value optimized out> rc = <value optimized out> ldap_version = 3 ldap_errno = 0 res = <value optimized out> tv = {tv_sec = 1, tv_usec = 0} (gdb) info args instance = 0xd24cf0 dn = 0x7f3f98022818 "uid=1234567,dc=cuhk,dc=edu,dc=hk" password=0x..... "hideme"auth = 1 result = 0x7f401d780fac err = 0x0 (gdb) print inst->tls_require_cert $1 = 0x0 (gdb) print inst->xlat_name $2 = 0xd1d110 "ldap" --------------------- cut here ---------------------------- inst->tls_require_cert becomes null while we've 'require_cert = "never"' defined in modules/ldap. There is no problem when the loading is not high: Mon Oct 28 09:10:48 2013 : Debug: [ldap] setting TLS mode to 1 Mon Oct 28 09:10:48 2013 : Debug: [ldap] setting TLS CACert File to /usr/local/etc/raddb/certs/cacert.pem Mon Oct 28 09:10:48 2013 : Debug: [ldap] setting TLS Require Cert to never Would anyone please advise? Thanks a lot. Besides, as we must use LDAP for authentication backend which maybe a bottleneck, is it feasible to have kind of QoS/rate control for incoming authentication requests, and/or some LDAP authentication result caching? Sorry for the naïve questions. Thanks again. Best Regards, /ST Wong
ST Wong (ITSC) wrote:
We’re using 2.2.0 on RHEL 6.2 using LDAPS as authentication backend. The servers are running well over 2 years until recently. The server fails with SIGSEGV or SIGABRT whenever there is ‘burst’ of authentication requests, say over 100 requests in the same second. In the SIGSEGV case, coredump shows: ... 2410 if (strcmp(TLS_DEFAULT_VERIFY, inst->tls_require_cert ) != 0 ) {
TLS_DEFAULT_VERIFY is a static string. int->tls_require_cert is initialized when the module loads, and s never changed. The issue is some kind of memory corruption. i.e. it's not this line, it's elsewhere.
Besides, as we must use LDAP for authentication backend which maybe a bottleneck, is it feasible to have kind of QoS/rate control for incoming authentication requests, and/or some LDAP authentication result caching? Sorry for the naïve questions. Thanks again.
You can't do LDAP result caching, sorry. There's also no ability to do rate control on incoming authentication. You may try the v2.x.x branch from git. It may have fixes which solve this issue. Alan DeKok.
TLS_DEFAULT_VERIFY is a static string. int->tls_require_cert is initialized when the module loads, and s never changed.
The issue is some kind of memory corruption. i.e. it's not this line, it's elsewhere.
Right. It's strange that inst->tls_require_cert becomes null (or undefined) in the middle (yes, due to memory corruption and caused segmentation fault). Any hint on how to locate the line(s) that cause the problem? I'm playing with valgrind but not sure if it's the way to go. Thanks. /ST -----Original Message----- From: freeradius-users-bounces+st=itsc.cuhk.edu.hk@lists.freeradius.org [mailto:freeradius-users-bounces+st=itsc.cuhk.edu.hk@lists.freeradius.org] On Behalf Of Alan DeKok Sent: Tuesday, October 29, 2013 9:58 PM To: FreeRadius users mailing list Subject: Re: Segmentation fault with LDAP authentication and rate control? ST Wong (ITSC) wrote:
We’re using 2.2.0 on RHEL 6.2 using LDAPS as authentication backend. The servers are running well over 2 years until recently. The server fails with SIGSEGV or SIGABRT whenever there is ‘burst’ of authentication requests, say over 100 requests in the same second. In the SIGSEGV case, coredump shows: ... 2410 if (strcmp(TLS_DEFAULT_VERIFY, inst->tls_require_cert ) != 0 ) {
TLS_DEFAULT_VERIFY is a static string. int->tls_require_cert is initialized when the module loads, and s never changed. The issue is some kind of memory corruption. i.e. it's not this line, it's elsewhere.
Besides, as we must use LDAP for authentication backend which maybe a bottleneck, is it feasible to have kind of QoS/rate control for incoming authentication requests, and/or some LDAP authentication result caching? Sorry for the naïve questions. Thanks again.
You can't do LDAP result caching, sorry. There's also no ability to do rate control on incoming authentication. You may try the v2.x.x branch from git. It may have fixes which solve this issue. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
ST Wong (ITSC) wrote:
Right. It's strange that inst->tls_require_cert becomes null (or undefined) in the middle (yes, due to memory corruption and caused segmentation fault). Any hint on how to locate the line(s) that cause the problem? I'm playing with valgrind but not sure if it's the way to go. Thanks.
Use valgrind. There's really not much else you can do. Alan DeKok.
participants (2)
-
Alan DeKok -
ST Wong (ITSC)