Segmentation fault with LDAP authentication and rate control?

[ST Wong (ITSC)]

Hi all,

 

We're using 2.2.0 on RHEL 6.2 using LDAPS as authentication backend.  The servers are running well over 2 years until recently.  The server fails with SIGSEGV or SIGABRT whenever there is 'burst' of authentication requests, say over 100 requests in the same second.   In the SIGSEGV case, coredump shows:

 

--------------------- cut here ----------------------------

Core was generated by `/usr/local/sbin/radiusd -d /usr/local/etc/raddb'.

Program terminated with signal 11, Segmentation fault.

#0  0x00007f402afa9b89 in ldap_connect (instance=0xd24cf0,

    dn=0x7f3f98022818 "uid=1234576,dc=cuhk,dc=edu,dc=hk", password=0x..... "hideme", auth=1, result=0x7f401d780fac,

    err=0x0) at rlm_ldap.c:2410

2410            if (strcmp(TLS_DEFAULT_VERIFY, inst->tls_require_cert ) != 0 ) {

Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-13.el6.x86_64 glibc-2.12-1.107.el6_4.5.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-10.el6_4.3.x86_64 libcom_err-1.41.12-11.el6.x86_64 libgcc-4.4.6-3.el6.x86_64 libselinux-2.0.94-5.2.el6.x86_64 mysql-libs-5.1.69-1.el6_4.x86_64 nspr-4.9.5-2.el6_4.x86_64 nss-3.14.3-4.el6_4.x86_64 nss-softokn-3.14.3-3.el6_4.x86_64 nss-softokn-freebl-3.14.3-3.el6_4.x86_64 nss-util-3.14.3-3.el6_4.x86_64 openldap-2.4.23-32.el6_4.1.x86_64 openssl-1.0.0-27.el6_4.2.x86_64 sqlite-3.6.20-1.el6.x86_64 zlib-1.2.3-29.el6.x86_64

(gdb) bt

#0  0x00007f402afa9b89 in ldap_connect (instance=0xd24cf0,

    dn=0x7f3f98022818 " uid=1234576,dc=cuhk,dc=edu,dc=hk", password=0x..... "hideme", auth=1, result=0x7f401d780fac,

    err=0x0) at rlm_ldap.c:2410

#1  0x00007f402afacecf in ldap_authenticate (instance=0xd24cf0,

    request=0xd8bfe0) at rlm_ldap.c:1938

#2  0x000000000041c742 in call_modsingle (component=0,

    c=<value optimized out>, request=<value optimized out>) at modcall.c:304

#3  modcall (component=0, c=<value optimized out>,

    request=<value optimized out>) at modcall.c:686

#4  0x0000000000418c83 in indexed_modcall (comp=0, idx=7980103,

    request=0xd8bfe0) at modules.c:740

#5  0x000000000040941c in rad_check_password (request=0xd8bfe0) at auth.c:382

#6  rad_authenticate (request=0xd8bfe0) at auth.c:667

#7  0x0000000000428365 in radius_handle_request (request=0xd8bfe0,

    fun=0x408ae0 <rad_authenticate>) at event.c:3784

#8  0x00000000004204cb in request_handler_thread (arg=0xd8f700)

    at threads.c:537

#9  0x0000003a12c07851 in start_thread () from /lib64/libpthread.so.0

#10 0x0000003a124e894d in clone () from /lib64/libc.so.6

(gdb) frame 0

#0  0x00007f402afa9b89 in ldap_connect (instance=0xd24cf0,

    dn=0x7f3f98022818 " uid=1234576,dc=cuhk,dc=edu,dc=hk", password=0x..... "hideme", auth=1, result=0x7f401d780fac,

    err=0x0) at rlm_ldap.c:2410

2410            if (strcmp(TLS_DEFAULT_VERIFY, inst->tls_require_cert ) != 0 ) {

(gdb) info locals

inst = 0xd24cf0

ld = 0x7f3f980238d0

msgid = <value optimized out>

rc = <value optimized out>

ldap_version = 3

ldap_errno = 0

res = <value optimized out>

tv = {tv_sec = 1, tv_usec = 0}

(gdb) info args

instance = 0xd24cf0

dn = 0x7f3f98022818 "uid=1234567,dc=cuhk,dc=edu,dc=hk"

password=0x..... "hideme"auth = 1

result = 0x7f401d780fac

err = 0x0

(gdb) print inst->tls_require_cert

$1 = 0x0

(gdb) print inst->xlat_name

$2 = 0xd1d110 "ldap"  

--------------------- cut here ----------------------------

 

inst->tls_require_cert becomes null while we've 'require_cert = "never"' defined in modules/ldap.   There is no problem when the loading is not high:

 

Mon Oct 28 09:10:48 2013 : Debug:   [ldap] setting TLS mode to 1

Mon Oct 28 09:10:48 2013 : Debug:   [ldap] setting TLS CACert File to /usr/local/etc/raddb/certs/cacert.pem

Mon Oct 28 09:10:48 2013 : Debug:   [ldap] setting TLS Require Cert to never

 

Would anyone please advise?   Thanks a lot.

 

Besides, as we must use LDAP for authentication backend which maybe a bottleneck, is it feasible to have kind of QoS/rate control for incoming authentication requests, and/or some LDAP authentication result caching?    Sorry for the naïve questions.   Thanks again.

 

Best Regards,

/ST Wong

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freeradius.org/pipermail/freeradius-users/attachments/20131029/11897bb8/attachment.html>