FreeRADIUS 1.0.5 rlm_ldap crashing
Hi Guys. I'm doing a stress test on 1.0.5 running on FreeBSD 5.4-RELEASE and when start_tls is enabled radiusd crashed somwhere in the rlm_ldap module. I can't seem to make it produce a core file to properly inspect it. The server is running as root and core dumps are enabled in the config file. When i disable start_tls it does not crash. It always crased with signal 6 at random periods. Here's an error: rlm_ldap: attempting LDAP reconnection rlm_ldap: ldap_get_conn: Checking Id: 0 rlm_ldap: (re)connect to ldap01.totftds.int.distributel.net:389, authentication 0 rlm_ldap: ldap_get_conn: Checking Id: 1 rlm_ldap: ldap_get_conn: Got Id: 1 rlm_ldap: attempting LDAP reconnection rlm_ldap: (re)connect to ldap01.totftds.int.distributel.net:389, authentication 0 rlm_ldap: setting TLS CACert File to /usr/local/radiusd/current/etc/raddb/certs/cacert.pem rlm_ldap: setting TLS CACert File to /usr/local/radiusd/current/etc/raddb/certs/cacert.pem radiusd in free(): error: chunk is already free rlm_ldap: starting TLS Abort Any pointers on how to debug this problem ? Thanx Paul
"Paul Khavkine" <paul.khavkine@distributel.ca> wrote:
rlm_ldap: setting TLS CACert File to /usr/local/radiusd/current/etc/raddb/certs/cacert.pem rlm_ldap: setting TLS CACert File to /usr/local/radiusd/current/etc/raddb/certs/cacert.pem
That second entry should be a directory, not a file. There's a typo in the debugging output which makes that harder to figure out. Alan DeKok.
On Thu, 2005-12-29 at 16:56 -0500, Alan DeKok wrote:
"Paul Khavkine" <paul.khavkine@distributel.ca> wrote:
rlm_ldap: setting TLS CACert File to /usr/local/radiusd/current/etc/raddb/certs/cacert.pem rlm_ldap: setting TLS CACert File to /usr/local/radiusd/current/etc/raddb/certs/cacert.pem
That second entry should be a directory, not a file. There's a typo in the debugging output which makes that harder to figure out.
Alan, I'm not sure i follow. I don't have a CA cert directory set. Only a file. Should i also set tls_cacertdir ? Thanx Paul
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Here's another one: rlm_ldap: (re)connect to ldap01.mtlcnds.int.distributel.net:389, authentication 0 User-Password = "test123" NAS-IP-Address = 1.1.1.1 rlm_ldap: - authorize rlm_ldap: - authorize rlm_ldap: - authorize rlm_ldap: ldap_start_tls_s() User-Name = "ip_fixe@distributel.net"rlm_ldap: performing user authorization for ip_fixe rlm_ldap: performing user authorization for ip_fixe rlm_ldap: could not start TLS Connect error rlm_ldap: ldap_get_conn: Checking Id: 0 rlm_ldap: performing user authorization for ip_fixe rlm_ldap: ldap_get_conn: Checking Id: 0 User-Password = "test123"rlm_ldap: ldap_get_conn: Checking Id: 1 rad_recv: Access-Request packet from host 10.224.4.117:65039rlm_ldap: ldap_get_conn: Checking Id: 0 rlm_ldap: (re)connection attempt failed rlm_ldap: setting TLS CACert File to /usr/local/radiusd/current/etc/raddb/certs/cacert.pem, id=86, length=69 rlm_ldap: ldap_get_conn: Checking Id: 1 rlm_ldap: ldap_get_conn: Checking Id: 2 rlm_ldap: ldap_get_conn: Checking Id: 1 rlm_ldap: setting TLS CACert File to /usr/local/radiusd/current/etc/raddb/certs/cacert.pem rlm_ldap: search failed rlm_ldap: ldap_get_conn: Checking Id: 2 rlm_ldap: setting TLS CACert File to /usr/local/radiusd/current/etc/raddb/certs/cacert.pem rlm_ldap: setting TLS CACert File to /usr/local/radiusd/current/etc/raddb/certs/ NAS-IP-Address = 206.80.253.241rlm_ldap: ldap_get_conn: Checking Id: 3 rlm_ldap: ldap_get_conn: Checking Id: 2 rlm_ldap: ldap_release_conn: Release Id: 1 rlm_ldap: ldap_get_conn: Checking Id: 3 User-Name = "ip_fixe@distributel.net"radiusd in free(): error: chunk is already free rlm_ldap: ldap_get_conn: Checking Id: 4 Abort Crashed at about same place. Thanx Paul On Thu, 2005-12-29 at 16:56 -0500, Alan DeKok wrote:
"Paul Khavkine" <paul.khavkine@distributel.ca> wrote:
rlm_ldap: setting TLS CACert File to /usr/local/radiusd/current/etc/raddb/certs/cacert.pem rlm_ldap: setting TLS CACert File to /usr/local/radiusd/current/etc/raddb/certs/cacert.pem
That second entry should be a directory, not a file. There's a typo in the debugging output which makes that harder to figure out.
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
-----Original Message----- From: freeradius-users-bounces+paul.khavkine=distributel.ca@lists.freeradius.org on behalf of Alan DeKok Sent: Thu 12/29/2005 7:06 PM To: FreeRadius users mailing list Subject: Re: FreeRADIUS 1.0.5 rlm_ldap crashing "Paul Khavkine" <paul.khavkine@distributel.ca> wrote:
Crashed at about same place.
doc/bugs The server is not producing a core dump. I did compile it with --enable-developer and allow_core_dumps is set to yes. The server is running as root. Still no core. Thanx Paul Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
So anyone have an idea how can i fugure out where the server is crasing or how to force it to produce a core dump ? Thanx Paul On Thu, 2005-12-29 at 20:27 -0500, Paul Khavkine wrote:
-----Original Message----- From: freeradius-users-bounces +paul.khavkine=distributel.ca@lists.freeradius.org on behalf of Alan DeKok Sent: Thu 12/29/2005 7:06 PM To: FreeRadius users mailing list Subject: Re: FreeRADIUS 1.0.5 rlm_ldap crashing
"Paul Khavkine" <paul.khavkine@distributel.ca> wrote:
Crashed at about same place.
doc/bugs
The server is not producing a core dump. I did compile it with --enable-developer and allow_core_dumps is set to yes. The server is running as root. Still no core.
Thanx Paul
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On Fri, 2005-12-30 at 15:09 -0500, Alan DeKok wrote:
"Paul Khavkine" <paul.khavkine@distributel.ca> wrote:
So anyone have an idea how can i fugure out where the server is crasing or how to force it to produce a core dump ?
Run it in debugging mode. Set "ulimit". It SHOULD dump core.
Alan DeKok. -
Hi Alan. I did run the server in debug mode and ulimit -c unlimited and I still do not get a core dump file to analyze. The server is running as root, core dumps are allowed in the config, it crashes with either signal 6 or signal 11 when running with -x or -xx, does not crash when running -X Any ideas ? Thanx Paul
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
"Paul Khavkine" <paul.khavkine@distributel.ca> wrote:
The server is running as root, core dumps are allowed in the config, it crashes with either signal 6 or signal 11 when running with -x or -xx, does not crash when running -X
Ah... your OS may not support threaded core dumps. Yuck. It sounds like a thread problem to me, but I don't know why. What OS are you using? What version of FreeRADIUS? Where is it core dumping? Can you run freeradius under "gdb"? Alan DeKok.
On Tue, 2006-01-10 at 18:42 -0500, Alan DeKok wrote:
"Paul Khavkine" <paul.khavkine@distributel.ca> wrote:
The server is running as root, core dumps are allowed in the config, it crashes with either signal 6 or signal 11 when running with -x or -xx, does not crash when running -X
Ah... your OS may not support threaded core dumps. Yuck.
It sounds like a thread problem to me, but I don't know why. What OS are you using? What version of FreeRADIUS? Where is it core dumping? Can you run freeradius under "gdb"?
Afte a few days investgation it seems like a threading problem on FreeBSD 5.4-RELEASE and 6.0-RELEASE. The problem can be reproduced when: 1) running FreeRADIUS 1.0.5 2) running SMP kernel 3) radiusd is linked against libpthread 4) Using start_tls with LDAP Under 5.4-RELASE the process will die with an Abort trap and not dump core when linked against libpthread. If radiusd is linked against libthr (1:1 threading library) it simply jams and stops responding but does not terminate. Killing it with signal 9 or 6 does not produce a core dump. When run inside gdb it crashes gdb and DOES produce gdb.core, but gdb.core is useless since gdb is not compiled with -g Under 6.0-RELEASE when linked against libpthread, the result is the same as 5.4-RELEASE When linked against libthr it does not crash and has passed my stress test with flying colors. Both FreeBSD releases seems to have no problem with UP kernel and libpthread. I'm going to a a longer stress test to see if indeed FreeBSD 6.0/SMP/libthr is stable. Thanx Paul
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Hi folks. I was finally able to get a core dump under FreeBSD 6.0-RELEASE/SMP/libthr. rlm_ldap: ldap_get_conn: Got Id: 15 rlm_ldap: (re)connect to ldap01.mtlcnds.int.distributel.net:389, authentication 0 rlm_ldap: ldap_get_conn: Checking Id: 0 rlm_ldap: setting TLS CACert File to /usr/local/radiusd/current/etc/raddb/certs/cacert.pem rlm_ldap: setting TLS CACert Directory to /usr/local/radiusd/current/etc/raddb/certs/ rlm_ldap: ldap_get_conn: Checking Id: 9 Assertion failed: (LBER_VALID( ber )), function ber_free_buf, file io.c, line 171. rlm_ldap: setting TLS CACert File to /usr/local/radiusd/current/etc/raddb/certs/cacert.pem Program received signal SIGABRT, Aborted. [Switching to Thread 0x81c6f00 (LWP 100232)] 0x28231773 in thr_kill () from /lib/libc.so.6 (gdb) bt #0 0x28231773 in thr_kill () from /lib/libc.so.6 #1 0x281eb701 in pthread_mutex_unlock () from /usr/lib/libthr.so.2 #2 0x281e60b7 in pthread_kill () from /usr/lib/libthr.so.2 #3 0x281e4b02 in raise () from /usr/lib/libthr.so.2 #4 0x282a2c5c in abort () from /lib/libc.so.6 #5 0x2828100c in __assert () from /lib/libc.so.6 #6 0x28330a3b in ber_free_buf () from /usr/local/lib/liblber-2.3.so.1 #7 0x28330ae0 in ber_free () from /usr/local/lib/liblber-2.3.so.1 #8 0x283094e9 in ldap_free_request_int () from /usr/local/lib/libldap_r.so #9 0x28309601 in ldap_free_request () from /usr/local/lib/libldap_r.so #10 0x282f55e6 in try_read1msg () from /usr/local/lib/libldap_r.so #11 0x282f48f4 in wait4msg () from /usr/local/lib/libldap_r.so #12 0x282f40fa in ldap_result () from /usr/local/lib/libldap_r.so #13 0x282f8a57 in ldap_extended_operation_s () from /usr/local/lib/libldap_r.so #14 0x2831bd62 in ldap_start_tls_s () from /usr/local/lib/libldap_r.so #15 0x2834e762 in ldap_connect (instance=0x8066800, dn=0x811d4e0 "cn=xxxxxx,dc=distributel,dc=net", password=0x807f9d0 "xxxxxxx", auth=0, result=0xbebef72c, err=0x0) at rlm_ldap.c:2069 #16 0x2834eebc in perform_search (instance=0x8066800, conn=0x811e810, search_basedn=0xbebef8d4 "ou=passwd,ou=NIS,dc=distributel,dc=net", scope=2, filter=0xbebefcd4 "(uid=ip_fixe)", attrs=0x8066900, result=0xbebef7cc) at rlm_ldap.c:772 #17 0x28351221 in ldap_authorize (instance=0x8066800, request=0x81c6600) at rlm_ldap.c:1247 #18 0x08055457 in modcall (component=1, c=0x8127580, request=0x81c6600) at modcall.c:236 #19 0x080559fb in call_one (component=0, p=0x8127580, request=0x81c6600, priority=0xbebf01a0, result=0xbebf01a4) at modcall.c:269 #20 0x080556bf in modcall (component=1, c=0x81222c0, request=0x81c6600) at modcall.c:324 #21 0x080559fb in call_one (component=0, p=0x81222c0, request=0x81c6600, priority=0xbebf0220, result=0xbebf0224) at modcall.c:269 #22 0x080556bf in modcall (component=1, c=0x8122100, request=0x81c6600) at modcall.c:324 #23 0x08054a10 in indexed_modcall (comp=1, idx=100232, request=0x81c6600) at modules.c:469 #24 0x0805207a in rad_authenticate (request=0x81c6600) at auth.c:589 #25 0x0804c16e in rad_respond (request=0x81c6600, fun=0x8051fb0 <rad_authenticate>) at radiusd.c:1642 #26 0x080579ba in request_handler_thread (arg=0x81c47e0) at threads.c:517 #27 0x281ed05d in pthread_create () from /usr/lib/libthr.so.2 #28 0x00000000 in ?? () (gdb) Any clues ? bug in OpenLDAP library maybe ? Thanx Paul On Thu, 2006-01-12 at 15:57 -0500, Paul Khavkine wrote:
On Tue, 2006-01-10 at 18:42 -0500, Alan DeKok wrote:
"Paul Khavkine" <paul.khavkine@distributel.ca> wrote:
The server is running as root, core dumps are allowed in the config, it crashes with either signal 6 or signal 11 when running with -x or -xx, does not crash when running -X
Ah... your OS may not support threaded core dumps. Yuck.
It sounds like a thread problem to me, but I don't know why. What OS are you using? What version of FreeRADIUS? Where is it core dumping? Can you run freeradius under "gdb"?
Afte a few days investgation it seems like a threading problem on FreeBSD 5.4-RELEASE and 6.0-RELEASE.
The problem can be reproduced when:
1) running FreeRADIUS 1.0.5 2) running SMP kernel 3) radiusd is linked against libpthread 4) Using start_tls with LDAP
Under 5.4-RELASE the process will die with an Abort trap and not dump core when linked against libpthread. If radiusd is linked against libthr (1:1 threading library) it simply jams and stops responding but does not terminate. Killing it with signal 9 or 6 does not produce a core dump. When run inside gdb it crashes gdb and DOES produce gdb.core, but gdb.core is useless since gdb is not compiled with -g
Under 6.0-RELEASE when linked against libpthread, the result is the same as 5.4-RELEASE
When linked against libthr it does not crash and has passed my stress test with flying colors.
Both FreeBSD releases seems to have no problem with UP kernel and libpthread.
I'm going to a a longer stress test to see if indeed FreeBSD 6.0/SMP/libthr is stable.
Thanx Paul
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On Fri, 13 Jan 2006, Alan DeKok wrote:
"Paul Khavkine" <paul.khavkine@distributel.ca> wrote:
Any clues ? bug in OpenLDAP library maybe ?
Looks that way to me.
What version of openldap are you using for this and how did you install it? For example, perhaps you installed freeradius from ports and it pulled in openldap-client as well for the build?
On Fri, 2006-01-13 at 23:26 -0500, Dusty Doris wrote:
On Fri, 13 Jan 2006, Alan DeKok wrote:
"Paul Khavkine" <paul.khavkine@distributel.ca> wrote:
Any clues ? bug in OpenLDAP library maybe ?
Looks that way to me.
What version of openldap are you using for this and how did you install it? For example, perhaps you installed freeradius from ports and it pulled in openldap-client as well for the build?
OpenLDAP version is 2.3.9. Nothing is installed from ports, everything is built manually. Thanx Paul
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On Friday 30 December 2005 21:24, Paul Khavkine wrote:
So anyone have an idea how can i fugure out where the server is crasing or how to force it to produce a core dump ?
can you check sysctl kern.coredump? It should be 1. In my environment(FreeBSD-6) freeradius dumps core.
Thanx Paul
On Thu, 2005-12-29 at 20:27 -0500, Paul Khavkine wrote:
-----Original Message----- From: freeradius-users-bounces +paul.khavkine=distributel.ca@lists.freeradius.org on behalf of Alan DeKok Sent: Thu 12/29/2005 7:06 PM To: FreeRadius users mailing list Subject: Re: FreeRADIUS 1.0.5 rlm_ldap crashing
"Paul Khavkine" <paul.khavkine@distributel.ca> wrote:
Crashed at about same place.
doc/bugs
The server is not producing a core dump. I did compile it with --enable-developer and allow_core_dumps is set to yes. The server is running as root. Still no core.
Thanx Paul
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
participants (4)
-
Alan DeKok -
Dusty Doris -
Nikos Vassiliadis -
Paul Khavkine