Failure to reconnect to ldaps server after idle_timeout
Hi all, I have a freeradius v3.0.7 server running in a test setup that uses the rlm_ldap module to verify users and groups against an LDAPS server (ie LDAP with SSL enabled). With radius -X the server starts up, successfully connects to the LDAPS server, and successfully returns the correct results to requests. After some time has passed, requests start failing as follows: rlm_ldap (ldap): Closing connection (6): Hit idle_timeout, was idle for 601 seconds rlm_ldap (ldap): You probably need to lower "min" rlm_ldap (ldap): Closing connection (5): Hit idle_timeout, was idle for 601 seconds rlm_ldap (ldap): You probably need to lower "min" TLS: could not shutdown NSS - error -8053:NSS could not shutdown. Objects are still in use.. rlm_ldap (ldap): 0 of 0 connections in use. You may need to increase "spare" rlm_ldap (ldap): Opening additional connection (7), 1 of 32 pending slots used rlm_ldap (ldap): Connecting to ldap://ldap.example.com:636 TLS: could not find the slot for the certificate '/etc/raddb/certs/ldap-ca.pem' - error -8127:The security card or token does not exist, needs to be initialized, or has been removed.. TLS: /etc/raddb/certs/ldap-ca.pem is not a valid CA certificate file - error -8127:The security card or token does not exist, needs to be initialized, or has been removed.. TLS: could not perform TLS system initialization. TLS: error: could not initialize moznss security context - error -8127:The security card or token does not exist, needs to be initialized, or has been removed. TLS: can't create ssl handle. rlm_ldap (ldap): Bind with cn=Radius,o=Example,c=XX to ldap://ldap.example.com:636 failed: Can't contact LDAP server TLS: could not shutdown NSS - error -8053:NSS could not shutdown. Objects are still in use.. rlm_ldap (ldap): Opening connection failed (7) (28) [ldap] = fail The server is then broken until restarted. It appears that after a normal idle timeout occurs and the last LDAP connection is shut down, an attempt is made to shut down NSS, which fails as follows: TLS: could not shutdown NSS - error -8053:NSS could not shutdown. Objects are still in use.. With NSS in a broken state, all subsequent reconnection attempts break. Is this a known issue in v3.0.7? Regards, Graham —
TLS: could not shutdown NSS - error -8053:NSS could not shutdown. Objects are still in use.. rlm_ldap (ldap): 0 of 0 connections in use. You may need to increase "spare" rlm_ldap (ldap): Opening additional connection (7), 1 of 32 pending slots used rlm_ldap (ldap): Connecting to ldap://ldap.example.com:636 TLS: could not find the slot for the certificate '/etc/raddb/certs/ldap-ca.pem' - error -8127:The security card or token does not exist, needs to be initialized, or has been removed.. TLS: /etc/raddb/certs/ldap-ca.pem is not a valid CA certificate file - error -8127:The security card or token does not exist, needs to be initialized, or has been removed.. TLS: could not perform TLS system initialization. TLS: error: could not initialize moznss security context - error -8127:The security card or token does not exist, needs to be initialized, or has been removed. TLS: can't create ssl handle. rlm_ldap (ldap): Bind with cn=Radius,o=Example,c=XX to ldap://ldap.example.com:636 failed: Can't contact LDAP server TLS: could not shutdown NSS - error -8053:NSS could not shutdown. Objects are still in use.. rlm_ldap (ldap): Opening connection failed (7) (28) [ldap] = fail
I've seen this at STFC before... but despite my prodding, this was not raised on the list. :-/ Stefan Paetow Moonshot Industry & Research Liaison Coordinator t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet Lumen House, Library Avenue, Harwell Oxford, Didcot, OX11 0SG jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. Jisc Collections and Janet Ltd. is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under Company No. number 2881024, VAT No. GB 197 0632 86. The registered office is: Lumen House, Library Avenue, Harwell, Didcot, Oxfordshire, OX11 0SG. T 01235 822200.
On 10 Mar 2015, at 13:48, Stefan Paetow <Stefan.Paetow@jisc.ac.uk> wrote:
TLS: could not shutdown NSS - error -8053:NSS could not shutdown. Objects are still in use.. rlm_ldap (ldap): 0 of 0 connections in use. You may need to increase "spare" rlm_ldap (ldap): Opening additional connection (7), 1 of 32 pending slots used rlm_ldap (ldap): Connecting to ldap://ldap.example.com:636 TLS: could not find the slot for the certificate '/etc/raddb/certs/ldap-ca.pem' - error -8127:The security card or token does not exist, needs to be initialized, or has been removed.. TLS: /etc/raddb/certs/ldap-ca.pem is not a valid CA certificate file - error -8127:The security card or token does not exist, needs to be initialized, or has been removed.. TLS: could not perform TLS system initialization. TLS: error: could not initialize moznss security context - error -8127:The security card or token does not exist, needs to be initialized, or has been removed. TLS: can't create ssl handle. rlm_ldap (ldap): Bind with cn=Radius,o=Example,c=XX to ldap://ldap.example.com:636 failed: Can't contact LDAP server TLS: could not shutdown NSS - error -8053:NSS could not shutdown. Objects are still in use.. rlm_ldap (ldap): Opening connection failed (7) (28) [ldap] = fail
I've seen this at STFC before... but despite my prodding, this was not raised on the list. :-/
OpenLDAP built against NSS is not compatible with rlm_ldap when used with Start TLS or LDAPS, see here: https://github.com/FreeRADIUS/freeradius-server/pull/866 It's not something that can be fixed our side. The OpenLDAP guys say they're not responsible for the NSS bindings. It's an issue you'll need to raise with Redhat. -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
I've seen this at STFC before... but despite my prodding, this was not raised on the list. :-/
fixed it with your colleague last week! :-)
Cheeky sod! :-) Thank you very much, Alan. I'll raise this with RedHat as an issue... Stefan Paetow Moonshot Industry & Research Liaison Coordinator t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. Jisc Collections and Janet Ltd. is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under Company No. number 2881024, VAT No. GB 197 0632 86. The registered office is: Lumen House, Library Avenue, Harwell, Didcot, Oxfordshire, OX11 0SG. T 01235 822200.
On Mar 10, 2015, at 1:36 PM, Graham Leggett <minfrin@sharp.fm> wrote:
I have a freeradius v3.0.7 server running in a test setup that uses the rlm_ldap module to verify users and groups against an LDAPS server (ie LDAP with SSL enabled).
With radius -X the server starts up, successfully connects to the LDAPS server, and successfully returns the correct results to requests. ... TLS: could not shutdown NSS - error -8053:NSS could not shutdown. Objects are still in use..
Ugh. You’re using a version of libldap which was built against NSS. Don’t do that. Switch to one which uses OpenSSL. The server uses OpenSSL for everything. Mixing OpenSSL and NSS is probably not a good idea.
With NSS in a broken state, all subsequent reconnection attempts break.
Is this a known issue in v3.0.7?
It’s a known issue with NSS. Alan DeKok.
Hi,
I have a freeradius v3.0.7 server running in a test setup that uses the rlm_ldap module to verify users and groups against an LDAPS server (ie LDAP with SSL enabled).
seen this in 3.0.x (before 3.0.7) where the LDAP timers are set to aggressively. dont expire the connections and have lifetime = 0 - then the sockets are nicely kept open and will be reconnected if theres connectivity issue alan
On 10 Mar 2015, at 16:25, A.L.M.Buxey@lboro.ac.uk wrote:
Hi,
I have a freeradius v3.0.7 server running in a test setup that uses the rlm_ldap module to verify users and groups against an LDAPS server (ie LDAP with SSL enabled).
seen this in 3.0.x (before 3.0.7) where the LDAP timers are set to aggressively. dont expire the connections and have lifetime = 0 - then the sockets are nicely kept open and will be reconnected if theres connectivity issue
Given the root cause I would expect this to break if ldap_unbind was called on any LDAP handle. Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
On 10 Mar 2015, at 18:57, A.L.M.Buxey@lboro.ac.uk wrote:
Hi,
Given the root cause I would expect this to break if ldap_unbind was called on any LDAP handle.
yep. by keeping the connection open, you avoid the issue
It might break if any operation returns: - LDAP_BUSY - LDAP_UNAVAILABLE - LDAP_SERVER_DOWN The connection pool code will destroy the handle and create a new one. Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
seen this in 3.0.x (before 3.0.7) where the LDAP timers are set to aggressively. don't expire the connections and have lifetime = 0 - then the sockets are nicely kept open and will be reconnected if theres connectivity issue
Alan D, Arran, can we document this in the Wiki? I'll happily put a Wiki entry for that together if you're ok with this? Stefan Paetow Moonshot Industry & Research Liaison Coordinator t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. Jisc Collections and Janet Ltd. is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under Company No. number 2881024, VAT No. GB 197 0632 86. The registered office is: Lumen House, Library Avenue, Harwell, Didcot, Oxfordshire, OX11 0SG. T 01235 822200.
On 10 Mar 2015, at 17:01, Stefan Paetow <Stefan.Paetow@jisc.ac.uk> wrote:
seen this in 3.0.x (before 3.0.7) where the LDAP timers are set to aggressively. don't expire the connections and have lifetime = 0 - then the sockets are nicely kept open and will be reconnected if theres connectivity issue
Alan D, Arran, can we document this in the Wiki? I'll happily put a Wiki entry for that together if you're ok with this?
Sure. Something along the lines of "NSS is garbage, don't use NSS"? Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
On Tue, Mar 10, 2015 at 05:08:31PM -0400, Arran Cudbard-Bell wrote:
On 10 Mar 2015, at 17:01, Stefan Paetow <Stefan.Paetow@jisc.ac.uk> wrote:
Alan D, Arran, can we document this in the Wiki? I'll happily put a Wiki entry for that together if you're ok with this?
Sure.
Something along the lines of "NSS is garbage, don't use NSS"?
If it's a bug in an O/S that people are paying support for but it's not fixed then that should surely be s/NSS/RedHat/g ? :-) Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
On Mar 10, 2015, at 8:22 PM, Matthew Newton <mcn4@leicester.ac.uk> wrote:
If it's a bug in an O/S that people are paying support for but it's not fixed then that should surely be s/NSS/RedHat/g ?
It’s a bug in how libldap uses NSS. Some of that is the NSS patch. Some of that is the libldap code. Libldap has no “initialize library” function. As a result, any initialization is done by various ad-hoc heuristics. Libldap has no “close library” function. As a result, any cleanup is done by various ad-hoc heuristics. These heuristics can get things wrong, and can cause your program to crash. This is a design requirement of libldap, according to the OpenLDAP / Symas people. Alan DeKok.
On Wed, Mar 11, 2015 at 09:13:20AM -0400, Alan DeKok wrote:
It’s a bug in how libldap uses NSS. Some of that is the NSS patch. Some of that is the libldap code.
Libldap has no “initialize library” function. As a result, any initialization is done by various ad-hoc heuristics. Libldap has no “close library” function. As a result, any cleanup is done by various ad-hoc heuristics. These heuristics can get things wrong, and can cause your program to crash.
Oh nasty. That's exactly what I've just been through with the Samba guys. I did a patch that used thread local storage, but then in the best case you end up likely to leak memory because the right destructor isn't called, or worse the library is unloaded but there are still pointers left for destructor functions in some threads, leading to crashes. Solution? Nice clean initialise and free functions, then pass the context into everything you use. All the messy heuristics go out of the window and everyone is happy. Agreed, there is no real solution for FreeRADIUS to work around it in this case; the library needs fixing. Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
On 11 Mar 2015, at 4:48 PM, Matthew Newton <mcn4@leicester.ac.uk> wrote:
Oh nasty. That's exactly what I've just been through with the Samba guys. I did a patch that used thread local storage, but then in the best case you end up likely to leak memory because the right destructor isn't called, or worse the library is unloaded but there are still pointers left for destructor functions in some threads, leading to crashes.
Solution? Nice clean initialise and free functions, then pass the context into everything you use. All the messy heuristics go out of the window and everyone is happy.
Agreed, there is no real solution for FreeRADIUS to work around it in this case; the library needs fixing.
I don’t follow - how does libldap work with software like httpd, but not work with freeradius? Regards, Graham —
On Mar 11, 2015, at 11:30 AM, Graham Leggett <minfrin@sharp.fm> wrote:
I don’t follow - how does libldap work with software like httpd, but not work with freeradius?
Many of the other users open connections and keep them open, or open connections in a child process. Not many use multiple threads and open/close connections. Alan DeKok.
On 11 Mar 2015, at 11:33, Alan DeKok <aland@deployingradius.com> wrote:
On Mar 11, 2015, at 11:30 AM, Graham Leggett <minfrin@sharp.fm> wrote:
I don’t follow - how does libldap work with software like httpd, but not work with freeradius? Many of the other users open connections and keep them open, or open connections in a child process. Not many use multiple threads and open/close connections.
It's because mod_ldap uses a global SSL configuration context. We use per connection configuration contexts, as that's the only way to ensure the module instance specific TLS parameters are honoured. If you attempted to use multiple TLS certificates with mod_ldap you would likely find that it did not work at intended. If you tried the same configuration with FreeRADIUS, it would work as intended. The mod_ldap code is available here: http://svn.apache.org/repos/asf/httpd/httpd/trunk/modules/ldap/util_ldap.c The github issue describes the root cause in greater detail: https://github.com/FreeRADIUS/freeradius-server/pull/866 This seems like an NSS defect TBH. It should do reference counting for its modules, to ensure they're not referenced by multiple contexts, so that they're not unintentionally unloaded when contexts are freed. TL;DR mod_ldap is broken in other ways which happens to mask this issue. Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
Alan D, Arran, can we document this in the Wiki? I'll happily put a Wiki entry for that together if you're ok with this?
Sure.
Something along the lines of "NSS is garbage, don't use NSS"?
Haha. Not quite in those words, no. But the rlm_ldap entry will have something at the bottom of the page. Stefan Paetow Moonshot Industry & Research Liaison Coordinator t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc¹s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. Jisc Collections and Janet Ltd. is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under Company No. number 2881024, VAT No. GB 197 0632 86. The registered office is: Lumen House, Library Avenue, Harwell, Didcot, Oxfordshire, OX11 0SG. T 01235 822200.
On 10/03/15 21:08, Arran Cudbard-Bell wrote:
On 10 Mar 2015, at 17:01, Stefan Paetow <Stefan.Paetow@jisc.ac.uk> wrote:
seen this in 3.0.x (before 3.0.7) where the LDAP timers are set to aggressively. don't expire the connections and have lifetime = 0 - then the sockets are nicely kept open and will be reconnected if theres connectivity issue
Alan D, Arran, can we document this in the Wiki? I'll happily put a Wiki entry for that together if you're ok with this?
Sure.
Something along the lines of "NSS is garbage, don't use NSS"?
NSS is a generally well-written library, and thought by some to be superior to OpenSSL. I have mixed feelings - there are some design decisions I'm not wild on - but I think it's unfair to describe it as "garbage" ;o) It's definitely tedious this problem exists - I assume because RedHat have linked libldap with the NSS/OpenSSL compat shim? - but that's hardly the fault of NSS.
NSS is a generally well-written library, and thought by some to be superior to OpenSSL. I have mixed feelings - there are some design decisions I'm not wild on - but I think it's unfair to describe it as "garbage" ;o)
It's definitely tedious this problem exists - I assume because RedHat have linked libldap with the NSS/OpenSSL compat shim? - but that's hardly the fault of NSS.
Phil, could you have a look at the new entry at the end of http://wiki.freeradius.org/modules/Rlm_ldap to check whether it's sufficiently... diplomatic? ;-) Stefan Paetow Moonshot Industry & Research Liaison Coordinator t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet Lumen House, Library Avenue, Harwell Oxford, Didcot, OX11 0SG jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. Jisc Collections and Janet Ltd. is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under Company No. number 2881024, VAT No. GB 197 0632 86. The registered office is: Lumen House, Library Avenue, Harwell, Didcot, Oxfordshire, OX11 0SG. T 01235 822200.
On 11/03/15 12:02, Stefan Paetow wrote:
Phil, could you have a look at the new entry at the end of http://wiki.freeradius.org/modules/Rlm_ldap to check whether it's sufficiently... diplomatic? ;-)
I'm not the boss of anyone - write what you like! ;o) - but it looks fine to me.
I'm not the boss of anyone - write what you like! ;o) - but it looks fine to me.
Thank you, Phil. Stefan Paetow Moonshot Industry & Research Liaison Coordinator t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet Lumen House, Library Avenue, Harwell Oxford, Didcot, OX11 0SG jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. Jisc Collections and Janet Ltd. is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under Company No. number 2881024, VAT No. GB 197 0632 86. The registered office is: Lumen House, Library Avenue, Harwell, Didcot, Oxfordshire, OX11 0SG. T 01235 822200.
participants (7)
-
A.L.M.Buxey＠lboro.ac.uk -
Alan DeKok -
Arran Cudbard-Bell -
Graham Leggett -
Matthew Newton -
Phil Mayers -
Stefan Paetow