Hi folks, In my RADIUS config on 3.0.12 I've listed a number of LDAP servers, identically configured, and used a redundant-load-balance section to list them. This has always worked in a "normal" situation but over Christmas we had a couple of outages and I have some questions about the behaviour of redundant-load-balance. Firstly, one of the LDAP servers lost power. redundant-load-balance did the right thing, reported "Opening connection failed" and skipped that server. Later, another LDAP server got itself in a funny state where it was up and responsive, accepting connections but timing out when FreeRADIUS tried to bind, with the message "Timed out while waiting for server to respond". For some reason, redundant-load-balance continued to send LDAP queries to the server even though it wasn't working properly. How does rlm_ldap determine if a server is up or down, and what can I configure in my ldap modules or redundant-load-balance to ensure that if the LDAP server misbehaves in future, it will get marked as dead and cause redundant-load-balance to send queries to a different server? Unfortunately as the event happened in the past I don't have debug logs - only the regular radius.log. We also don't know what caused the LDAP server to get into that state (our Windows guys are looking into it) so currently we can't recreate the problem, either. Thanks, Jonathan -- Jonathan Gazeley Senior Systems Administrator IT Services University of Bristol
On Jan 4, 2017, at 8:17 AM, Jonathan Gazeley <Jonathan.Gazeley@bristol.ac.uk> wrote:
Later, another LDAP server got itself in a funny state where it was up and responsive, accepting connections but timing out when FreeRADIUS tried to bind, with the message "Timed out while waiting for server to respond". For some reason, redundant-load-balance continued to send LDAP queries to the server even though it wasn't working properly.
How does rlm_ldap determine if a server is up or down, and what can I configure in my ldap modules or redundant-load-balance to ensure that if the LDAP server misbehaves in future, it will get marked as dead and cause redundant-load-balance to send queries to a different server?
FreeRADUS tracks connections. It assumes that if the connection is up and responsive, then the LDAP server is up. If the connection is up but slow, FreeRADIUS assumes that the LDAP server is up and slow. There are few good ways to fix this problem. Due to threading issues, the "redundant-load-balance" code doesn't remember data from previous requests. We're looking at perhaps changing that in v4.
Unfortunately as the event happened in the past I don't have debug logs - only the regular radius.log. We also don't know what caused the LDAP server to get into that state (our Windows guys are looking into it) so currently we can't recreate the problem, either.
In short, if FreeRADIUS uses a database, you need to ensure that the database is up and responsive. There is only so much that we can do in order to work around issues with broken databases. Alan DeKok.
Hi, using connection pools? the server is used if theres a known connection to it if using stateful methods - eg TCP and SSL... are you using SSL based connections? if so, is your OpenLDAP compiled against OpenSSL or against LibNSS? if the later then theres a hideous pit of incompatibilities (that we hit) that can only be resolved by using a locally compiled version of OpenLDAP compiled against OpenSSL (which results in a pretty flawless pool system). we use the template method to define the basic properties of the servers too to reduce the configuration (and possible mistakes). alan
On Jan 4, 2017, at 9:14 AM, A.L.M.Buxey@lboro.ac.uk wrote:
using connection pools? the server is used if theres a known connection to it if using stateful methods - eg TCP and SSL...
The connection pools work only for one LDAP server. If you have 2 and want to load-balance, you need 2 connection pools.
are you using SSL based connections? if so, is your OpenLDAP compiled against OpenSSL or against LibNSS? if the later then theres a hideous pit of incompatibilities (that we hit) that can only be resolved by using a locally compiled version of OpenLDAP compiled against OpenSSL (which results in a pretty flawless pool system).
Yeah. Alan DeKok.
On 04/01/17 14:14, A.L.M.Buxey@lboro.ac.uk wrote:
Hi,
using connection pools? the server is used if theres a known connection to it if using stateful methods - eg TCP and SSL...
We are using one connection pool for each of the LDAP servers we have here. They are templated to ensure consistency.
are you using SSL based connections? if so, is your OpenLDAP compiled against OpenSSL or against LibNSS? if the later then theres a hideous pit of incompatibilities (that we hit) that can only be resolved by using a locally compiled version of OpenLDAP compiled against OpenSSL (which results in a pretty flawless pool system).
Yes, we are using SSL. We're using the standard OpenLDAP package distributed with CentOS 7, which is built against libnss. Obviously I prefer to rebuild as little of the system as possible but I'm open to the idea of rebuilding OpenLDAP against OpenSSL if necessary. What problems did you run into? Thanks, Jonathan -- Jonathan Gazeley Senior Systems Administrator IT Services University of Bristol
Hi,
Yes, we are using SSL. We're using the standard OpenLDAP package distributed with CentOS 7, which is built against libnss. Obviously I prefer to rebuild as little of the system as possible but I'm open to the idea of rebuilding OpenLDAP against OpenSSL if necessary.
yes, rebuild. gfairly simple process, get openldap source, compile, install (eg into /usr/local) , remove openldap-devel RPM (keep the system openldap for other things...) - then compile FR again but ensure its picking up the openldap installed in /usr/local (use the configure flags)
What problems did you run into?
several things - most based on connections not reestablishing, connections not noted as failed, TLS issues etc. alan
A.L.M.Buxey@lboro.ac.uk wrote:
Yes, we are using SSL. We're using the standard OpenLDAP package distributed with CentOS 7, which is built against libnss. Obviously I prefer to rebuild as little of the system as possible but I'm open to the idea of rebuilding OpenLDAP against OpenSSL if necessary.
yes, rebuild. gfairly simple process, get openldap source, compile, install (eg into /usr/local) , remove openldap-devel RPM (keep the system openldap for other things...) -
Note that the LTB project provides OpenLDAP RPMs also for CentOS7 linked against OpenSSL (built with prefix /usr/local/openldap): https://ltb-project.org/wiki/documentation/openldap-rpm#yum_repository Ciao, Michael.
Hi,
Note that the LTB project provides OpenLDAP RPMs also for CentOS7 linked against OpenSSL (built with prefix /usr/local/openldap):
https://ltb-project.org/wiki/documentation/openldap-rpm#yum_repository
thats a very useful bit of info alan
On 04/01/17 17:18, Michael Ströder wrote:
Note that the LTB project provides OpenLDAP RPMs also for CentOS7 linked against OpenSSL (built with prefix /usr/local/openldap):
https://ltb-project.org/wiki/documentation/openldap-rpm#yum_repository
Thanks, this looks useful. It doesn't look like it includes a -devel package though. Is this not required in order to rebuild FreeRADIUS against the new OpenLDAP? Or are the headers in the main package? Thanks, Jonathan -- Jonathan Gazeley Senior Systems Administrator IT Services University of Bristol
Jonathan Gazeley wrote:
On 04/01/17 17:18, Michael Ströder wrote:
Note that the LTB project provides OpenLDAP RPMs also for CentOS7 linked against OpenSSL (built with prefix /usr/local/openldap):
https://ltb-project.org/wiki/documentation/openldap-rpm#yum_repository
Thanks, this looks useful. It doesn't look like it includes a -devel package though. Is this not required in order to rebuild FreeRADIUS against the new OpenLDAP? Or are the headers in the main package?
No separate -devel package. Directory /usr/local/openldap/include has all the necessary *.h files. Ciao, Michael.
On Jan 4, 2017, at 12:18 PM, Michael Ströder <michael@stroeder.com> wrote:
A.L.M.Buxey@lboro.ac.uk wrote:
Yes, we are using SSL. We're using the standard OpenLDAP package distributed with CentOS 7, which is built against libnss. Obviously I prefer to rebuild as little of the system as possible but I'm open to the idea of rebuilding OpenLDAP against OpenSSL if necessary.
yes, rebuild. gfairly simple process, get openldap source, compile, install (eg into /usr/local) , remove openldap-devel RPM (keep the system openldap for other things...) -
Note that the LTB project provides OpenLDAP RPMs also for CentOS7 linked against OpenSSL (built with prefix /usr/local/openldap):
https://ltb-project.org/wiki/documentation/openldap-rpm#yum_repository
Yes, that's what NetworkRADIUS uses for customer deployments on RHEL, or Symas OpenLDAP depending on the level of support required. -Arran
On Jan 4, 2017, at 10:14 AM, Jonathan Gazeley <Jonathan.Gazeley@bristol.ac.uk> wrote:
Yes, we are using SSL. We're using the standard OpenLDAP package distributed with CentOS 7, which is built against libnss. Obviously I prefer to rebuild as little of the system as possible but I'm open to the idea of rebuilding OpenLDAP against OpenSSL if necessary.
I would say it's necessary. libnss has wrappers for OpenSSL functions, which means FreeRADIUS *may* accidentally use nss functions where it expects OpenSSL, or vice versa. Alan DeKok.
participants (5)
-
A.L.M.Buxey＠lboro.ac.uk -
Alan DeKok -
Arran Cudbard-Bell -
Jonathan Gazeley -
Michael Ströder