Multiple LDAP servers best practice
Hi, We use EAP-PEAP and use ldap to normalize usernames and retrieve group membership (via ldaps). We recently found that our freeradius servers were only connecting to the first of four ldap servers specified in mods-available/ldap. The servers are listed on separate lines like this: server = 'ldaps://dc1.ad.stolaf.edu' server = 'ldaps://dc2.ad.stolaf.edu' server = 'ldaps://dc3.ad.stolaf.edu' server = 'ldaps://dc4.ad.stolaf.edu' The certificates used for ldaps have SANs that include our domain name ' ad.stolaf.edu' and on my dev server I found that using "server = ' ad.stolaf.edu'" will start connections with all ldap servers, modulo DNS round robin results. I consulted some openldap documentation, but am still left with the question: what is the best practice for listing and utilizing multiple LDAP servers (for both failover and load balancing)? Also, is URI or hostname more preferred? (both seem to work in my testing, I do specify "port = 636" elsewhere in the config) radiusd -X output from our production config (4 servers listed one per line): Wed Feb 15 07:38:33 2023 : Info: rlm_ldap: libldap vendor: OpenLDAP, version: 20446 Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Initialising connection pool Wed Feb 15 07:38:33 2023 : Debug: pool { Wed Feb 15 07:38:33 2023 : Debug: start = 5 Wed Feb 15 07:38:33 2023 : Debug: min = 3 Wed Feb 15 07:38:33 2023 : Debug: max = 32 Wed Feb 15 07:38:33 2023 : Debug: spare = 10 Wed Feb 15 07:38:33 2023 : Debug: uses = 0 Wed Feb 15 07:38:33 2023 : Debug: lifetime = 0 Wed Feb 15 07:38:33 2023 : Debug: cleanup_interval = 30 Wed Feb 15 07:38:33 2023 : Debug: idle_timeout = 60 Wed Feb 15 07:38:33 2023 : Debug: retry_delay = 30 Wed Feb 15 07:38:33 2023 : Debug: spread = no Wed Feb 15 07:38:33 2023 : Debug: } Wed Feb 15 07:38:33 2023 : Info: rlm_ldap (ldap): Opening additional connection (0), 1 of 32 pending slots used Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Connecting to ldaps:// dc1.ad.stolaf.edu:636 ldaps://dc2.ad.stolaf.edu:636 ldaps:// dc3.ad.stolaf.edu:636 ldaps://dc4.ad.stolaf.edu:636 Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): New libldap handle 0x564cf7a99500 Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Waiting for bind result... Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Bind successful Wed Feb 15 07:38:33 2023 : Info: rlm_ldap (ldap): Opening additional connection (1), 1 of 31 pending slots used Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Connecting to ldaps:// dc1.ad.stolaf.edu:636 ldaps://dc2.ad.stolaf.edu:636 ldaps:// dc3.ad.stolaf.edu:636 ldaps://dc4.ad.stolaf.edu:636 Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): New libldap handle 0x564cf7ac04b0 Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Waiting for bind result... Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Bind successful Wed Feb 15 07:38:33 2023 : Info: rlm_ldap (ldap): Opening additional connection (2), 1 of 30 pending slots used Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Connecting to ldaps:// dc1.ad.stolaf.edu:636 ldaps://dc2.ad.stolaf.edu:636 ldaps:// dc3.ad.stolaf.edu:636 ldaps://dc4.ad.stolaf.edu:636 Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): New libldap handle 0x564cf7aae7a0 Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Waiting for bind result... Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Bind successful Wed Feb 15 07:38:33 2023 : Info: rlm_ldap (ldap): Opening additional connection (3), 1 of 29 pending slots used Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Connecting to ldaps:// dc1.ad.stolaf.edu:636 ldaps://dc2.ad.stolaf.edu:636 ldaps:// dc3.ad.stolaf.edu:636 ldaps://dc4.ad.stolaf.edu:636 Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): New libldap handle 0x564cf7ac1390 Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Waiting for bind result... Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Bind successful Wed Feb 15 07:38:33 2023 : Info: rlm_ldap (ldap): Opening additional connection (4), 1 of 28 pending slots used Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Connecting to ldaps:// dc1.ad.stolaf.edu:636 ldaps://dc2.ad.stolaf.edu:636 ldaps:// dc3.ad.stolaf.edu:636 ldaps://dc4.ad.stolaf.edu:636 Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): New libldap handle 0x564cf7ab5cc0 Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Waiting for bind result... Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Bind successful Connections from radius server to ldap server - note 5 connections to same server: [root@rad-dev ~]# lsof -i | grep radiusd | grep ldaps radiusd 1023016 radiusd 5u IPv6 16259147 0t0 TCP rad-dev:56548->[DEAD::BEEF]:ldaps (ESTABLISHED) radiusd 1023016 radiusd 6u IPv6 16259152 0t0 TCP rad-dev:56554->[DEAD::BEEF]:ldaps (ESTABLISHED) radiusd 1023016 radiusd 7u IPv6 16259157 0t0 TCP rad-dev:56556->[DEAD::BEEF]:ldaps (ESTABLISHED) radiusd 1023016 radiusd 8u IPv6 16259162 0t0 TCP rad-dev:56562->[DEAD::BEEF]:ldaps (ESTABLISHED) radiusd 1023016 radiusd 9u IPv6 16259167 0t0 TCP rad-dev:56566->[DEAD::BEEF]:ldaps (ESTABLISHED) dc1.ad.stolaf.edu <--> DEAD::BEEF Thanks! ajs -- *Tony Skalski (he/him/his)* System Administrator | IT Office: 507-786-3227 <(507)786-3227> 1510 St. Olaf Avenue Northfield, MN 55057 stolaf.edu
On Feb 15, 2023, at 2:11 PM, Tony Skalski via Freeradius-Users <freeradius-users@lists.freeradius.org> wrote:
We use EAP-PEAP and use ldap to normalize usernames and retrieve group membership (via ldaps). We recently found that our freeradius servers were only connecting to the first of four ldap servers specified in mods-available/ldap. The servers are listed on separate lines like this:
server = 'ldaps://dc1.ad.stolaf.edu' server = 'ldaps://dc2.ad.stolaf.edu' server = 'ldaps://dc3.ad.stolaf.edu' server = 'ldaps://dc4.ad.stolaf.edu'
When that configuration is used, the failover from one LDAP server to the other is handled by the LDAP libraries. FreeRADIUS isn't really involved. The LDAP libraries fail over from one server to another when a server fails. I don't think they do round robin or load balancing.
I consulted some openldap documentation, but am still left with the question: what is the best practice for listing and utilizing multiple LDAP servers (for both failover and load balancing)?
It depends what you want. FreeRADIUS allows you to configure almost anything. The configuration above only does failover. It doesn't do load balancing. But if it works... it's fine.
Also, is URI or hostname more preferred? (both seem to work in my testing, I do specify "port = 636" elsewhere in the config)
It doesn't really matter. Alan DeKok.
if you do a nslookup on "ad.stolaf.edu", all of the domain controller's IP's will be listed. hence, the round robin load balancing. while not "balanced", it is load sharing and provides high availability and fault tolerance. The missing piece is knowing when a DC is out of the mix and being able to adjust traffic, so that it does not go to that host. i use HAProxy, and have setup a VIP for ldap. HAProxy can perform a check against ldap, to confirm the service is up and operational. using this, the VIP can load balance across several hosts, and know when/if the services on the host, not just the host, are alive. there are some tweaks to OpenLDAP i had to perform, in order for the load balancing to work. one of the tweaks was setting olcSuffix to the DN i wanted all servers to reply to. with the HAProxy VIP in place, i point my FreeRADIUS instances at the ldap VIP, and dont worry about configuring failover in FreeRADIUS. On 2/15/23 2:11 PM, Tony Skalski via Freeradius-Users wrote:
Hi,
We use EAP-PEAP and use ldap to normalize usernames and retrieve group membership (via ldaps). We recently found that our freeradius servers were only connecting to the first of four ldap servers specified in mods-available/ldap. The servers are listed on separate lines like this:
server = 'ldaps://dc1.ad.stolaf.edu' server = 'ldaps://dc2.ad.stolaf.edu' server = 'ldaps://dc3.ad.stolaf.edu' server = 'ldaps://dc4.ad.stolaf.edu'
The certificates used for ldaps have SANs that include our domain name ' ad.stolaf.edu' and on my dev server I found that using "server = ' ad.stolaf.edu'" will start connections with all ldap servers, modulo DNS round robin results.
I consulted some openldap documentation, but am still left with the question: what is the best practice for listing and utilizing multiple LDAP servers (for both failover and load balancing)?
Also, is URI or hostname more preferred? (both seem to work in my testing, I do specify "port = 636" elsewhere in the config)
radiusd -X output from our production config (4 servers listed one per line):
Wed Feb 15 07:38:33 2023 : Info: rlm_ldap: libldap vendor: OpenLDAP, version: 20446 Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Initialising connection pool Wed Feb 15 07:38:33 2023 : Debug: pool { Wed Feb 15 07:38:33 2023 : Debug: start = 5 Wed Feb 15 07:38:33 2023 : Debug: min = 3 Wed Feb 15 07:38:33 2023 : Debug: max = 32 Wed Feb 15 07:38:33 2023 : Debug: spare = 10 Wed Feb 15 07:38:33 2023 : Debug: uses = 0 Wed Feb 15 07:38:33 2023 : Debug: lifetime = 0 Wed Feb 15 07:38:33 2023 : Debug: cleanup_interval = 30 Wed Feb 15 07:38:33 2023 : Debug: idle_timeout = 60 Wed Feb 15 07:38:33 2023 : Debug: retry_delay = 30 Wed Feb 15 07:38:33 2023 : Debug: spread = no Wed Feb 15 07:38:33 2023 : Debug: } Wed Feb 15 07:38:33 2023 : Info: rlm_ldap (ldap): Opening additional connection (0), 1 of 32 pending slots used Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Connecting to ldaps:// dc1.ad.stolaf.edu:636 ldaps://dc2.ad.stolaf.edu:636 ldaps:// dc3.ad.stolaf.edu:636 ldaps://dc4.ad.stolaf.edu:636 Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): New libldap handle 0x564cf7a99500 Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Waiting for bind result... Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Bind successful Wed Feb 15 07:38:33 2023 : Info: rlm_ldap (ldap): Opening additional connection (1), 1 of 31 pending slots used Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Connecting to ldaps:// dc1.ad.stolaf.edu:636 ldaps://dc2.ad.stolaf.edu:636 ldaps:// dc3.ad.stolaf.edu:636 ldaps://dc4.ad.stolaf.edu:636 Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): New libldap handle 0x564cf7ac04b0 Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Waiting for bind result... Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Bind successful Wed Feb 15 07:38:33 2023 : Info: rlm_ldap (ldap): Opening additional connection (2), 1 of 30 pending slots used Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Connecting to ldaps:// dc1.ad.stolaf.edu:636 ldaps://dc2.ad.stolaf.edu:636 ldaps:// dc3.ad.stolaf.edu:636 ldaps://dc4.ad.stolaf.edu:636 Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): New libldap handle 0x564cf7aae7a0 Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Waiting for bind result... Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Bind successful Wed Feb 15 07:38:33 2023 : Info: rlm_ldap (ldap): Opening additional connection (3), 1 of 29 pending slots used Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Connecting to ldaps:// dc1.ad.stolaf.edu:636 ldaps://dc2.ad.stolaf.edu:636 ldaps:// dc3.ad.stolaf.edu:636 ldaps://dc4.ad.stolaf.edu:636 Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): New libldap handle 0x564cf7ac1390 Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Waiting for bind result... Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Bind successful Wed Feb 15 07:38:33 2023 : Info: rlm_ldap (ldap): Opening additional connection (4), 1 of 28 pending slots used Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Connecting to ldaps:// dc1.ad.stolaf.edu:636 ldaps://dc2.ad.stolaf.edu:636 ldaps:// dc3.ad.stolaf.edu:636 ldaps://dc4.ad.stolaf.edu:636 Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): New libldap handle 0x564cf7ab5cc0 Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Waiting for bind result... Wed Feb 15 07:38:33 2023 : Debug: rlm_ldap (ldap): Bind successful
Connections from radius server to ldap server - note 5 connections to same server:
[root@rad-dev ~]# lsof -i | grep radiusd | grep ldaps radiusd 1023016 radiusd 5u IPv6 16259147 0t0 TCP rad-dev:56548->[DEAD::BEEF]:ldaps (ESTABLISHED) radiusd 1023016 radiusd 6u IPv6 16259152 0t0 TCP rad-dev:56554->[DEAD::BEEF]:ldaps (ESTABLISHED) radiusd 1023016 radiusd 7u IPv6 16259157 0t0 TCP rad-dev:56556->[DEAD::BEEF]:ldaps (ESTABLISHED) radiusd 1023016 radiusd 8u IPv6 16259162 0t0 TCP rad-dev:56562->[DEAD::BEEF]:ldaps (ESTABLISHED) radiusd 1023016 radiusd 9u IPv6 16259167 0t0 TCP rad-dev:56566->[DEAD::BEEF]:ldaps (ESTABLISHED)
dc1.ad.stolaf.edu <--> DEAD::BEEF
Thanks!
ajs -- *Tony Skalski (he/him/his)* System Administrator | IT Office: 507-786-3227 <(507)786-3227> 1510 St. Olaf Avenue Northfield, MN 55057 stolaf.edu - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Brendan Kearney wrote:
if you do a nslookup on "ad.stolaf.edu", all of the domain controller's IP's will be listed. hence, the round robin load balancing. while not "balanced", it is load sharing and provides high availability and fault tolerance. The missing piece is knowing when a DC is out of the mix and being able to adjust traffic, so that it does not go to that host.
The less involved solution is just to list the same RRDNS record multiple times instead of listing each server. Supposedly the LDAP libraries will then fail over and do another RRDNS lookup. I haven't actually tested it for a while, so as to exactly under what conditions libldap re-performs the DNS lookup, I'm not currently sure. ldap ldap_ad { server = "ad.foo.edu" server = "ad.foo.edu" server = "ad.foo.edu" server = "ad.foo.edu" port=636 ... }
Thanks for the ideas! I am curious, say I start FR and I have 4 connections open to 4 unique ldap servers (start = 4), what methodology does FR use to select a connection to use? Notwithstanding issues with the LDAP servers, given I have "min = 3", I would expect there to be a minimum of 3 connections at all times, correct? Are there any scenarios where FR will violate that? We had a wireless outage this morning and when I first checked I noticed there were 0 connections to our ldap servers - it wasn't an LDAP outage as other services did not have any issues. An FR restart got FR reconnected. On Wed, Feb 15, 2023 at 2:31 PM Brian Julin <BJulin@clarku.edu> wrote:
Brendan Kearney wrote:
if you do a nslookup on "ad.stolaf.edu", all of the domain controller's IP's will be listed. hence, the round robin load balancing. while not "balanced", it is load sharing and provides high availability and fault tolerance. The missing piece is knowing when a DC is out of the mix and being able to adjust traffic, so that it does not go to that host.
The less involved solution is just to list the same RRDNS record multiple times instead of listing each server. Supposedly the LDAP libraries will then fail over and do another RRDNS lookup. I haven't actually tested it for a while, so as to exactly under what conditions libldap re-performs the DNS lookup, I'm not currently sure.
ldap ldap_ad { server = "ad.foo.edu" server = "ad.foo.edu" server = "ad.foo.edu" server = "ad.foo.edu" port=636 ... } - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
-- *Tony Skalski (he/him/his)* System Administrator | IT Office: 507-786-3227 <(507)786-3227> 1510 St. Olaf Avenue Northfield, MN 55057 stolaf.edu
On Feb 15, 2023, at 3:59 PM, Tony Skalski via Freeradius-Users <freeradius-users@lists.freeradius.org> wrote:
Thanks for the ideas! I am curious, say I start FR and I have 4 connections open to 4 unique ldap servers (start = 4), what methodology does FR use to select a connection to use?
It tries to use the same one over and over. The reason is that it's likely to be faster. If that one is in use, it picks the next-most recently used.
Notwithstanding issues with the LDAP servers, given I have "min = 3", I would expect there to be a minimum of 3 connections at all times, correct?'
Mostly. If you also set "idle_timeout", it will close connections.
Are there any scenarios where FR will violate that? We had a wireless outage this morning and when I first checked I noticed there were 0 connections to our ldap servers - it wasn't an LDAP outage as other services did not have any issues. An FR restart got FR reconnected.
I haven't seen that. Are you using 3.2.1? Alan DeKok.
We do have "idle_timeout = 60" set, but also "lifetime = 0". Does "lifetime = 0" mean 0 seconds or infinite? And we are on the RH-approved 3.0.20. On Wed, Feb 15, 2023 at 3:27 PM Alan DeKok <aland@deployingradius.com> wrote:
On Feb 15, 2023, at 3:59 PM, Tony Skalski via Freeradius-Users < freeradius-users@lists.freeradius.org> wrote:
Thanks for the ideas! I am curious, say I start FR and I have 4
connections
open to 4 unique ldap servers (start = 4), what methodology does FR use to select a connection to use?
It tries to use the same one over and over. The reason is that it's likely to be faster.
If that one is in use, it picks the next-most recently used.
Notwithstanding issues with the LDAP servers, given I have "min = 3", I would expect there to be a minimum of 3 connections at all times, correct?'
Mostly. If you also set "idle_timeout", it will close connections.
Are there any scenarios where FR will violate that? We had a wireless outage this morning and when I first checked I noticed there were 0 connections to our ldap servers - it wasn't an LDAP outage as other services did not have any issues. An FR restart got FR reconnected.
I haven't seen that. Are you using 3.2.1?
Alan DeKok.
-- *Tony Skalski (he/him/his)* System Administrator | IT Office: 507-786-3227 <(507)786-3227> 1510 St. Olaf Avenue Northfield, MN 55057 stolaf.edu
On Feb 15, 2023, at 4:42 PM, Tony Skalski <ajs@stolaf.edu> wrote:
We do have "idle_timeout = 60" set, but also "lifetime = 0". Does "lifetime = 0" mean 0 seconds or infinite?
It means "infinite". If a connection hits idle_timeout, it will be closed. This may cause it to temporarily have fewer than "min" connections open.
And we are on the RH-approved 3.0.20.
Yeah, that's 3 years old. There have been many bugs fixed and features added since then. RH was *very* energetic about upgrading everything to OpenSSL 3, even in an allegedly "stable" release. But for FreeRADIUS, they never upgrade. I would very much recommend dropping the RH packages, and using the ones from http://packages.networkradius.com. Those packages are up to date, and contain all known fixes. I don't know what the benefit is of staying with the RH packages. RH *will not* do bug fixes for you. They will only do security fixes, of which there are very few. I'm sure RH is a nice company, but their history of "supporting" FreeRADIUS is largely limited to taking money, and telling people "we won't upgrade our packages to a newer version of the server". Alan DeKok.
participants (4)
-
Alan DeKok -
Brendan Kearney -
Brian Julin -
Tony Skalski