Hi all, FR 3.1.0 64aa7f9 I'm trying to get to the bottom of the ldap issues we're seeing with AD within freeradius. Basically when we take down one AD server (we have two sites, doesn't matter which it is), the ldap lookups from Freeradius have massive issues with timeouts, failure to bind errors etc. Now, I'd say this was an issue with AD, but we've tried this with various tools now, both linux and windows based, and we don't see any issues using, for example, ldapsearch, from the same server where FR is struggling. Obviously I would love it if someone would say "oh yes, you need to change /etc/ldap/ ... " but that's not so likely, so I wanted just any information on what ldap libraries FR uses, whether some of the code is internal or not so I can start to try and find a solution as I'm guessing this will be a management reason to go towards clearpass. Thanks! Andy
On 10 Mar 2016, at 15:04, Franks Andy (IT Technical Architecture Manager) <Andy.Franks@sath.nhs.uk> wrote:
Hi all, FR 3.1.0 64aa7f9 I'm trying to get to the bottom of the ldap issues we're seeing with AD within freeradius. Basically when we take down one AD server (we have two sites, doesn't matter which it is), the ldap lookups from Freeradius have massive issues with timeouts, failure to bind errors etc.
Post your ldap configuration (redacted), and packet traces, and debug output. You can do it off list if you'd like.
Now, I'd say this was an issue with AD, but we've tried this with various tools now, both linux and windows based, and we don't see any issues using, for example, ldapsearch, from the same server where FR is struggling.
Have you configured network timeouts? Do you have obscenely large connection pools? What is the server doing when AD goes down?
Obviously I would love it if someone would say "oh yes, you need to change /etc/ldap/ ... " but that's not so likely, so I wanted just any information on what ldap libraries FR uses
OpenLDAP's libldap, exactly the same as ldapsearch et al.
whether some of the code is internal or not so I can start to try and find a solution as I'm guessing this will be a management reason to go towards clearpass.
You know clearpass is still FreeRADIUS right? It'll have exactly the same issues, except then you'll introduce multiple layers between the people supporting the product and the people writing the code. If it's a support issue you may want to consider pinging sales@networkradius.com, there is support available, and it may be most cost effective than other commercial solutions. -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
Thanks Arran, So it seems that disabling "chase referals" might fix things, it's certainly a huge amount quicker and doesn't sit there binding until timeout. If, as part of the redundant-load-balance module handling, the "downed" server gets called, it waits for a search result from the dead server for the res_timeout figure, and then tries to get back it's required pool amount since it deleted the inviable connection, but it tries the same server. This is the bit I'd like to understand - it then waits "Connecting to..." for about 75-80 seconds before it idles out all the pooled connections, even if the idle option in mods-enabled/ldap is set to something quite a bit lower like 20 seconds. Is this expected? It's not a huge problem as stuff eventually catches up and when new connections are tried, the fail response is quite quick and will try the next server in the redundant... bit. Two questions : - Is the 75-80 seconds expected (not a huge deal) - Is it worth me debugging to find out why AD is giving out referals that presumably can't be connected to, and if so how? I tried adding debug options "debug 255" and "logdir /var/log" to ldap.conf, and turning on the 0x0028 option in mods-enabled/ldap but I can't see any debugging information - any clues? Thanks again! Andy. -----Original Message----- From: Freeradius-Users [mailto:freeradius-users-bounces+andy.franks=sath.nhs.uk@lists.freeradius.org] On Behalf Of Arran Cudbard-Bell Sent: 10 March 2016 15:27 To: FreeRadius users mailing list Subject: Re: Ldap query
On 10 Mar 2016, at 15:04, Franks Andy (IT Technical Architecture Manager) <Andy.Franks@sath.nhs.uk> wrote:
Hi all, FR 3.1.0 64aa7f9 I'm trying to get to the bottom of the ldap issues we're seeing with AD within freeradius. Basically when we take down one AD server (we have two sites, doesn't matter which it is), the ldap lookups from Freeradius have massive issues with timeouts, failure to bind errors etc.
Post your ldap configuration (redacted), and packet traces, and debug output. You can do it off list if you'd like.
Now, I'd say this was an issue with AD, but we've tried this with various tools now, both linux and windows based, and we don't see any issues using, for example, ldapsearch, from the same server where FR is struggling.
Have you configured network timeouts? Do you have obscenely large connection pools? What is the server doing when AD goes down?
Obviously I would love it if someone would say "oh yes, you need to change /etc/ldap/ ... " but that's not so likely, so I wanted just any information on what ldap libraries FR uses
OpenLDAP's libldap, exactly the same as ldapsearch et al.
whether some of the code is internal or not so I can start to try and find a solution as I'm guessing this will be a management reason to go towards clearpass.
You know clearpass is still FreeRADIUS right? It'll have exactly the same issues, except then you'll introduce multiple layers between the people supporting the product and the people writing the code. If it's a support issue you may want to consider pinging sales@networkradius.com, there is support available, and it may be most cost effective than other commercial solutions. -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
On Mar 10, 2016, at 5:43 PM, Franks Andy (IT Technical Architecture Manager) <Andy.Franks@sath.nhs.uk> wrote:
Thanks Arran, So it seems that disabling "chase referals" might fix things, it's certainly a huge amount quicker and doesn't sit there binding until timeout. If, as part of the redundant-load-balance module handling, the "downed" server gets called, it waits for a search result from the dead server for the res_timeout figure, and then tries to get back it's required pool amount since it deleted the inviable connection, but it tries the same server.
The problem here is interaction between the FreeRADIUS fail-over and the LDAP referrals. FreeRADIUS assumes that an LDAP server is just an LDAP server. When AD sends a referral... many things can happen.
This is the bit I'd like to understand - it then waits "Connecting to..." for about 75-80 seconds before it idles out all the pooled connections, even if the idle option in mods-enabled/ldap is set to something quite a bit lower like 20 seconds. Is this expected?
idle_timeout means "used successfully, and then not used for a while". If it sits in "connecting to" for 75-80 seconds, you want just a connection timeout. There should be one in the LDAP module configuration.
It's not a huge problem as stuff eventually catches up and when new connections are tried, the fail response is quite quick and will try the next server in the redundant... bit. Two questions : - Is the 75-80 seconds expected (not a huge deal)
Blame AD. And see the timeout configurations in the LDAP module.
- Is it worth me debugging to find out why AD is giving out referals that presumably can't be connected to, and if so how?
Uh... ask AD why it's referring you to servers that are down.
I tried adding debug options "debug 255" and "logdir /var/log" to ldap.conf, and turning on the 0x0028 option in mods-enabled/ldap but I can't see any debugging information - any clues?
You need to run the server in debug mode to see the LDAP debugging. Alan DeKok.
On 10 Mar 2016, at 23:26, Alan DeKok <aland@deployingradius.com> wrote:
On Mar 10, 2016, at 5:43 PM, Franks Andy (IT Technical Architecture Manager) <Andy.Franks@sath.nhs.uk> wrote:
Thanks Arran, So it seems that disabling "chase referals" might fix things, it's certainly a huge amount quicker and doesn't sit there binding until timeout. If, as part of the redundant-load-balance module handling, the "downed" server gets called, it waits for a search result from the dead server for the res_timeout figure, and then tries to get back it's required pool amount since it deleted the inviable connection, but it tries the same server.
The problem here is interaction between the FreeRADIUS fail-over and the LDAP referrals. FreeRADIUS assumes that an LDAP server is just an LDAP server. When AD sends a referral... many things can happen.
This is the bit I'd like to understand - it then waits "Connecting to..." for about 75-80 seconds before it idles out all the pooled connections, even if the idle option in mods-enabled/ldap is set to something quite a bit lower like 20 seconds. Is this expected?
idle_timeout means "used successfully, and then not used for a while". If it sits in "connecting to" for 75-80 seconds, you want just a connection timeout. There should be one in the LDAP module configuration.
It's in the pool {} section. I went through and standardized it, so it's the same for every module that supports a connection timeout. It's also documented in the pool section for every module. -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
Franks Andy (IT Technical Architecture Manager) wrote:
- Is it worth me debugging to find out why AD is giving out referals that presumably can't be connected to, and if so how? I tried adding debug options "debug 255" and "logdir /var/log" to ldap.conf, and turning on the 0x0028 option in mods-enabled/ldap but I can't see any debugging information - any clues?
LDAPv3 referrals are under-specified broken concept anyway. Personally I'd recommend to 1. always switch of client-side referral chasing unless you're really know what you're doing 2. with MS AD point your LDAP client to the global catalog port if you have a more complex AD forest for which you provide RADIUS service @freeradius developers: IMHO the default should be chase_referrals = no in the installed configuration. chase_referrals = yes causes more issues than it solves. Ciao, Michael.
participants (4)
-
Alan DeKok -
Arran Cudbard-Bell -
Franks Andy (IT Technical Architecture Manager) -
Michael Ströder