FreeRADIUS Thread Behaviour
On a RHEL 7 system, you can use “ps –eLF | grep ^fradius” to watch the v 3.0.15 freeradius process threads turn over. In particular you can see the start time of the main process, which remains constant, and the start times of the threads, which change as the threads die off and are re-spawned. For example: UID PID PPID LWP C NLWP SZ RSS PSR STIME TTY TIME CMD <root@auth8:0 /var/log/radius>$ ps -eLF | grep ^fradius fradius 3840 1 3840 1 9 535166 383420 7 03:39 ? 00:06:05 /usr/local/sbin/radiusd fradius 3840 1 35620 1 9 535166 383420 12 10:16 ? 00:00:23 /usr/local/sbin/radiusd fradius 3840 1 35621 1 9 535166 383420 17 10:16 ? 00:00:23 /usr/local/sbin/radiusd fradius 3840 1 35622 1 9 535166 383420 6 10:16 ? 00:00:22 /usr/local/sbin/radiusd fradius 3840 1 35623 1 9 535166 383420 2 10:16 ? 00:00:22 /usr/local/sbin/radiusd fradius 3840 1 35626 1 9 535166 383420 3 10:20 ? 00:00:19 /usr/local/sbin/radiusd fradius 3840 1 35627 1 9 535166 383420 3 10:20 ? 00:00:19 /usr/local/sbin/radiusd fradius 3840 1 35628 1 9 535166 383420 12 10:22 ? 00:00:17 /usr/local/sbin/radiusd fradius 3840 1 35629 1 9 535166 383420 12 10:22 ? 00:00:17 /usr/local/sbin/radius Note that a few minutes later, two of the threads have died off and two others have fired up. <root@auth8:0 /var/log/radius>$ ps -eLF | grep ^fradius fradius 3840 1 3840 1 9 535166 385280 4 03:39 ? 00:06:07 /usr/local/sbin/radiusd fradius 3840 1 35622 1 9 535166 385280 2 10:16 ? 00:00:24 /usr/local/sbin/radiusd fradius 3840 1 35623 1 9 535166 385280 2 10:16 ? 00:00:23 /usr/local/sbin/radiusd fradius 3840 1 35626 1 9 535166 385280 12 10:20 ? 00:00:21 /usr/local/sbin/radiusd fradius 3840 1 35627 1 9 535166 385280 6 10:20 ? 00:00:21 /usr/local/sbin/radiusd fradius 3840 1 35628 1 9 535166 385280 12 10:22 ? 00:00:19 /usr/local/sbin/radiusd fradius 3840 1 35629 1 9 535166 385280 8 10:22 ? 00:00:19 /usr/local/sbin/radiusd fradius 3840 1 35701 1 9 535166 385280 6 10:42 ? 00:00:00 /usr/local/sbin/radiusd fradius 3840 1 35702 1 9 535166 385280 2 10:42 ? 00:00:00 /usr/local/sbin/radiusd When a thread dies off, it tears down its LDAP connection and when the new thread starts up it stands up a new LDAP connection. If you start these threads on an idle system you can see (using Wireshark) that they do not turn over and the LDAP connections remain stable through the use of keepalives. But on a busy system, the threads turn over and thus so do the LDAP connections. I thought I had things configured so that threads never die off. The question I am trying to answer is why the threads turn over at all, and in particular so frequently. Here is my config for the settings I thought control the thread behavior: From “radiusd.conf” # THREAD POOL CONFIGURATION thread pool { start_servers = 8 max_servers = 32 min_spare_servers = 2 max_spare_servers = 8 # max_queue_size = 65536 # '0' = 'infinity', or 'the servers never exit' max_requests_per_server = 0 # Automatically limit the number of accounting requests. auto_limit_acct = no } From my “ldap” module: pool { start = ${thread[pool].start_servers} min = ${thread[pool].min_spare_servers} max = ${thread[pool].max_servers} spare = ${thread[pool].max_spare_servers} uses = 0 retry_delay = 5 lifetime = 0 idle_timeout = 0 } options { rebind = no res_timeout = 5 srv_timelimit = 5 net_timeout = 3 # ldap_debug = 0x0028 } Have I overlooked a setting or misunderstood the way these settings work? The motivation for looking into this is that I have occasions where freeradius reports “Could not start TLS: Can't contact LDAP server” which seems to only occur when a new thread fires up and tries to start TLS to the LDAP server. This happens randomly for a few seconds a couple times a month. Regardless of whether or not this indicates a problem with my LDAP server or the connection to it, I would like to know why I have threads continuously dying and spawning every few minutes even though my config, as I understand it, is set so that the threads should not die. Doug Doug Wussler Florida State University RK Shaw Building 644 W. Call Street Tallahassee, FL 32304
On Oct 16, 2017, at 11:21 AM, Doug Wussler <doug.wussler@fsu.edu> wrote:
When a thread dies off, it tears down its LDAP connection and when the new thread starts up it stands up a new LDAP connection. If you start these threads on an idle system you can see (using Wireshark) that they do not turn over and the LDAP connections remain stable through the use of keepalives. But on a busy system, the threads turn over and thus so do the LDAP connections. I thought I had things configured so that threads never die off.
The documentation describes when threads die off.
The question I am trying to answer is why the threads turn over at all, and in particular so frequently. Here is my config for the settings I thought control the thread behavior:
From “radiusd.conf” # THREAD POOL CONFIGURATION thread pool { start_servers = 8 max_servers = 32 min_spare_servers = 2 max_spare_servers = 8
i.e. if there are more than 8 spare threads, one is deleted. If there are fewer than 2 spare threads, one is created.
Have I overlooked a setting or misunderstood the way these settings work?
I'm not sure. The documentation is pretty clear on this.
The motivation for looking into this is that I have occasions where freeradius reports “Could not start TLS: Can't contact LDAP server” which seems to only occur when a new thread fires up and tries to start TLS to the LDAP server.
Because the OpenLDAP API is crappy. There's no explicit "initialize" routine, so the OpenLDAP code has to guess, and can get it wrong.
This happens randomly for a few seconds a couple times a month. Regardless of whether or not this indicates a problem with my LDAP server or the connection to it, I would like to know why I have threads continuously dying and spawning every few minutes even though my config, as I understand it, is set so that the threads should not die.
Again, the documentation is VERY clear on this: # It does this by periodically checking how many servers are # waiting for a request. If there are fewer than # min_spare_servers, it creates a new spare. If there are # more than max_spare_servers, some of the spares die off. If you don't want threads to spawn / die, se: start_servers = 32 max_servers = 32 min_spare_servers = 0 max_spare_servers = 32 Alan DeKok.
Alan DeKok wrote:
On Oct 16, 2017, at 11:21 AM, Doug Wussler <doug.wussler@fsu.edu> wrote:
The motivation for looking into this is that I have occasions where freeradius reports “Could not start TLS: Can't contact LDAP server” which seems to only occur when a new thread fires up and tries to start TLS to the LDAP server.
Because the OpenLDAP API is crappy. There's no explicit "initialize" routine, so the OpenLDAP code has to guess, and can get it wrong.
What do you mean by "initialize" and "has to guess"? Disclaimer: I did not look at FreeRADIUS code. No news that OpenLDAP API (with its roots in RFC 1823) has real deficiencies. Everybody involved agrees on that. But most issues developers experience are caused by the lazy connect in libldap: ldap_initialize() and friends do *not* to the LDAP server. You have send the first LDAP operation to really connect and handle connection errors *after* that. In multi-threaded software with a persistent connection (pool) I usually use a lock to ensure that ldap_initialize() and bind (simple or SASL) were successfully done before the connection can be used by another thread. That's also the reason why I usually avoid a code path with anon-search-without-bind because I want to provoke a connect error with an explicit bind right from the beginning. This also might get tricky when the OpenLDAP server drops idle connections or was simply restarted and your multi-threaded LDAP client has to re-connect. Ciao, Michael.
On Oct 16, 2017, at 12:53 PM, Michael Ströder <michael@stroeder.com> wrote:
What do you mean by "initialize" and "has to guess"?
Disclaimer: I did not look at FreeRADIUS code.
It isn't in the FreRADIUS code. It's in the OpenLDAP code. If you go through the OpenLDAP client code, you'll see global variables, and all kinds of magic to determine if the LDAP library is initialized. That's bad.
No news that OpenLDAP API (with its roots in RFC 1823) has real deficiencies. Everybody involved agrees on that.
But most issues developers experience are caused by the lazy connect in libldap: ldap_initialize() and friends do *not* to the LDAP server. You have send the first LDAP operation to really connect and handle connection errors *after* that.
Yes, we've had issues with that ourselves.
In multi-threaded software with a persistent connection (pool) I usually use a lock to ensure that ldap_initialize() and bind (simple or SASL) were successfully done before the connection can be used by another thread. That's also the reason why I usually avoid a code path with anon-search-without-bind because I want to provoke a connect error with an explicit bind right from the beginning.
FreeRADIUS doesn't do a search until the LDAP connection is initialized. FreeRADIUS has a connection manager which handles all of that. The connections can move between threads, but only after a connection is explicitly released by a thread. So there is no multi-threaded usage of LDAP connections. Each thread uses a connection, and when it's done using the connection, the connection is released back to the connection manager for use by another thread.
This also might get tricky when the OpenLDAP server drops idle connections or was simply restarted and your multi-threaded LDAP client has to re-connect.
The connection manager (src/main/connection.c) handles all of that. Alan DeKok.
Alan DeKok wrote:
If you go through the OpenLDAP client code, you'll see global variables, and all kinds of magic to determine if the LDAP library is initialized. That's bad. Yes. That's why python-ldap serializes all calls to ldap_initialize() with a module-wide function lock. After that connection-specific locks are used.
Ciao, Michael.
Michael Ströder wrote:
Alan DeKok wrote:
If you go through the OpenLDAP client code, you'll see global variables, and all kinds of magic to determine if the LDAP library is initialized. That's bad.
Yes. That's why python-ldap serializes all calls to ldap_initialize() with a module-wide function lock. After that connection-specific locks are used.
And btw: Everybody should really avoid pam_ldap/nss_ldap and use a service like nslcd (aka nss-pam-ldapd) or sssd instead. Ciao, Michael.
participants (3)
-
Alan DeKok -
Doug Wussler -
Michael Ströder