FreeRADIUS Thread Behaviour

[Doug Wussler]

On a RHEL 7 system, you can use “ps –eLF | grep ^fradius” to watch the v 3.0.15 freeradius process
threads turn over.  In particular you can see the start time of the main process, which remains
constant, and the start times of the threads, which change as the threads die off and are
re-spawned.  For example:

UID        PID  PPID   LWP  C NLWP    SZ   RSS PSR STIME TTY          TIME CMD

<root at auth8:0 /var/log/radius>$ ps -eLF | grep ^fradius
fradius   3840     1  3840  1    9 535166 383420 7 03:39 ?        00:06:05 /usr/local/sbin/radiusd
fradius   3840     1 35620  1    9 535166 383420 12 10:16 ?       00:00:23 /usr/local/sbin/radiusd
fradius   3840     1 35621  1    9 535166 383420 17 10:16 ?       00:00:23 /usr/local/sbin/radiusd
fradius   3840     1 35622  1    9 535166 383420 6 10:16 ?        00:00:22 /usr/local/sbin/radiusd
fradius   3840     1 35623  1    9 535166 383420 2 10:16 ?        00:00:22 /usr/local/sbin/radiusd
fradius   3840     1 35626  1    9 535166 383420 3 10:20 ?        00:00:19 /usr/local/sbin/radiusd
fradius   3840     1 35627  1    9 535166 383420 3 10:20 ?        00:00:19 /usr/local/sbin/radiusd
fradius   3840     1 35628  1    9 535166 383420 12 10:22 ?       00:00:17 /usr/local/sbin/radiusd
fradius   3840     1 35629  1    9 535166 383420 12 10:22 ?       00:00:17 /usr/local/sbin/radius

Note that a few minutes later, two of the threads have died off and two others have fired up.

<root at auth8:0 /var/log/radius>$ ps -eLF | grep ^fradius
fradius   3840     1  3840  1    9 535166 385280 4 03:39 ?        00:06:07 /usr/local/sbin/radiusd
fradius   3840     1 35622  1    9 535166 385280 2 10:16 ?        00:00:24 /usr/local/sbin/radiusd
fradius   3840     1 35623  1    9 535166 385280 2 10:16 ?        00:00:23 /usr/local/sbin/radiusd
fradius   3840     1 35626  1    9 535166 385280 12 10:20 ?       00:00:21 /usr/local/sbin/radiusd
fradius   3840     1 35627  1    9 535166 385280 6 10:20 ?        00:00:21 /usr/local/sbin/radiusd
fradius   3840     1 35628  1    9 535166 385280 12 10:22 ?       00:00:19 /usr/local/sbin/radiusd
fradius   3840     1 35629  1    9 535166 385280 8 10:22 ?        00:00:19 /usr/local/sbin/radiusd
fradius   3840     1 35701  1    9 535166 385280 6 10:42 ?        00:00:00 /usr/local/sbin/radiusd
fradius   3840     1 35702  1    9 535166 385280 2 10:42 ?        00:00:00 /usr/local/sbin/radiusd

When a thread dies off, it tears down its LDAP connection and when the new thread starts up
it stands up a new LDAP connection.  If you start these threads on an idle system you can see
(using Wireshark) that they do not turn over and the LDAP connections remain stable through
the use of keepalives.  But on a busy system, the threads turn over and thus so do the LDAP
connections.  I thought I had things configured so that threads never die off.

The question I am trying to answer is why the threads turn over at all, and in particular
so frequently.  Here is my config for the settings I thought control the thread behavior:

From “radiusd.conf”
# THREAD POOL CONFIGURATION
thread pool {
        start_servers = 8
        max_servers = 32
        min_spare_servers = 2
        max_spare_servers = 8
        # max_queue_size = 65536
        #  '0' = 'infinity', or 'the servers never exit'
        max_requests_per_server = 0
        #  Automatically limit the number of accounting requests.
        auto_limit_acct = no
}

From my “ldap” module:
        pool {
                start = ${thread[pool].start_servers}
                min   = ${thread[pool].min_spare_servers}
                max   = ${thread[pool].max_servers}
                spare = ${thread[pool].max_spare_servers}
                uses  = 0

                retry_delay  =  5
                lifetime     =  0
                idle_timeout =  0
        }

        options {
                rebind          = no
                res_timeout     =  5
                srv_timelimit   =  5
                net_timeout     =  3
                # ldap_debug    = 0x0028
        }


Have I overlooked a setting or misunderstood the way these settings work?

The motivation for looking into this is that I have occasions where freeradius
reports “Could not start TLS: Can't contact LDAP server”
which seems to only occur when a new thread fires up and tries to start TLS
to the LDAP server.   This happens randomly for a few seconds a couple times a month.
Regardless of whether or not this indicates a problem
with my LDAP server or the connection to it, I would like to know why
I have threads continuously dying and spawning every few minutes even
though my config, as I understand it, is set so that the threads should not die.

Doug


Doug Wussler
Florida State University
RK Shaw Building
644 W. Call Street
Tallahassee, FL  32304