Initial access requests getting dropped, successive requests succeed

[Jeremy Stretch]

Hi folks,

We run three FreeRADIUS servers as proxies in front of some LDAP servers
for network authentication. These servers have been running for well over a
year without issue until recently. Two of the three servers have started
inexplicably dropping access requests, but still respond intermittently.

As an isolated test, I have a Juniper switch configured to authenticate to
one of the FreeRADIUS servers, which in turn authenticates against one
backend LDAP server. When I try to log into the switch, tcpdump on the
RADIUS server confirms that it receives an Access-Request packet. I've
stopped the normal daemon and am running `freeradius -X` on the server, but
it prints only a single line in response to the Access-Request:

    Ready to process requests.

It prints this same line each time a request is dropped. I can't find any
information about what's actually happening. tcpdump confirms that
FreeRADIUS is receiving the Access-Request packet, but it does not even
attempt to contact the LDAP server.

However, when I attempt to authenticate again a few seconds later (after
the switch's first request has timed out), the RADIUS server responds
normally with a successful authentication, with no indication (AFAICT) of
any error. I can log out and immediately log back in with no problems, but
if I wait for more than a few seconds, the request gets dropped again. Even
stranger, this only appears to affect the primary and secondary server;
forcing authentication requests to the tertiary server succeeds.

To rule out LDAP as a problem, I've been trying to authenticate using an
account defined locally on the FreeRADIUS server, but I have the same
problem: The initial access request gets dropped; successive requests
succeed. Here is my config from /etc/freeradius/sites-enabled/default:

    authorize {
        preprocess
        suffix
        files
        ldap
        expiration
        logintime
    }
    authenticate {
        Auth-Type LDAP {
            ldap
        }
    }
    preacct {
        preprocess
        acct_unique
        suffix
        files
    }
    accounting {
        detail
        unix
        radutmp
        exec
        attr_filter.accounting_response
    }
    session {
        radutmp
    }
    post-auth {
        exec
    }

The three servers were originally all running v2.1.12. We upgraded the
primary to v2.2.9 but it still has the same issue. Hard-coding the IP of
the LDAP server didn't help, so it doesn't appear to be related to DNS, nor
do there appear to be any disk or memory issues.

I'm really at a loss for what to try next, other than blowing them away and
rebuilding all three servers. Any pointers are much appreciated.

Jeremy