Initial access requests getting dropped, successive requests succeed
Hi folks, We run three FreeRADIUS servers as proxies in front of some LDAP servers for network authentication. These servers have been running for well over a year without issue until recently. Two of the three servers have started inexplicably dropping access requests, but still respond intermittently. As an isolated test, I have a Juniper switch configured to authenticate to one of the FreeRADIUS servers, which in turn authenticates against one backend LDAP server. When I try to log into the switch, tcpdump on the RADIUS server confirms that it receives an Access-Request packet. I've stopped the normal daemon and am running `freeradius -X` on the server, but it prints only a single line in response to the Access-Request: Ready to process requests. It prints this same line each time a request is dropped. I can't find any information about what's actually happening. tcpdump confirms that FreeRADIUS is receiving the Access-Request packet, but it does not even attempt to contact the LDAP server. However, when I attempt to authenticate again a few seconds later (after the switch's first request has timed out), the RADIUS server responds normally with a successful authentication, with no indication (AFAICT) of any error. I can log out and immediately log back in with no problems, but if I wait for more than a few seconds, the request gets dropped again. Even stranger, this only appears to affect the primary and secondary server; forcing authentication requests to the tertiary server succeeds. To rule out LDAP as a problem, I've been trying to authenticate using an account defined locally on the FreeRADIUS server, but I have the same problem: The initial access request gets dropped; successive requests succeed. Here is my config from /etc/freeradius/sites-enabled/default: authorize { preprocess suffix files ldap expiration logintime } authenticate { Auth-Type LDAP { ldap } } preacct { preprocess acct_unique suffix files } accounting { detail unix radutmp exec attr_filter.accounting_response } session { radutmp } post-auth { exec } The three servers were originally all running v2.1.12. We upgraded the primary to v2.2.9 but it still has the same issue. Hard-coding the IP of the LDAP server didn't help, so it doesn't appear to be related to DNS, nor do there appear to be any disk or memory issues. I'm really at a loss for what to try next, other than blowing them away and rebuilding all three servers. Any pointers are much appreciated. Jeremy
On Mar 31, 2017, at 10:01 AM, Jeremy Stretch via Freeradius-Users <freeradius-users@lists.freeradius.org> wrote:
As an isolated test, I have a Juniper switch configured to authenticate to one of the FreeRADIUS servers, which in turn authenticates against one backend LDAP server. When I try to log into the switch, tcpdump on the RADIUS server confirms that it receives an Access-Request packet. I've stopped the normal daemon and am running `freeradius -X` on the server, but it prints only a single line in response to the Access-Request:
Ready to process requests.
It prints this same line each time a request is dropped.
That means that the OS told FR there was a packet, but when it tried to read the packet, there was no RADIUS packet. If it was from an unknown client, it would print that. If it was a malformed packet, it would print that. So something else is going on.
I can't find any information about what's actually happening. tcpdump confirms that FreeRADIUS is receiving the Access-Request packet, but it does not even attempt to contact the LDAP server.
If it doesn't get a RADIUS packet, it doesn't run that through the virtual server, and it doesn't contact LDAP.
However, when I attempt to authenticate again a few seconds later (after the switch's first request has timed out), the RADIUS server responds normally with a successful authentication, with no indication (AFAICT) of any error. I can log out and immediately log back in with no problems, but if I wait for more than a few seconds, the request gets dropped again. Even stranger, this only appears to affect the primary and secondary server; forcing authentication requests to the tertiary server succeeds.
Honestly, it sounds like an OS problem.
To rule out LDAP as a problem,
It's not an LDAP problem.
The three servers were originally all running v2.1.12. We upgraded the primary to v2.2.9 but it still has the same issue.
Which sounds like an OS issue.
I'm really at a loss for what to try next, other than blowing them away and rebuilding all three servers. Any pointers are much appreciated.
Try using a new machine. If that works, it's a machine / OS issue. Alan DeKok.
On Mar 31, 2017, at 10:27 AM, Alan DeKok <aland@deployingradius.com> wrote:
On Mar 31, 2017, at 10:01 AM, Jeremy Stretch via Freeradius-Users <freeradius-users@lists.freeradius.org> wrote:
As an isolated test, I have a Juniper switch configured to authenticate to one of the FreeRADIUS servers, which in turn authenticates against one backend LDAP server. When I try to log into the switch, tcpdump on the RADIUS server confirms that it receives an Access-Request packet. I've stopped the normal daemon and am running `freeradius -X` on the server, but it prints only a single line in response to the Access-Request:
Ready to process requests.
It prints this same line each time a request is dropped.
That means that the OS told FR there was a packet, but when it tried to read the packet, there was no RADIUS packet.
If it was from an unknown client, it would print that. If it was a malformed packet, it would print that. So something else is going on.
It might be RPF causing the issue. I vaguely remember this being a symptom of that, but I could be incorrect. http://stackoverflow.com/questions/31000939/disable-reverse-path-filtering-f... Simple to check... -Arran Arran Cudbard-Bell FreeRADIUS Core Developer FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
Tracing "freeradius -X" yields the following each time a request is dropped: select(9, [4 6 7 8], NULL, NULL, NULL) = 1 (in [6]) recvfrom(6, 0x7ffe802938d0, 4, 2, 0x7ffe802938e0, 0x7ffe802938cc) = -1 EAGAIN (Resource temporarily unavailable) write(1, "Ready to process requests.\n", 27Ready to process requests.) = 27 Not sure what to make of that. Seems like it might be related to this bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=808293;msg=2 As far as I'm aware though, nothing significant has been changed on these serves. On Fri, Mar 31, 2017 at 10:01 AM, Jeremy Stretch <jstretch@digitalocean.com> wrote:
Hi folks,
We run three FreeRADIUS servers as proxies in front of some LDAP servers for network authentication. These servers have been running for well over a year without issue until recently. Two of the three servers have started inexplicably dropping access requests, but still respond intermittently.
As an isolated test, I have a Juniper switch configured to authenticate to one of the FreeRADIUS servers, which in turn authenticates against one backend LDAP server. When I try to log into the switch, tcpdump on the RADIUS server confirms that it receives an Access-Request packet. I've stopped the normal daemon and am running `freeradius -X` on the server, but it prints only a single line in response to the Access-Request:
Ready to process requests.
It prints this same line each time a request is dropped. I can't find any information about what's actually happening. tcpdump confirms that FreeRADIUS is receiving the Access-Request packet, but it does not even attempt to contact the LDAP server.
However, when I attempt to authenticate again a few seconds later (after the switch's first request has timed out), the RADIUS server responds normally with a successful authentication, with no indication (AFAICT) of any error. I can log out and immediately log back in with no problems, but if I wait for more than a few seconds, the request gets dropped again. Even stranger, this only appears to affect the primary and secondary server; forcing authentication requests to the tertiary server succeeds.
To rule out LDAP as a problem, I've been trying to authenticate using an account defined locally on the FreeRADIUS server, but I have the same problem: The initial access request gets dropped; successive requests succeed. Here is my config from /etc/freeradius/sites-enabled/default:
authorize { preprocess suffix files ldap expiration logintime } authenticate { Auth-Type LDAP { ldap } } preacct { preprocess acct_unique suffix files } accounting { detail unix radutmp exec attr_filter.accounting_response } session { radutmp } post-auth { exec }
The three servers were originally all running v2.1.12. We upgraded the primary to v2.2.9 but it still has the same issue. Hard-coding the IP of the LDAP server didn't help, so it doesn't appear to be related to DNS, nor do there appear to be any disk or memory issues.
I'm really at a loss for what to try next, other than blowing them away and rebuilding all three servers. Any pointers are much appreciated.
Jeremy
On Mar 31, 2017, at 1:34 PM, Jeremy Stretch via Freeradius-Users <freeradius-users@lists.freeradius.org> wrote:
Tracing "freeradius -X" yields the following each time a request is dropped:
select(9, [4 6 7 8], NULL, NULL, NULL) = 1 (in [6]) recvfrom(6, 0x7ffe802938d0, 4, 2, 0x7ffe802938e0, 0x7ffe802938cc) = -1 EAGAIN (Resource temporarily unavailable) write(1, "Ready to process requests.\n", 27Ready to process requests.) = 27
Yup. It's an OS issue.
Not sure what to make of that. Seems like it might be related to this bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=808293;msg=2 As far as I'm aware though, nothing significant has been changed on these serves.
There was probably a kernel change. Run a kernel that works. No amount of poking FreeRADIUS will fix the underlying OS. Alan DeKok.
participants (3)
-
Alan DeKok -
Arran Cudbard-Bell -
Jeremy Stretch