FreeRADIUS allows connections locally, but not remotely
Hi everyone. I have a difficult problem that appears to have come out of the blue. After rebooting the Debian Wheezy server that hosts our RADIUS authentication this morning, suddenly FreeRADIUS v2.1.12+dfsg-1.2 is no longer working. I can successfully test the connection locally using 'radtest' like this: # /usr/bin/radtest customer password localhost:1812 5 localsecret -4 127.0.0.1 Sending Access-Request of id 218 to 127.0.0.1 port 1812 User-Name = "customer" User-Password = "password" NAS-IP-Address = 127.0.0.1 NAS-Port = 5 Message-Authenticator = 0x00000000000000000000000000000000 Framed-Protocol = PPP rad_recv: Access-Accept packet from host 127.0.0.1 port 1812, id=218, length=38 Framed-Protocol = PPP Service-Type = Framed-User Framed-Compression = Van-Jacobson-TCP-IP I get the rad_recv response right away. When I test it from the same machine, but *saying* I'm connecting remotely, it apparently times out: www:/etc/freeradius# /usr/bin/radtest customer password localhost:1812 5 remotesecret -4 XXX.XXX.XXX.254 Sending Access-Request of id 34 to 127.0.0.1 port 1812 User-Name = "customer" User-Password = "password" NAS-IP-Address = XXX.XXX.XXX.254 NAS-Port = 5 Message-Authenticator = 0x00000000000000000000000000000000 Framed-Protocol = PPP Sending Access-Request of id 34 to 127.0.0.1 port 1812 User-Name = "customer" User-Password = "password" NAS-IP-Address = XXX.XXX.XXX.254 NAS-Port = 5 Message-Authenticator = 0x00000000000000000000000000000000 Framed-Protocol = PPP ^C The connection evidently times out and radtest retries, at which point I hit Ctrl-C. I get the same results when I actually test the connection from a remote site as when I pretend to make the connection from a remote site with -4 XXX.XXX.XXX.254. I can't really test *from* that site, since it's a Cisco AS5300. The only thing I can think of that *might* have changed was the Debian package version, but I can't even confirm that really, and it's in the old stable branch, which I believe is no longer supported with new patches. Nothing has changed in the FreeRADIUS configuration since about 2010. Any assistance would be appreciated!
On Dec 28, 2015, at 5:29 PM, Ernie Dunbar <maillist@lightspeed.ca> wrote:
Hi everyone. I have a difficult problem that appears to have come out of the blue.
After rebooting the Debian Wheezy server that hosts our RADIUS authentication this morning, suddenly FreeRADIUS v2.1.12+dfsg-1.2 is no longer working
..
When I test it from the same machine, but *saying* I'm connecting remotely, it apparently times out:
It's a firewall / networking issue. It's not FreeRADIUS. Alan DeKok.
On 2015-12-28 14:49, Alan DeKok wrote:
On Dec 28, 2015, at 5:29 PM, Ernie Dunbar <maillist@lightspeed.ca> wrote:
Hi everyone. I have a difficult problem that appears to have come out of the blue.
After rebooting the Debian Wheezy server that hosts our RADIUS authentication this morning, suddenly FreeRADIUS v2.1.12+dfsg-1.2 is no longer working
..
When I test it from the same machine, but *saying* I'm connecting remotely, it apparently times out:
It's a firewall / networking issue. It's not FreeRADIUS.
Alan DeKok.
Okay, then why would radtest fail with the command-line argument "-4 XXX.XXX.XXX.254" from localhost? Is this not merely data passed to the FreeRADIUS server about the source of the data, as opposed to radtest spoofing the source IP address and making a UDP connection? I am requesting a connection to localhost with radtest. Also, there is no firewall rule blocking the connection. I also tried the radtest command locally after flushing the IPTables rules. I have tried to specify FreeRADIUS' listening IP address instead of * (in radiusd.conf), and that hasn't worked either. I think I've been pretty thorough trying to rule out networking issues, but maybe I've missed something?
On Dec 28, 2015, at 6:22 PM, Ernie Dunbar <maillist@lightspeed.ca> wrote:
Okay, then why would radtest fail with the command-line argument "-4 XXX.XXX.XXX.254" from localhost?
It's your network. You should be able to see what's going on with it. And the firewall / filter rules for localhost are usually more open than that for the network.
Is this not merely data passed to the FreeRADIUS server
Please don't say that. Radclient is *sending a packet over the network*. saying "data passed to FreeRADIUS" is (a) vague and meaningless, and (b) makes it sound like data is magically going from A to B. It's not. It's sending a packet over the network. If the packet doesn't arrive, blame the network.
about the source of the data, as opposed to radtest spoofing the source IP address and making a UDP connection? I am requesting a connection to localhost with radtest.
You're sending a packet. If the packet doesn't arrive, blame the network.
Also, there is no firewall rule blocking the connection. I also tried the radtest command locally after flushing the IPTables rules.
Well, there's nothing magical in FreeRADIUS saying "suddenly start ignoring packets".
I have tried to specify FreeRADIUS' listening IP address instead of * (in radiusd.conf), and that hasn't worked either.
Run the server in debug mode. If it doesn't show any packets received... it's not receiving any packets. Blame the OS and the network. Not FreeRADIUS.
I think I've been pretty thorough trying to rule out networking issues, but maybe I've missed something?
Probably. Alan DeKok.
On 2015-12-28 15:32, Alan DeKok wrote:
Run the server in debug mode. If it doesn't show any packets received... it's not receiving any packets.
Blame the OS and the network. Not FreeRADIUS.
Okay, fair enough. I've made some changes to the FreeRADIUS configuration with respect to the listening port and IP address, and I've added a new "client" for remote testing. Here's the redacted client configuration, according to FreeRADIUS' debug output. radiusd: #### Loading Clients #### client 127.0.0.1 { require_message_authenticator = no secret = "SECRET" shortname = "localhost" } client 204.XXX.XX.254 { require_message_authenticator = no secret = "SECRET" shortname = "dialup" } client 65.XXX.XX.178 { require_message_authenticator = no secret = "SECRET" shortname = "dialup7" } and the IP addresses and ports section: radiusd: #### Opening IP addresses and Ports #### listen { type = "auth" ipaddr = 206.XXX.XX.4 port = 1812 } listen { type = "acct" ipaddr = 206.XXX.XX.4 port = 1813 } listen { type = "auth" ipaddr = 127.0.0.1 port = 18120 } My understanding is that port 18120 is only used internally, especially after this debug output: Listening on authentication address 127.0.0.1 port 18120 as server inner-tunnel. Then I try to get FreeRADIUS to authenticate locally, as I've been successful in doing today: $ /usr/bin/radtest customer password 206.XXX.XX.4:1812 5 SECRET And for each unsuccessful attempt, I get this output from FreeRADIUS: Ignoring request to authentication address 206.XXX.XX.4 port 1812 from unknown client 206.XXX.XX.205 port 47980 Ready to process requests. Ignoring request to authentication address 206.XXX.XX.4 port 1812 from unknown client 206.XXX.XX.205 port 47980 Ready to process requests. Ok, Fair enough. Those attempts are coming through this server's other IP address (206.XXX.XX.205) to its other IP address (206.XXX.XX.4). It's pretty clear that packets are reaching the FreeRADIUS daemon, it's just rejecting them because this other IP address isn't in the clients configuration. No problem, I switch to another Linux box that *does* have its IP address configured in the clients: # radtest customer password 206.12.82.4:1812 5 l1ghtsp33d -4 65.XX.XXX.178 Sending Access-Request of id 43 to 206.XXX.XX.4 port 1812 User-Name = "customer" User-Password = "password" NAS-IP-Address = 65.XX.XXX.178 NAS-Port = 5 Message-Authenticator = 0x00000000000000000000000000000000 Framed-Protocol = PPP Sending Access-Request of id 43 to 206.XXX.XX.4 port 1812 User-Name = "customer" User-Password = "password" NAS-IP-Address = 65.XX.XXX.178 NAS-Port = 5 Message-Authenticator = 0x00000000000000000000000000000000 Framed-Protocol = PPP Sending Access-Request of id 43 to 206.XXX.XX.4 port 1812 User-Name = "customer" User-Password = "password" NAS-IP-Address = 65.XX.XXX.178 NAS-Port = 5 Message-Authenticator = 0x00000000000000000000000000000000 Framed-Protocol = PPP ^C And I get this output from the RADIUS server: Ready to process requests. Ready to process requests. Ready to process requests. Each time "Ready to process requests" comes up in the console, is exactly timed to a new Access-Request from the radtest client at 65.XX.XXX.178. It's not much output, but it appears to demonstrate that FreeRADIUS is accepting the connection over the network and then... doing nothing. If it does do something, it doesn't produce any output. It certainly doesn't complain about a connection coming from an incorrect host, or return a message about how it's correctly authenticated the user or denied the authentication request. I don't know what to make of this, but I don't think it's a network problem. There are also other servers on this physical machine that are working just fine (like ssh and apache, for example). Also, I've correctly configured the 206.XXX.XX.205 IP address as a client, and then gotten the radtest program to successfully connect and authenticate. Installing the client on another, separate physical machine which exists on the same network switch and class C at 206.XXX.XX.0/24 also results in the same result as connections from our office at 65.XX.XXX.178. I'm not trying to blame you for this issue, I just need to get it fixed.
On Dec 28, 2015, at 7:59 PM, Ernie Dunbar <maillist@lightspeed.ca> wrote:
Okay, fair enough. I've made some changes to the FreeRADIUS configuration with respect to the listening port and IP address, and I've added a new "client" for remote testing. Here's the redacted client configuration, according to FreeRADIUS' debug output. ... And for each unsuccessful attempt, I get this output from FreeRADIUS:
Ignoring request to authentication address 206.XXX.XX.4 port 1812 from unknown client 206.XXX.XX.205 port 47980
You did't list that IP as a known client.
Ok, Fair enough. Those attempts are coming through this server's other IP address (206.XXX.XX.205) to its other IP address (206.XXX.XX.4). It's pretty clear that packets are reaching the FreeRADIUS daemon, it's just rejecting them because this other IP address isn't in the clients configuration. No problem, I switch to another Linux box that *does* have its IP address configured in the clients:
Yes.
And I get this output from the RADIUS server:
Ready to process requests. Ready to process requests. Ready to process requests.
Each time "Ready to process requests" comes up in the console, is exactly timed to a new Access-Request from the radtest client at 65.XX.XXX.178.
Which means that the server didn't receive a RADIUS packet. It got a *UDP* packet, but not a well formed RADIUS one.
It's not much output, but it appears to demonstrate that FreeRADIUS is accepting the connection over the network and then... doing nothing. If it does do something, it doesn't produce any output. It certainly doesn't complain about a connection coming from an incorrect host, or return a message about how it's correctly authenticated the user or denied the authentication request.
It's not complaining about bad packets by design. Anyone can send non-RADIUS packets via UDP. Even the "unknown client" messages are rate limited in later versions of the server.
I don't know what to make of this, but I don't think it's a network problem. There are also other servers on this physical machine that are working just fine (like ssh and apache, for example).
They're TCP. Not UDP.
Also, I've correctly configured the 206.XXX.XX.205 IP address as a client, and then gotten the radtest program to successfully connect and authenticate. Installing the client on another, separate physical machine which exists on the same network switch and class C at 206.XXX.XX.0/24 also results in the same result as connections from our office at 65.XX.XXX.178.
It's a networking problem. You've demonstrated that FreeRADIUS can send and receive UDP packets. But something is preventing the packets from reaching the server. You could try running a more recent version of the server. But I doubt it would help. Alan DeKok.
On 2015-12-28 18:07, Alan DeKok wrote:
On Dec 28, 2015, at 7:59 PM, Ernie Dunbar <maillist@lightspeed.ca> wrote:
I don't know what to make of this, but I don't think it's a network problem. There are also other servers on this physical machine that are working just fine (like ssh and apache, for example).
They're TCP. Not UDP.
Also, I've correctly configured the 206.XXX.XX.205 IP address as a client, and then gotten the radtest program to successfully connect and authenticate. Installing the client on another, separate physical machine which exists on the same network switch and class C at 206.XXX.XX.0/24 also results in the same result as connections from our office at 65.XX.XXX.178.
It's a networking problem. You've demonstrated that FreeRADIUS can send and receive UDP packets. But something is preventing the packets from reaching the server.
You could try running a more recent version of the server. But I doubt it would help.
Okay, just to follow up on this with my own findings for the benefit of future readers, I've discovered that whatever differences there are between Ubuntu 14.04 LTS and Debian Wheezy, are the cause of this issue. I copied the configuration from the original Debian server to another server we have that's running Ubuntu, installed the packages for FreeRADIUS (v 2.1.12 on both servers, by the way, so it's not application-specific), and found that the Ubuntu server was responding to remote hosts, while the Debian server was not. We're also running DNS and NTP on the old Debian server, so Debian's issues with FreeRADIUS appear to be very weirdly specific to that server, and not to the UDP protocol or networking in general. I don't really know why this is, but I can tell you that moving FreeRADIUS away from Debian Wheezy is definitely a solution (or possibly going back to a previous kernel version, since it worked for about 9 years before Monday morning).
On 2015-12-29 14:29, Ernie Dunbar wrote:
On 2015-12-28 18:07, Alan DeKok wrote:
On Dec 28, 2015, at 7:59 PM, Ernie Dunbar <maillist@lightspeed.ca> wrote:
I don't know what to make of this, but I don't think it's a network problem. There are also other servers on this physical machine that are working just fine (like ssh and apache, for example).
They're TCP. Not UDP.
Also, I've correctly configured the 206.XXX.XX.205 IP address as a client, and then gotten the radtest program to successfully connect and authenticate. Installing the client on another, separate physical machine which exists on the same network switch and class C at 206.XXX.XX.0/24 also results in the same result as connections from our office at 65.XX.XXX.178.
It's a networking problem. You've demonstrated that FreeRADIUS can send and receive UDP packets. But something is preventing the packets from reaching the server.
You could try running a more recent version of the server. But I doubt it would help.
Okay, just to follow up on this with my own findings for the benefit of future readers, I've discovered that whatever differences there are between Ubuntu 14.04 LTS and Debian Wheezy, are the cause of this issue. I copied the configuration from the original Debian server to another server we have that's running Ubuntu, installed the packages for FreeRADIUS (v 2.1.12 on both servers, by the way, so it's not application-specific), and found that the Ubuntu server was responding to remote hosts, while the Debian server was not. We're also running DNS and NTP on the old Debian server, so Debian's issues with FreeRADIUS appear to be very weirdly specific to that server, and not to the UDP protocol or networking in general.
I don't really know why this is, but I can tell you that moving FreeRADIUS away from Debian Wheezy is definitely a solution (or possibly going back to a previous kernel version, since it worked for about 9 years before Monday morning). -
Further addendum: downgrading the kernel to the slightly older version of 3.2.73-2+deb7u1 has not fixed the problem.
On 2015-12-29 15:01, Ernie Dunbar wrote:
On 2015-12-29 14:29, Ernie Dunbar wrote:
On 2015-12-28 18:07, Alan DeKok wrote:
On Dec 28, 2015, at 7:59 PM, Ernie Dunbar <maillist@lightspeed.ca> wrote:
I don't know what to make of this, but I don't think it's a network problem. There are also other servers on this physical machine that are working just fine (like ssh and apache, for example).
They're TCP. Not UDP.
Also, I've correctly configured the 206.XXX.XX.205 IP address as a client, and then gotten the radtest program to successfully connect and authenticate. Installing the client on another, separate physical machine which exists on the same network switch and class C at 206.XXX.XX.0/24 also results in the same result as connections from our office at 65.XX.XXX.178.
It's a networking problem. You've demonstrated that FreeRADIUS can send and receive UDP packets. But something is preventing the packets from reaching the server.
You could try running a more recent version of the server. But I doubt it would help.
Okay, just to follow up on this with my own findings for the benefit of future readers, I've discovered that whatever differences there are between Ubuntu 14.04 LTS and Debian Wheezy, are the cause of this issue. I copied the configuration from the original Debian server to another server we have that's running Ubuntu, installed the packages for FreeRADIUS (v 2.1.12 on both servers, by the way, so it's not application-specific), and found that the Ubuntu server was responding to remote hosts, while the Debian server was not. We're also running DNS and NTP on the old Debian server, so Debian's issues with FreeRADIUS appear to be very weirdly specific to that server, and not to the UDP protocol or networking in general.
I don't really know why this is, but I can tell you that moving FreeRADIUS away from Debian Wheezy is definitely a solution (or possibly going back to a previous kernel version, since it worked for about 9 years before Monday morning). -
Further addendum: downgrading the kernel to the slightly older version of 3.2.73-2+deb7u1 has not fixed the problem.
Oops, that's not true at all, I was just testing the server wrong. And the kernel version that works is 3.2.68-1+deb7u3 - the newer version 3.2.73-2+deb7u1 is broken.
On Tue, Dec 29, 2015 at 03:08:34PM -0800, Ernie Dunbar wrote:
Oops, that's not true at all, I was just testing the server wrong. And the kernel version that works is 3.2.68-1+deb7u3 - the newer version 3.2.73-2+deb7u1 is broken.
radius0:/# cat /etc/debian_version 7.9 radius0:/# uname -a Linux radius0 3.2.0-4-amd64 #1 SMP Debian 3.2.73-2+deb7u1 x86_64 GNU/Linux radius0:/# Working fine here. Possibly driver issue with network hardware on your machine? Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
participants (3)
-
Alan DeKok -
Ernie Dunbar -
Matthew Newton