Multiple Interfaces, Not Seeing Requests
Hello FreeRADIUS List, Thank you for taking the time to help me troubleshoot an issue: the radiusd service on a server with two network interfaces is not receiving RADIUS requests from network. When requests are made locally using radtest and eapol_test (with the EAP-MSCHAPv2 file,) the service responds just fine. I've done both packet-capture on the local router and a tcp dump showing packets arriving on the server. My guess is that the service isn't listening to the correct interface, but it is configured to listen to all interfaces (from what I can tell). Environment: * radiusd: FreeRADIUS Version 3.0.4, for host x86_64-redhat-linux-gnu * CentOS Linux release 7.3.1611 (Core) * Network adapters: - ens32: 172.16.xxx.253 - used for management of server - ens33: 10.xxx.xxx.253 - used for RADIUS requests radiusd -X Startup: # radiusd -X radiusd: FreeRADIUS Version 3.0.4, for host x86_64-redhat-linux-gnu, built on Jan 17 2017 at 18:49:55 ... Listening on auth address * port 1812 as server default Listening on acct address * port 1813 as server default Listening on auth address :: port 1812 as server default Listening on acct address :: port 1813 as server default Listening on auth address 127.0.0.1 port 18120 as server inner-tunnel Opening new proxy socket 'proxy address * port 0' Listening on proxy address * port 35952 Ready to process requests Proof packet is arriving: 23:20:50.935823 IP 10.xxx.xxx.205.1645 > 10.xxx.xxx.253.1812: RADIUS, Access Request (1), id: 0x55 length: 49 In clients.conf: client NSO-ACCESS-SWI { ipaddr = 10.xxx.xxx.205 secret = testing123 } I'm happy to do the legwork for this one. Can someone point me in the right direction for further troubleshooting? Much Thanks, Matthew West
Hi,
When requests are made locally using radtest and eapol_test (with the EAP-MSCHAPv2 file,) the service responds just fine. I've done both
right. so service is working.
packet-capture on the local router and a tcp dump showing packets arriving on the server. My guess is that the service isn't listening to the correct interface, but it is configured to listen to all interfaces (from what I can tell).
netstat -anp look to verify the server is listening. you have a firewall running - firewalld service running? use the tools to verify what the rules on the firewall are
Listening on auth address * port 1812 as server default Listening on acct address * port 1813 as server default Listening on auth address :: port 1812 as server default Listening on acct address :: port 1813 as server default Listening on auth address 127.0.0.1 port 18120 as server inner-tunnel Opening new proxy socket 'proxy address * port 0' Listening on proxy address * port 35952 Ready to process requests
..and no further debug output? even if you hadnt configured a client you'd get output - unknown client! ;-)
Proof packet is arriving: 23:20:50.935823 IP 10.xxx.xxx.205.1645 > 10.xxx.xxx.253.1812: RADIUS, Access Request (1), id: 0x55 length: 49
arriving to the doorstep of the server...theres still plenty that can stop that packet getting to a daemon. alan
On 6 Feb 2017, at 22:38, Matthew West <matthew.t.west@gmail.com> wrote:
I'm happy to do the legwork for this one. Can someone point me in the right direction for further troubleshooting?
RHEL/CentOS does not work well with multiple interfaces out of the box for some network configurations. You need to enable a few kernel settings to make it do the right thing. It sounds like you may have one of the affected configurations. The issue is detailed here: https://access.redhat.com/solutions/53031 You can confirm this by enabling martian logging using sysctl: net.ipv4.conf.*.log_martians=1 I wouldn't enable martial logging permanently; it's not necessary in normal operation and could cause your logging process (rysslog/journald) to start discarding useful traffic. Note that in my experience, setting default/all is not sufficient; you need to apply it to each individual interface explicitly. This may have changed in 7.3, or may not be the case if you're using an interface naming scheme that doesn't start with 'eno'. If you're using firewalld, you also need to make sure that auxiliary interfaces are assigned to the correct zone. Regards, Adam Bishop gpg: E75B 1F92 6407 DFDF 9F1C BF10 C993 2504 6609 D460 jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. Jisc Services Limited is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under company number 2881024, VAT number GB 197 0632 86. The registered office is: One Castle Park, Tower Hill, Bristol BS2 0JA. T 0203 697 5800.
Hi FR List,
look to verify the server is listening.
Appears to be listening to all: udp 0 0 0.0.0.0:1812 0.0.0.0:* 12779/radiusd udp 0 0 0.0.0.0:1813 0.0.0.0:* 12779/radiusd Firewall off: # systemctl status firewalld.service ● firewalld.service - firewalld - dynamic firewall daemon Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled) Active: inactive (dead) Docs: man:firewalld(1)
RHEL/CentOS does not work well with multiple interfaces out of the box for some network configurations. You need to enable a > few kernel settings to make it do the right thing.
It sounds like you may have one of the affected configurations.
The issue is detailed here: https://access.redhat.com/solutions/53031
OK, that makes sense in this circumstance. I checked the setting and it appears that the OS is running in strict mode for reverse path filtering. net.ipv4.conf.default.rp_filter = 1 Since the requests are coming in the same interface that they would be going out (same interface/address) why is this required? I'm going to do some troubleshooting and will let you know the results. Thank You, Matthew On Mon, Feb 6, 2017 at 3:16 PM, Adam Bishop <Adam.Bishop@jisc.ac.uk> wrote:
On 6 Feb 2017, at 22:38, Matthew West <matthew.t.west@gmail.com> wrote:
I'm happy to do the legwork for this one. Can someone point me in the right direction for further troubleshooting?
RHEL/CentOS does not work well with multiple interfaces out of the box for some network configurations. You need to enable a few kernel settings to make it do the right thing.
It sounds like you may have one of the affected configurations.
The issue is detailed here: https://access.redhat.com/solutions/53031
You can confirm this by enabling martian logging using sysctl: net.ipv4.conf.*.log_martians=1
I wouldn't enable martial logging permanently; it's not necessary in normal operation and could cause your logging process (rysslog/journald) to start discarding useful traffic.
Note that in my experience, setting default/all is not sufficient; you need to apply it to each individual interface explicitly. This may have changed in 7.3, or may not be the case if you're using an interface naming scheme that doesn't start with 'eno'.
If you're using firewalld, you also need to make sure that auxiliary interfaces are assigned to the correct zone.
Regards,
Adam Bishop
gpg: E75B 1F92 6407 DFDF 9F1C BF10 C993 2504 6609 D460
jisc.ac.uk
Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800.
Jisc Services Limited is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under company number 2881024, VAT number GB 197 0632 86. The registered office is: One Castle Park, Tower Hill, Bristol BS2 0JA. T 0203 697 5800.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Hi List, It appears that the issue was indeed net.ipv4.conf.default.rp_filter. I'm still not sure why this would block an incoming and outgoing packet if the incoming and outgoing interface/ip address was the same, but I'm glad the issue was confirmed. Thanks for your insight, Adam. It would have taken me a long time to find that by digging through logs. Take Care, Matthew On Mon, Feb 6, 2017 at 4:08 PM, Matthew West <matthew.t.west@gmail.com> wrote:
Hi FR List,
look to verify the server is listening.
Appears to be listening to all: udp 0 0 0.0.0.0:1812 0.0.0.0:* 12779/radiusd udp 0 0 0.0.0.0:1813 0.0.0.0:* 12779/radiusd
Firewall off: # systemctl status firewalld.service ● firewalld.service - firewalld - dynamic firewall daemon Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled) Active: inactive (dead) Docs: man:firewalld(1)
RHEL/CentOS does not work well with multiple interfaces out of the box for some network configurations. You need to enable a > few kernel settings to make it do the right thing.
It sounds like you may have one of the affected configurations.
The issue is detailed here: https://access.redhat.com/solutions/53031
OK, that makes sense in this circumstance. I checked the setting and it appears that the OS is running in strict mode for reverse path filtering.
net.ipv4.conf.default.rp_filter = 1
Since the requests are coming in the same interface that they would be going out (same interface/address) why is this required? I'm going to do some troubleshooting and will let you know the results.
Thank You,
Matthew
On Mon, Feb 6, 2017 at 3:16 PM, Adam Bishop <Adam.Bishop@jisc.ac.uk> wrote:
On 6 Feb 2017, at 22:38, Matthew West <matthew.t.west@gmail.com> wrote:
I'm happy to do the legwork for this one. Can someone point me in the right direction for further troubleshooting?
RHEL/CentOS does not work well with multiple interfaces out of the box for some network configurations. You need to enable a few kernel settings to make it do the right thing.
It sounds like you may have one of the affected configurations.
The issue is detailed here: https://access.redhat.com/solutions/53031
You can confirm this by enabling martian logging using sysctl: net.ipv4.conf.*.log_martians=1
I wouldn't enable martial logging permanently; it's not necessary in normal operation and could cause your logging process (rysslog/journald) to start discarding useful traffic.
Note that in my experience, setting default/all is not sufficient; you need to apply it to each individual interface explicitly. This may have changed in 7.3, or may not be the case if you're using an interface naming scheme that doesn't start with 'eno'.
If you're using firewalld, you also need to make sure that auxiliary interfaces are assigned to the correct zone.
Regards,
Adam Bishop
gpg: E75B 1F92 6407 DFDF 9F1C BF10 C993 2504 6609 D460
jisc.ac.uk
Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800.
Jisc Services Limited is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under company number 2881024, VAT number GB 197 0632 86. The registered office is: One Castle Park, Tower Hill, Bristol BS2 0JA. T 0203 697 5800.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On 7 Feb 2017, at 00:08, Matthew West <matthew.t.west@gmail.com> wrote:
Since the requests are coming in the same interface that they would be going out (same interface/address) why is this required? I'm going to do some troubleshooting and will let you know the results.
I can't be certain it is the cause as you redacted your addresses and didn't mention your gateway/prefix lengths, but the root cause would be that strict mode requires that a packet arrives at the "best" source interface. e.g: eth0 as 10.0.0.2/24 eth1 as 172.16.0.2/24 default gw as 10.0.0.1/24 A packet from 0.0.0.0/0 to eth0 will pass, as it's not from a directly connected subnet and is via the default routed interface A packet from 172.16.0.0/12 to eth0 will pass, as it's not from a directly connected subnet and is via the default routed interface A packet from 172.16.0.0/24 to eth1 will pass, as it's from the directly connected subnet for eth0 A packet from 10.0.0.0/24 to eth1 will be rejected, as it's from the directly connected subnet for eth0 A packet from 172.16.0.0/12 to eth1 will be rejected, as it's not from a directly connected subnet but is not via the default routed interface When you switch to loose filtering the "best" requirement is dropped; all that matters is that the packets source is reachable via some interface. Regards, Adam Bishop gpg: E75B 1F92 6407 DFDF 9F1C BF10 C993 2504 6609 D460 jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. Jisc Services Limited is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under company number 2881024, VAT number GB 197 0632 86. The registered office is: One Castle Park, Tower Hill, Bristol BS2 0JA. T 0203 697 5800.
I can't be certain it is the cause as you redacted your addresses and didn't mention your gateway/prefix lengths, but the root cause would be that strict mode requires that a packet arrives at the "best" source interface.
I'll investigate this further. I apologize for having to redact, but none of the RADIUS traffic is from a directly connected network - all is routed. Both the 172. and 10. addresses have their own gateways in the same /24 or /26 networks. Once I get back to work, I have some more troubleshooting and will report back. Thanks again for your help. Matthew On Mon, Feb 6, 2017 at 4:31 PM, Adam Bishop <Adam.Bishop@jisc.ac.uk> wrote:
On 7 Feb 2017, at 00:08, Matthew West <matthew.t.west@gmail.com> wrote:
Since the requests are coming in the same interface that they would be going out (same interface/address) why is this required? I'm going to do some troubleshooting and will let you know the results.
I can't be certain it is the cause as you redacted your addresses and didn't mention your gateway/prefix lengths, but the root cause would be that strict mode requires that a packet arrives at the "best" source interface.
e.g:
eth0 as 10.0.0.2/24 eth1 as 172.16.0.2/24 default gw as 10.0.0.1/24
A packet from 0.0.0.0/0 to eth0 will pass, as it's not from a directly connected subnet and is via the default routed interface A packet from 172.16.0.0/12 to eth0 will pass, as it's not from a directly connected subnet and is via the default routed interface A packet from 172.16.0.0/24 to eth1 will pass, as it's from the directly connected subnet for eth0 A packet from 10.0.0.0/24 to eth1 will be rejected, as it's from the directly connected subnet for eth0 A packet from 172.16.0.0/12 to eth1 will be rejected, as it's not from a directly connected subnet but is not via the default routed interface
When you switch to loose filtering the "best" requirement is dropped; all that matters is that the packets source is reachable via some interface.
Regards,
Adam Bishop
gpg: E75B 1F92 6407 DFDF 9F1C BF10 C993 2504 6609 D460
jisc.ac.uk
Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800.
Jisc Services Limited is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under company number 2881024, VAT number GB 197 0632 86. The registered office is: One Castle Park, Tower Hill, Bristol BS2 0JA. T 0203 697 5800.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Hi,
I'll investigate this further. I apologize for having to redact, but none of the RADIUS traffic is from a directly connected network - all is routed. Both the 172. and 10. addresses have their own gateways in the same /24 or /26 networks.
depends on what your defaut route is - and where the OS thinks it can send its packet. do you NEED the FR to be listening on all ports? I keep mine seperated to only be active on the interface on which RADIUS requests are arriving. alan
do you NEED the FR to be listening on all ports? I keep mine seperated to only be active on the interface on which RADIUS requests are arriving.
Indeed I don't. I will be making modifications to limit where it's listening. Do you recommend IP address or interface? Thank You, Matthew On Tue, Feb 7, 2017 at 7:40 AM, <A.L.M.Buxey@lboro.ac.uk> wrote:
Hi,
I'll investigate this further. I apologize for having to redact, but none of the RADIUS traffic is from a directly connected network - all is routed. Both the 172. and 10. addresses have their own gateways in the same /24 or /26 networks.
depends on what your defaut route is - and where the OS thinks it can send its packet.
do you NEED the FR to be listening on all ports? I keep mine seperated to only be active on the interface on which RADIUS requests are arriving.
alan - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Hi List, Thank you all for pointing me in the right direction. We had asymmetric routing going on.
I can't be certain it is the cause as you redacted your addresses and didn't mention your gateway/prefix lengths, but the root cause would be that strict mode requires that a packet arrives at the "best" source interface.
The sysadmin that built the system and I worked through the issue. There was, in fact, a default route causing the packets to be returned out of the management interface rather than the source interface. He implemented source-based routing and I returned the reverse packet filter back to 'strict' and FreeRADIUS still works. On to returning to configuration. Thank you guys! You're the best, Matthew On Mon, Feb 6, 2017 at 4:31 PM, Adam Bishop <Adam.Bishop@jisc.ac.uk> wrote:
On 7 Feb 2017, at 00:08, Matthew West <matthew.t.west@gmail.com> wrote:
Since the requests are coming in the same interface that they would be going out (same interface/address) why is this required? I'm going to do some troubleshooting and will let you know the results.
I can't be certain it is the cause as you redacted your addresses and didn't mention your gateway/prefix lengths, but the root cause would be that strict mode requires that a packet arrives at the "best" source interface.
e.g:
eth0 as 10.0.0.2/24 eth1 as 172.16.0.2/24 default gw as 10.0.0.1/24
A packet from 0.0.0.0/0 to eth0 will pass, as it's not from a directly connected subnet and is via the default routed interface A packet from 172.16.0.0/12 to eth0 will pass, as it's not from a directly connected subnet and is via the default routed interface A packet from 172.16.0.0/24 to eth1 will pass, as it's from the directly connected subnet for eth0 A packet from 10.0.0.0/24 to eth1 will be rejected, as it's from the directly connected subnet for eth0 A packet from 172.16.0.0/12 to eth1 will be rejected, as it's not from a directly connected subnet but is not via the default routed interface
When you switch to loose filtering the "best" requirement is dropped; all that matters is that the packets source is reachable via some interface.
Regards,
Adam Bishop
gpg: E75B 1F92 6407 DFDF 9F1C BF10 C993 2504 6609 D460
jisc.ac.uk
Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800.
Jisc Services Limited is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under company number 2881024, VAT number GB 197 0632 86. The registered office is: One Castle Park, Tower Hill, Bristol BS2 0JA. T 0203 697 5800.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
participants (3)
-
A.L.M.Buxey＠lboro.ac.uk -
Adam Bishop -
Matthew West