Remote users - proxying fails - NPS timeout
Hello, We have 2 CentOS 7 servers which act solely as RADIUS proxy servers using PEAP. They proxy to backend Microsoft NPS servers. The servers proxy local (wireless) users (to NPS), visiting eduroam users (to NRPS) and our own (eduroam) users (to NPS) who are visting external sites. The eduroam users are proxied to/from the UK national proxy servers (NRPS). Both local radius servers ran the freeradius 3.0.4 CentOS package. This all worked well. However, I have updated freeradius to 3.0.15 (locally built RPM) on one of the servers. Whilst the local and visiting eduroam users are authenticating correctly, the majority of our own eduroam users authenticating to us via the NRPS servers now fail. The radius log shows (for example): ========== Thu Aug 17 23:16:20 2017 : ERROR: (7299) ERROR: Failing proxied request for user "abc@students.plymouth.ac.uk", due to lack of any response from home server 141.163.xxx.yyy port 1812 Thu Aug 17 23:16:20 2017 : Auth: (7299) Login incorrect (Home Server failed to respond): [abc@students.plymouth.ac.uk] (from client NRPS-0 port 13 cli FC-FC- 48-FF-AA-DD) SSID: eduroam Thu Aug 17 23:16:20 2017 : Auth: (7299) Login incorrect (Failing proxied request for user "abc@students.plymouth.ac.uk", due to lack of any response from home server 141.163.xxx.yyy port 1812): [abc@students.plymouth.ac.uk] (from client NRPS-0 port 13 cli FC-FC-48-DD-AA-DD) SSID: eduroam ========== For the NRPS clients this occurs nearly 100% of the time. For some reason an occassional user will authenticate successfully. I have captured the debug output for a user who authenticated successfully on the 3.0.4 server, but then failed to authenticate on the 3.0.15 server. Although the last packet sent seemed acceptable, with no reply from the MS NPS server it is difficult to know what happened next. A little googling seemed to indicate that MS NPS is known for just dropping packets rather than rejecting them or even logging them. I asked the sysadmin of the NPS servers if the logs showed anything, but he said not. The servers themselves were not under any particular load, so that was not a reason to drop the packets. Although the freeradius config files have had to be modified a little bit due to differences between 3.0.4 and 3.0.15, they are basically the same. I tried downgrading to an RHEL 7 beta RPM of freeradius 3.0.13, but we got the same timeout problem. I have now gone back to 3.0.4, and the problem has gone. I am a bit stumped with this. My first thought was that we had changed something to cause the MS NPS servers to drop the packets. Filtering seemed possible as I gather if NPS is sent a packet with unrecognised radius attributes then it is likely to drop it. However, the filtering is the same between 3.0.4 and 3.0.15. I have also gone through the changelog from 3.0.5 to 3.0.15, but could not find anything obvious (to me) that might explain this. My thoughts are that perhaps some core part of freeradius has been modified such that it is now causing NPS (for us) a problem. We could start to work through the versions from 3.0.5 upwards to see at what version we hit the problem, and then look further at the commits for that version. However, before that I thought I would ask here if anyone has any thoughts about this? Has anyone had a similar problem? Thanks, John. -- John Horne | Senior Operations Analyst | Technology and Information Services University of Plymouth | Drake Circus | Plymouth | Devon | PL4 8AA | UK ________________________________ [http://www.plymouth.ac.uk/images/email_footer.gif]<http://www.plymouth.ac.uk/worldclass> This email and any files with it are confidential and intended solely for the use of the recipient to whom it is addressed. If you are not the intended recipient then copying, distribution or other use of the information contained is strictly prohibited and you should not rely on it. If you have received this email in error please let the sender know immediately and delete it from your system(s). Internet emails are not necessarily secure. While we take every care, Plymouth University accepts no responsibility for viruses and it is your responsibility to scan emails and their attachments. Plymouth University does not accept responsibility for any changes made after it was sent. Nothing in this email or its attachments constitutes an order for goods or services unless accompanied by an official order form.
On Aug 18, 2017, at 1:37 AM, John Horne <john.horne@plymouth.ac.uk> wrote:
However, I have updated freeradius to 3.0.15 (locally built RPM) on one of the servers. Whilst the local and visiting eduroam users are authenticating correctly, the majority of our own eduroam users authenticating to us via the NRPS servers now fail. The radius log shows (for example):
========== Thu Aug 17 23:16:20 2017 : ERROR: (7299) ERROR: Failing proxied request for user "abc@students.plymouth.ac.uk", due to lack of any response from home server 141.163.xxx.yyy port 1812
If NPS doesn't respond, NPS is the problem. :(
For the NRPS clients this occurs nearly 100% of the time. For some reason an occassional user will authenticate successfully. I have captured the debug output for a user who authenticated successfully on the 3.0.4 server, but then failed to authenticate on the 3.0.15 server. Although the last packet sent seemed acceptable, with no reply from the MS NPS server it is difficult to know what happened next.
Well, both 3.0.4 and 3.0.15 are compliant with all RADIUS standards. I suspect NPS isn't. I suspect that there are minor differences in packets which causes NPS to barf. I suggest looking at PCAPs (tcpdump / wireshark) to see what's different about the packets. Or, send me the PCAPs (off-list), and say which packets are the problems. I'll see if I can discover anything.
However, before that I thought I would ask here if anyone has any thoughts about this? Has anyone had a similar problem?
I've seen similar issues before with NPS. If NPS receives an attribute that it thinks is type "integer", but is instead 8 bytes long (i.e. not an integer), NPS will *drop the whole packet*. This behaviour is beyond stupid. I put text into RFC 6929 to address this exact problem: https://tools.ietf.org/html/rfc6929#section-2.8 I've also seen issues with other proprietary RADIUS servers that sometimes expect to see attributes in a particular order. One in particular had the idiotic behaviour of replying to "bad" packets by *echoing the bad packet back to the client*. I'm at a loss for how anyone ever thought that was a good idea. Alan DeKok.
participants (2)
-
Alan DeKok -
John Horne