Question on client retransmit behavior
So an EAP 802.1x supplicant can do one of the following if it receives no response from the NAS: 1) Restart EAP from the top. 2) Retransmit packets from the ongoing EAP session. 3) Just wait and maybe the NAS retransmits for it. 4) (Hopefully not) have a really old proxy in the chain that does retransmits for the NAS/client. If anyone has had the time to go down this rabbithole, is there any uniformity of client behavior here? That's really all I want to know... just trying to figure out if turning off NAS-initiiated retransmits is a wise move. But, if you'd like to hear a scary story, pull up a chair, I am glad to oblige: Lately we've had quite a few eduroam guest clients show up sending to lame servers that do not respond. Also, lately, supplicants have seem to have been getting quite aggressive at re-launching frequent authentication attempts after failures. But what good zombie doesn't enjoy bumping repeatedly against the door? Our NAS units serve these clients and our home users on the same SSID (we do fortunately have an additional non-"eduroam" one as well, where one can seek refuge, but many of our users are... adventurous) The NAS units have no idea that these two groups of users are different. So... in the wee hours of the night... ...when not many people are around to provide good authentications, ...when a lame eduroam guest client manages to authenticate three times (cue withered hand lifting a cast iron door knocker, spooky music, three loud knocks) ...the NAS declares our proxy dead. It then rolls over to another proxy, and declares it dead as well, if it happens again. Despite all that is evident to the naked eye, this NAS does not believe in zombies. So... it won't retry any of the dead servers until a generous timeout goes by. Oh yeah, and no Status-Server support. (cue suspensful music) the serial proxy killer is on the loose. Once all proxies are dead, no client can authenticate for quite a few minutes (cue tumbleweed) Our front-end proxy is a FreeRADIUS server. This then proxies to... a haunted old house owned by a mysterious vendor. Which... retransmits proxied requests. Yes. That old. There appears (sigh) to be no way to turn this behavior off. It also has a maximum 5 second timeout between retransmits, and though it will respond to Status-Server requests, it will not send them to upstream proxies. You can imagine the mess that makes (cue pixelized gore) On the other side... well we're special we have backend servers... but this could happen to anyone, since all eduroam requests go to the same TLRS system. You may be able to protect your own users, but you are not able to keep your guests safe from one another. If just one zombie gets into the guest house, all your friends may find themselves... well... So we have been trying to kill the zombies on the back-end... but 5 seconds, minus 1 for the reject-delay, minus 1 for the DMZ proxy reject delay... 3 seconds is perhaps a bit tight for a 7-way handshake zig-zagging across the globe as it trickles through the federation servers. Now we're resorting to slaughtering zombies on the front end before they manage to get to the haunted house on clearpass lane. Ooops. Did I just name names? So sorry. Forget I said that. This of course messes up our logs, since only the NAS and the front end server see the rejection. Not that the logs were in great condition to start out, since it appears plenty of clients are now configured to just drop their EAP conversation if they are rejected in the inner tunnel, before issuing the outer tunnel challenge to get an outer tunnel Access-Reject. And finally, the haunted house... it logs all such occurences simply as just "timeout"s, unless you drill down to see whether the timeout was client or server side. But what's a little salt in the headwound, after all. Anyway, thank you for patronizing our hanted-house ride. Remember to collect your belongings as you leave the carts and... try to get a good night's sleep.
On Apr 25, 2024, at 5:41 PM, Brian Julin <BJulin@clarku.edu> wrote:
So an EAP 802.1x supplicant can do one of the following if it receives no response from the NAS:
1) Restart EAP from the top. 2) Retransmit packets from the ongoing EAP session. 3) Just wait and maybe the NAS retransmits for it. 4) (Hopefully not) have a really old proxy in the chain that does retransmits for the NAS/client.
I've also seen: (5) retransmit aggressively 100-1000 times a second. :(
If anyone has had the time to go down this rabbithole, is there any uniformity of client behavior here?
IEEE 802.1X defines how all of this should work. I admit I haven't looked at that in detail in a while.
That's really all I want to know... just trying to figure out if turning off NAS-initiiated retransmits is a wise move.
IEEE STD 802.1X-2020 says in part: • The higher layer is responsible for re-transmission within a single authentication attempt, and should protect communication with the Authentication Server with retransmissions appropriate to the transport use. and Correct protocol operation depends upon the use of timer values by the Supplicant higher layer functions that are compatible with those used by the Authenticator’s higher layer functions to retransmit EAP-Requests. There is no automatic means of communicating changes in timer values between Authenticator and Supplicant, so deviation from the default timer values can adversely affect the operation of the protocol. Where "authenticator" is the NAS. So perhaps it's the responsibility of the NAS to do retransmissions.
But, if you'd like to hear a scary story, pull up a chair, I am glad to oblige:
Lately we've had quite a few eduroam guest clients show up sending to lame servers that do not respond.
Also, lately, supplicants have seem to have been getting quite aggressive at re-launching frequent authentication attempts after failures.
IEEE 802.1X also requires that systems send only a small number of packets per second.
...the NAS declares our proxy dead.
It then rolls over to another proxy, and declares it dead as well, if it happens again.
Despite all that is evident to the naked eye, this NAS does not believe in zombies.
<sigh> There's also issues with the RADIUS protocol. When upstream proxies fail, there's no way to return a "reroute this packet" message. Instead, the packet is dropped. And the NAS has no idea if it's server is down. It's really quite terrible. We're looking at fixing this in the updated RFCs, but it is likely to require TLS.
So... it won't retry any of the dead servers until a generous timeout goes by.
Oh yeah, and no Status-Server support.
It's only been published for a decade. Why would people implement it?
Our front-end proxy is a FreeRADIUS server. This then proxies to... a haunted old house owned by a mysterious vendor.
Which... retransmits proxied requests. Yes. That old.
There appears (sigh) to be no way to turn this behavior off. It also has a maximum 5 second timeout between retransmits, and though it will respond to Status-Server requests, it will not send them to upstream proxies.
That part is at least fine: not proxying Status-Server.
Anyway, thank you for patronizing our hanted-house ride. Remember to collect your belongings as you leave the carts and... try to get a good night's sleep.
RADIUS isn't a house of cards. It's a haunted house of brain-eating zombies. The fact that it works at all is a bit of a miracle. There's no real solution here other than faking rejected for the bad users. A longer term fix is people upgrading their NASes for modern protocol support. But that might take years, Alan DeKok.
Alan DeKok <aland@deployingradius.com> wrote:
• The higher layer is responsible for re-transmission within a single authentication attempt, and should protect communication with the Authentication Server with retransmissions appropriate to the transport use.
and
Correct protocol operation depends upon the use of timer values by the Supplicant higher layer functions that are compatible with those used by the Authenticator’s higher layer functions to retransmit EAP-Requests. There is no automatic means of communicating changes in timer values between Authenticator and Supplicant, so deviation from the default timer values can adversely affect the operation of the protocol.
Thanks. If I find the time and patience to take this up with the vendor those will be useful.
Where "authenticator" is the NAS.
So perhaps it's the responsibility of the NAS to do retransmissions.
Hrm... maybe I'll just give it a try when usage is really low just to see what happens. Something horrible, no doubt.
There appears (sigh) to be no way to turn this behavior off. It also has a maximum 5 second timeout between retransmits, and though it will respond to Status-Server requests, it will not send them to upstream proxies. That part is at least fine: not proxying Status-Server.
What I meant was... it does not allow you to initiate Status Server probes on that proxy hop to send to the next proxy. Wouldn't expect it to forward. Thanks again.
participants (2)
-
Alan DeKok -
Brian Julin