FreeRADIUS and EDUROAM timeout issues
How have other EDUROAM sites configured their EDUROAM servers with regard to timeout issues? The default setting seems to be less than optimal since if a remote site have problems with their home RADIUS servers then we risk having our local servers mark the upstream servers as "dead" since it's not receiving answers for a specific 'realm'... I've been using the default values so far: response_window = 20 zombie_period = 40 revive_interval = 120 status_check = status-server check_interval = 30 num_answers_to_alive = 3 But I wonder if these can be tuned a bit to better work in the EDUROAM environment. Perhaps increase the 'response_window', and lower 'zombie_period' and 'revive_interval' and 'check_interval' values... Best would probably be if FreeRadius kept a separate timeout for each 'server/realm' tuple... What have other sites done? - Peter
Peter Eriksson wrote:
The default setting seems to be less than optimal since if a remote site have problems with their home RADIUS servers then we risk having our local servers mark the upstream servers as "dead" since it's not receiving answers for a specific 'realm'...
That's been a bit of a problem in RADIUS proxying. The specification says that serves MUST answer Access-Requests. But some implementations don't do that when they're proxying. This causes all sorts of problems.
Perhaps increase the 'response_window', and lower 'zombie_period' and 'revive_interval' and 'check_interval' values...
If you're using "status-server", then "revive_interval" isn't used.
Best would probably be if FreeRadius kept a separate timeout for each 'server/realm' tuple...
Ugh. That's adding complexity to work around bugs in other RADIUS servers, IMHO. Rather than keeping track of N realms && M home servers, it now has to keep track of (N x M) combinations. That's expensive. Still, if someone sends a patch, I'll look at it. Alan DeKok.
Alan DeKok wrote:
Peter Eriksson wrote:
The default setting seems to be less than optimal since if a remote site have problems with their home RADIUS servers then we risk having our local servers mark the upstream servers as "dead" since it's not receiving answers for a specific 'realm'...
That's been a bit of a problem in RADIUS proxying. The specification says that serves MUST answer Access-Requests. But some implementations don't do that when they're proxying. This causes all sorts of problems.
Perhaps increase the 'response_window', and lower 'zombie_period' and 'revive_interval' and 'check_interval' values...
If you're using "status-server", then "revive_interval" isn't used.
Hmm.. When I have been testing stuff here it feels like it was that (review_interval) timeout that was being used before the server first sent a 'status-server' check after having marked it 'down'. But I might have been mistaken. Gonna do some more tests... I wonder how low I can set things to lessen this issue. Perhaps set zombie_period and check_interval to one second...
Best would probably be if FreeRadius kept a separate timeout for each 'server/realm' tuple...
Ugh. That's adding complexity to work around bugs in other RADIUS servers, IMHO. Rather than keeping track of N realms && M home servers,
Well, it doesn't necessarily have to be bugs in RADIUS server. It can be a multitude of stuff that causes a far away home server to not respond. Like a network outage. It doesn't feel right to have a system where a network outage in (for example) Australia can take out all the EDUROAM service for people at our university, just because we happen to have a guest from that Australian university that made an attempt to connect to the EDUROAM system...
it now has to keep track of (N x M) combinations. That's expensive.
Yes... But that is what I think the EDUROAM people that use 'Radiator' does use. - Peter
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Peter Eriksson wrote:
Alan DeKok wrote:
Peter Eriksson wrote:
The default setting seems to be less than optimal since if a remote site have problems with their home RADIUS servers then we risk having our local servers mark the upstream servers as "dead" since it's not receiving answers for a specific 'realm'... That's been a bit of a problem in RADIUS proxying. The specification says that serves MUST answer Access-Requests. But some implementations don't do that when they're proxying. This causes all sorts of problems.
Perhaps increase the 'response_window', and lower 'zombie_period' and 'revive_interval' and 'check_interval' values... If you're using "status-server", then "revive_interval" isn't used.
Hmm.. When I have been testing stuff here it feels like it was that (review_interval) timeout that was being used before the server first sent a 'status-server' check after having marked it 'down'. But I might have been mistaken. Gonna do some more tests...
I wonder how low I can set things to lessen this issue. Perhaps set zombie_period and check_interval to one second...
Best would probably be if FreeRadius kept a separate timeout for each 'server/realm' tuple... Ugh. That's adding complexity to work around bugs in other RADIUS servers, IMHO. Rather than keeping track of N realms && M home servers,
Well, it doesn't necessarily have to be bugs in RADIUS server. It can be a multitude of stuff that causes a far away home server to not respond. Like a network outage. It doesn't feel right to have a system where a network outage in (for example) Australia can take out all the EDUROAM service for people at our university, just because we happen to have a guest from that Australian university that made an attempt to connect to the EDUROAM system...
it now has to keep track of (N x M) combinations. That's expensive.
Yes... But that is what I think the EDUROAM people that use 'Radiator' does use.
Really in an system of chained proxy servers like EDUROAM you only want to be testing first hop connectivity. One way to do this (as there is no guarantee your NRPS will accept Status-Server packets) is to use the 'request' status_check and specify local credentials, so that the requests get looped back round your NRPS to your servers. Alan, do you think it might be a good idea to provide an option to disregard failures from standard authentication requests, and instead use periodic status_checks to mark servers alive or dead? This would make the EDUROAM problem go away... Thanks, Arran - -- Arran Cudbard-Bell (A.Cudbard-Bell@sussex.ac.uk), Authentication, Authorisation and Accounting Officer, Infrastructure Services (IT Services), E1-1-08, Engineering 1, University Of Sussex, Brighton, BN1 9QT DDI+FAX: +44 1273 873900 | INT: 3900 GPG: 86FF A285 1AA1 EE40 D228 7C2E 71A9 25BB 1E68 54A2 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkjtyKgACgkQcaklux5oVKKysgCfe1wnU+vJeoKes/4ovNXS/vnQ OxQAnRa0EcMItBQ192ZsaLYrtYgNX8PX =JQ0q -----END PGP SIGNATURE-----
Arran Cudbard-Bell wrote:
Really in an system of chained proxy servers like EDUROAM you only want to be testing first hop connectivity.
Exactly.
Alan, do you think it might be a good idea to provide an option to disregard failures from standard authentication requests, and instead use periodic status_checks to mark servers alive or dead?
How about having it send Status-Server packets (or whatever you configure) at the START of the zombie period. i.e. as soon as it determines that the server hasn't responded to a request, start pinging it with Status-Server packets. If it responds to the Status-Server, it will be marked "live", even if it doesn't respond to Access-Request packets. That will help, but the only solution to working with broken servers is to implement the N x M realm/server management. Alan DeKok.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Alan DeKok wrote:
Arran Cudbard-Bell wrote:
Really in an system of chained proxy servers like EDUROAM you only want to be testing first hop connectivity.
Exactly.
Alan, do you think it might be a good idea to provide an option to disregard failures from standard authentication requests, and instead use periodic status_checks to mark servers alive or dead?
How about having it send Status-Server packets (or whatever you configure) at the START of the zombie period. i.e. as soon as it determines that the server hasn't responded to a request, start pinging it with Status-Server packets.
That'd work. So when a server is marked as a Zombie Access-Requests still sent to it until the Zombie period has expired? If so do responses to Access-Requests sent during the Zombie Period force the server live again? But of course you can't guarantee successful authentication within the Zombie Period... So you send the Status-Server packets before you Mark the server as dead, if the server responds then the first hop is good, and it's the ORPS that's dead. If it doesn't, then the first hop is bad and we fail over to another server.
If it responds to the Status-Server, it will be marked "live", even if it doesn't respond to Access-Request packets.
That will help, but the only solution to working with broken servers is to implement the N x M realm/server management.
Thanks, Arran - -- Arran Cudbard-Bell (A.Cudbard-Bell@sussex.ac.uk), Authentication, Authorisation and Accounting Officer, Infrastructure Services (IT Services), E1-1-08, Engineering 1, University Of Sussex, Brighton, BN1 9QT DDI+FAX: +44 1273 873900 | INT: 3900 GPG: 86FF A285 1AA1 EE40 D228 7C2E 71A9 25BB 1E68 54A2 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkjuH7oACgkQcaklux5oVKJb8wCfb3ZEDi5ZVuCmHzA4HR05jHF9 WacAniG+Vpf7rGZBHE2m94RzQuR5oTsF =8Om7 -----END PGP SIGNATURE-----
Arran Cudbard-Bell wrote:
That'd work. So when a server is marked as a Zombie Access-Requests still sent to it until the Zombie period has expired?
Yes. I also noticed that the current code doesn't send Status-Server packets until "check_interval" time AFTER it's marked "dead". So we have "response_window" delay, followed by "zombie_period", followed by "check_interval". In some cases, it might not start pinging the home server until a minute after it stops responding. Not nice. My current proposal is to start pinging it at the start of "zombie_period". If you then set: zombie_period = 21 check_interval = 6 num_pings_to_alive = 3 It will start pinging the home server as soon as it stops responding. if it responds to all 3 pings, it will be marked "live" again, without ever being marked "dead".
If so do responses to Access-Requests sent during the Zombie Period force the server live again?
Yes.
But of course you can't guarantee successful authentication within the Zombie Period... So you send the Status-Server packets before you Mark the server as dead, if the server responds then the first hop is good, and it's the ORPS that's dead. If it doesn't, then the first hop is bad and we fail over to another server.
Yes. This still means that requests will be sent to that home server,even if they're for an upstream realm that's dead. If there are multiple paths to the upstream realm, then those other paths won't be discovered. But there is no RADIUS "routing protocol"[1]. So that's that. Alan DeKok. [1] For now.
Hi,
This still means that requests will be sent to that home server,even if they're for an upstream realm that's dead. If there are multiple paths to the upstream realm, then those other paths won't be discovered.
But there is no RADIUS "routing protocol"[1]. So that's that.
s'funny that you should mention that - what with a hierarchical system. I thought it would be neat if a downstream system could notify the upstream about what realms it could deal with and - via a trusted and encrypted channel - thus no more manually adding new records to higher tiers for new realms dealt with.. much like advertising routing space via BGP or OSPF still. that'd be a new RFC for sure ! :-) alan
A.L.M.Buxey@lboro.ac.uk wrote:
But there is no RADIUS "routing protocol"[1]. So that's that.
s'funny that you should mention that - what with a hierarchical system. I thought it would be neat if a downstream system could notify the upstream about what realms it could deal with and - via a trusted and encrypted channel - thus no more manually adding new records to higher tiers for new realms dealt with.. much like advertising routing space via BGP or OSPF
This will happen. There is sufficient buy-in from large telcos that it's necessary. FreeRADIUS will likely be the first compliant implementation... before the specification is written. Alan DeKok.
Peter Eriksson wrote:
I wonder how low I can set things to lessen this issue. Perhaps set zombie_period and check_interval to one second...
That's not a good idea. It means that the server will be marked dead MORE quickly.
Best would probably be if FreeRadius kept a separate timeout for each 'server/realm' tuple... Ugh. That's adding complexity to work around bugs in other RADIUS servers, IMHO. Rather than keeping track of N realms && M home servers,
Well, it doesn't necessarily have to be bugs in RADIUS server. It can be a multitude of stuff that causes a far away home server to not respond.
That isn't the problem. The problem is that the NEXT hop isn't responding, The RFC's say it MUST respond. Some implementations don't respond if they're also proxying the request, and the home server doesn't respond to them.
Like a network outage. It doesn't feel right to have a system where a network outage in (for example) Australia can take out all the EDUROAM service for people at our university, just because we happen to have a guest from that Australian university that made an attempt to connect to the EDUROAM system...
The eduroam servers SHOULD respond to the local university if Australia is down: "Authentication failed. Couldn't reach Australia!" Instead, some implementations don't respond. So your local university can't tell the difference between Australia being down, and the Eduroam servers being down... because the eduroam server is pretending it's down, too.
it now has to keep track of (N x M) combinations. That's expensive.
Yes... But that is what I think the EDUROAM people that use 'Radiator' does use.
Radiator is part of the problem. The only reason they implement that N x M combination is because *their* implementation behaves poorly. i.e. A radiator proxy doesn't respond to a local server if the upstream home server doesn't respond to radiator. Their solution? Add complexity. This is supposed to make things more "stable". Ugh. I think the preferred approach is what I said in my other message. Have FreeRADIUS start pinging the proxy as soon as the proxy MIGHT be down. That way, network outages are minimized. Alan DeKok.
participants (4)
-
A.L.M.Buxey＠lboro.ac.uk -
Alan DeKok -
Arran Cudbard-Bell -
Peter Eriksson