Hi, We tried to stress test (EAPTLS) FreeRADIUS 2.1.7 which sits behind Load Balancer We had 2 FreeRADIUS servers behind load balancer (R1,R2) Some requests got rejected (Access-Reject was sent) and the log showed Wed Sep 30 11:56:31 2009 : Error: rlm_eap: No EAP session matching the State variable. Wed Sep 30 11:56:31 2009 : Error: rlm_eap: No EAP session matching the State variable. Wed Sep 30 11:56:31 2009 : Error: rlm_eap: No EAP session matching the State variable. Wed Sep 30 11:56:31 2009 : Error: rlm_eap: No EAP session matching the State variable. Now we understand that if EAPTLS session started (we have 10 Access-Challenge messages because of our certificate chain) against Radius_1 and then continued to Radius_2 because load balancer reverted it there then EAPTLS handshake cannot succeed, but we expected that FreeRADIUS should drop packets and NOT RESPOND instead of sending Access-Reject when it cannot find STATE variable By looking at the code we think eap_tls module returns RLM_MODULE_INVALID or RLM_MODULE_FAIL when it cannot find EAP session in the tree. What is proper configuration that we can do? Is something like this recommended? authorize { .. eap { ok = return updated = return handled = return } if (invalid) { do_not_respond } # if we reach here then Auth-Type != EAP .. } Thanks you for your response -- View this message in context: http://www.nabble.com/EAPTLS-Stress-test%3A-2.1.7-tp25686662p25686662.html Sent from the FreeRadius - User mailing list archive at Nabble.com.
leopold wrote:
Hi, We tried to stress test (EAPTLS) FreeRADIUS 2.1.7 which sits behind Load Balancer
That doesn't work. Don't bother trying to "fix" FreeRADIUS. Instead, use a load balancer that is aware of EAP. e.g. FreeRADIUS.
Now we understand that if EAPTLS session started (we have 10 Access-Challenge messages because of our certificate chain) against Radius_1 and then continued to Radius_2 because load balancer reverted it there then EAPTLS handshake cannot succeed, but we expected that FreeRADIUS should drop packets and NOT RESPOND instead of sending Access-Reject when it cannot find STATE variable
No. Users that fail authentication get Access-Reject. The alternative is worse.
By looking at the code we think eap_tls module returns RLM_MODULE_INVALID or RLM_MODULE_FAIL when it cannot find EAP session in the tree.
Yes. This is what it's supposed to do.
What is proper configuration that we can do?
Use a load balancer that is aware of EAP. FreeRADIUS can do this. See "keyed-balance" in proxy.conf. You can load-balance over the client IP && the User-Name. This is often good enough to get most EAP sessions working.
Is something like this recommended?
No. All it does is force the client to retransmit. At which point, there's a 50% chance that it will go to the SAME server, and *again* be thrown away. For an EAP session of 10 packets, the odds are 1/(2^10) that *all* EAP packets will go to the right server. So you'll have 1 out of 1000 authentications succeed. The rest will be rejected, after processing many, many, duplicates. Don't increase your load by a factor of 1000 for nothin. Use a load-balancer that is aware of EAP, *or* configure your load balancer to hash on source IP, and balance based on that. It means that *most* EAP sessions will go to the same RADIUS server. It's not perfect, but it's a lot better than having a 99.9% failure rate. Alan DeKok.
Hello Alan, Thanks as always for your reply. Yes regarding Load Balancer we know that we need "IP stickiness" and we do have it. In this case we disabled IP stickiness just to be able to test how FreeRADIUS reacts when it recieves messages dedicated for another FR server. This scenario could happen during failover from RAdius1 to RAdius2, when EAPTLS handshake started against R1 and in the middle of handshake (low probability, but still can occur) R1 goes down and failver is done against R2 and we expected R2 to drop the message instead of sending Access-Reject Another test case we did was stressing one freeradius server (no loadbalancers in the middle) and it could cope gracefully with load of 200 eaptls authentications/sec, but when we increased load to 300 auth/sec things when really bad 1. We could reproduce constantly this error Wed Sep 30 17:33:28 2009 : Error: rlm_eap: Failed to store handler Wed Sep 30 17:33:28 2009 : Error: rlm_eap: Failed to store handler Wed Sep 30 17:33:28 2009 : Error: rlm_eap: Failed to store handler 2. Around 50% of the clients received Access-Reject Yes I understand your point regarding radius dropping/not responding to invalid eaptls messages and that it causes client retries and even more load on radius infrastructure, but unfortunately due to own bussiness requirements we can't send Access-Reject to a user/machine that "tries" to present a valid certificate during load conditions. We view a failure for a valid client as outage. At some point when no answer is received from radius a valid client will retry and get to network, on the other hand when receiving Access-Reject client state machine goes into a state when retry timeout is too long and it will cause client machine outage. We think when client presents invalid certificate (signed by untrusted CA or expired certificate or revoked) then it should get Access-Reject which is good, but when error is cause by load or other infrastructure or network problems we feel that not responding in a better choice. Unfortunately there is no other reply code in radius protocol in addition to Access-Reject that says Access-CriticalError that indicates that sort of error condition. If we still want to proceed with Do-Not-respond path, do you think it is doable? Something like config changes below help, not sure if they break other things need to test. authorize { .. eap { ok = return updated = return handled = return invalid = 1 fail = 2 } if (invalid || fail) { do_not_respond } .. } authenticate { .. Auth-Type eap { eap { invalid = 1 fail = 2 } if (invalid || fail) { do_not_respond } } .. } Alan DeKok-2 wrote:
leopold wrote:
Hi, We tried to stress test (EAPTLS) FreeRADIUS 2.1.7 which sits behind Load Balancer
That doesn't work. Don't bother trying to "fix" FreeRADIUS. Instead, use a load balancer that is aware of EAP. e.g. FreeRADIUS.
Now we understand that if EAPTLS session started (we have 10 Access-Challenge messages because of our certificate chain) against Radius_1 and then continued to Radius_2 because load balancer reverted it there then EAPTLS handshake cannot succeed, but we expected that FreeRADIUS should drop packets and NOT RESPOND instead of sending Access-Reject when it cannot find STATE variable
No. Users that fail authentication get Access-Reject. The alternative is worse.
By looking at the code we think eap_tls module returns RLM_MODULE_INVALID or RLM_MODULE_FAIL when it cannot find EAP session in the tree.
Yes. This is what it's supposed to do.
What is proper configuration that we can do?
Use a load balancer that is aware of EAP. FreeRADIUS can do this. See "keyed-balance" in proxy.conf. You can load-balance over the client IP && the User-Name. This is often good enough to get most EAP sessions working.
Is something like this recommended?
No.
All it does is force the client to retransmit. At which point, there's a 50% chance that it will go to the SAME server, and *again* be thrown away. For an EAP session of 10 packets, the odds are 1/(2^10) that *all* EAP packets will go to the right server. So you'll have 1 out of 1000 authentications succeed. The rest will be rejected, after processing many, many, duplicates.
Don't increase your load by a factor of 1000 for nothin. Use a load-balancer that is aware of EAP, *or* configure your load balancer to hash on source IP, and balance based on that. It means that *most* EAP sessions will go to the same RADIUS server.
It's not perfect, but it's a lot better than having a 99.9% failure rate.
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
-- View this message in context: http://www.nabble.com/EAPTLS-Stress-test%3A-2.1.7-tp25686662p25696829.html Sent from the FreeRadius - User mailing list archive at Nabble.com.
leopold wrote:
Another test case we did was stressing one freeradius server (no loadbalancers in the middle) and it could cope gracefully with load of 200 eaptls authentications/sec, but when we increased load to 300 auth/sec things when really bad 1. We could reproduce constantly this error Wed Sep 30 17:33:28 2009 : Error: rlm_eap: Failed to store handler Wed Sep 30 17:33:28 2009 : Error: rlm_eap: Failed to store handler Wed Sep 30 17:33:28 2009 : Error: rlm_eap: Failed to store handler
Hmm... the only thing I can suggest is to increase the "max_sessions" parameter in eap.conf.
Yes I understand your point regarding radius dropping/not responding to invalid eaptls messages and that it causes client retries and even more load on radius infrastructure, but unfortunately due to own bussiness requirements we can't send Access-Reject to a user/machine that "tries" to present a valid certificate during load conditions. We view a failure for a valid client as outage.
Well... that's a failure unusual requirement.
At some point when no answer is received from radius a valid client will retry and get to network, on the other hand when receiving Access-Reject client state machine goes into a state when retry timeout is too long and it will cause client machine outage.
And will the NAS think that the RADIUS server is down?
We think when client presents invalid certificate (signed by untrusted CA or expired certificate or revoked) then it should get Access-Reject which is good, but when error is cause by load or other infrastructure or network problems we feel that not responding in a better choice. Unfortunately there is no other reply code in radius protocol in addition to Access-Reject that says Access-CriticalError that indicates that sort of error condition.
That is an issue with the protocol.
If we still want to proceed with Do-Not-respond path, do you think it is doable?
It may work. Alan DeKok.
participants (2)
-
Alan DeKok -
leopold