generate_state() in rlm_eap module generates duplicate state ?
Hi, I am using version freeradius ver 1.1.6 and I am observing a problem for which I could not find an existing bug report. Here is the description Problem description I am testing a gateway appliance for large number of subscribers. I am sending around 100 EAP-authentication requests (EAP-MD5) per second (or more) to the freeradius server. Before starting my request I changed the max_requests configuration from 1024 to 8192. What I observed was that every so often I would get an Access-Reject for a request which should have succeeded and does succeed on subsequent attempts. I added debugs in the free radius server and found out that in eap_authenticate the handler for this request never gets inserted in the session tree. So the radius server sends the Access-Challenge but when the gateway replies to the Access-Challenge the radius server cannot find the EAP request handler in the session tree and so the authentication fails. On looking deeper in to the problem I saw that the reason the rbtree_insert in eaplist_add() fails is because there is already a node in the tree with the same state (PW_STATE), this node is for a different request altogether and so the insert in to the tree fails. Since I send multiple requests from the same gateway and the radius client on the gateway opens up 4 source ports to the radius server we have can send 4 requests in the same second that have the same ID and source IP address. The eap_handler_cmp() function uses the STATE to distinguish requests if the id and source IP address are the same, but in my case the state is also the same. Here is the relevant debug log for it Wed Sep 19 22:36:37 2007 : Info: STATE: Added state 82 f1 8b 49 6f bf b9 a4 ee 2a d3 d5 ef f8 ec 3b for {am= 1}201myid@mynet.net <1%7D201myid@mynet.net> Sending Access-Challenge to {am= 1}201myid@mynet.net <1%7D201myid@mynet.net> ... ... rad_recv: Access-Request packet from host 1.1.1.1:57979, id=8, length=160 User-Name = "{am= 1}245myid@mynet.net <1%7D245myid@mynet.net>" ... Wed Sep 19 22:36:37 2007 : Info: STATE: Added state 82 f1 8b 49 6f bf b9 a4 ee 2a d3 d5 ef f8 ec 3b for {am= 1}245myid@mynet.net<1%7D245myid@mynet.net> Wed Sep 19 22:36:37 2007 : Error: Found duplicate data, Id = {am= 1}201myid@mynet.net <1%7D201myid@mynet.net>, State = 82 f1 8b 49 6f bf b9 a4 ee 2a d3 d5 ef f8 ec 3b Wed Sep 19 22:36:37 2007 : Error: rlm_eap: Failed to remember handler! I am guessing this can cause other problems, not sure if this can result in sending Access-Accept to users who should not be authenticated but it definitely rejects users who should be authenticated. Another problem I see which seems related to this is that I receive Access-Accept on the gateway for a valid user but the reply has an invalid message authenticator. I think this is probably because it finds a different handler instead, but not sure about that. Is this a known bug ? If yes, is it fixed and in what release ? For now, I am changing the eap_handler_cmp() function to compare the identity if the state is the same, since in our lab setup we also have a unique identity for each user, do you see a problem with doing this. Thanks, Vinay
Vinay Wagh wrote:
On looking deeper in to the problem I saw that the reason the rbtree_insert in eaplist_add() fails is because there is already a node in the tree with the same state (PW_STATE), this node is for a different request altogether and so the insert in to the tree fails. Since I send multiple requests from the same gateway and the radius client on the gateway opens up 4 source ports to the radius server we have can send 4 requests in the same second that have the same ID and source IP address.
Hmm... that may be allowed by the spec, but it seems broken to me. The code in rlm_eap does NOT use the source port to distinguish requests, because some NASes send EAP requests for one session via multiple source ports. It just depends on how busy the NAS is.
The eap_handler_cmp() function uses the STATE to distinguish requests if the id and source IP address are the same, but in my case the state is also the same. Here is the relevant debug log for it
If the state is the same, then something is going seriously wrong inside of the server. The state that is generated SHOULD be random, and unique per request.
Wed Sep 19 22:36:37 2007 : Info: STATE: Added state 82 f1 8b 49 6f bf b9 a4 ee 2a d3 d5 ef f8 ec 3b for {am= 1}201myid@mynet.net
Can you instrument the code to see when the state is generated? My suspicion is that a new State attribute is NOT being generated. i.e. the server sees the "same" request, and re-uses the same State. That shouldn't be happening, though. On a new EAP-Identity packet, a new State is allocated, and added to the list. Since the State is generated from random numbers, it SHOULD be unique every time.
I am guessing this can cause other problems, not sure if this can result in sending Access-Accept to users who should not be authenticated but it definitely rejects users who should be authenticated. Another problem I see which seems related to this is that I receive Access-Accept on the gateway for a valid user but the reply has an invalid message authenticator. I think this is probably because it finds a different handler instead, but not sure about that.
The calculation of the Message-Authenticator has nothing to do with EAP, or with the State. If the Message-Authenticator is invalid, then something is seriously wrong.
Is this a known bug ? If yes, is it fixed and in what release ? For now, I am changing the eap_handler_cmp() function to compare the identity if the state is the same, since in our lab setup we also have a unique identity for each user, do you see a problem with doing this.
No. I would suggest in your situation also adding a comparison on the source port. That should work ard the problem. I'm hesitant to add that comparison to the main code, though. It may cause other peoples networks to break. Alan DeKok.
Can i use simultaneous use to limit particular users to use just one channel ISDN? And at the same time, to forbid async users multiple logins? For ISDN users i want to limit only some users, not all. I know that cisco questions should not be posted here and i am sorry for that but... I am not sure what can i do on NAS side to prevent double channel ISDN logins. Cisco is AS 3620, IOS is 12.0(7)T. If i can send some vendor specific packet to NAS that would be great. Thank you.
inline .. On 9/25/07, Alan DeKok <aland@deployingradius.com > wrote:
Vinay Wagh wrote:
On looking deeper in to the problem I saw that the reason the rbtree_insert in eaplist_add() fails is because there is already a node in the tree with the same state (PW_STATE), this node is for a different
request altogether and so the insert in to the tree fails. Since I send multiple requests from the same gateway and the radius client on the gateway opens up 4 source ports to the radius server we have can send 4 requests in the same second that have the same ID and source IP address.
Hmm... that may be allowed by the spec, but it seems broken to me. The code in rlm_eap does NOT use the source port to distinguish requests, because some NASes send EAP requests for one session via multiple source ports. It just depends on how busy the NAS is.
We need to use more than one source port on the client because the client supports only 255 requests per instance of radius client and we need about 1000 requests per second. Each EAP session will always use the same port while it is being authenticated but a new request can pick up a different Id on a different source port.
The eap_handler_cmp() function uses the STATE to distinguish requests if
the id and source IP address are the same, but in my case the state is also the same. Here is the relevant debug log for it
If the state is the same, then something is going seriously wrong inside of the server. The state that is generated SHOULD be random, and unique per request.
Which is why it took me some time to figure this out. What I did was added debug code in rbtree_insert to print out contents of the node if a duplicate node existed. In the logs I saw that the node had the same state but a different identity.
Wed Sep 19 22:36:37 2007 : Info: STATE: Added state 82 f1 8b 49 6f bf b9
a4 ee 2a d3 d5 ef f8 ec 3b for {am= 1}201myid@mynet.net<1%7D201myid@mynet.net>
Can you instrument the code to see when the state is generated? My suspicion is that a new State attribute is NOT being generated. i.e. the server sees the "same" request, and re-uses the same State.
That shouldn't be happening, though. On a new EAP-Identity packet, a new State is allocated, and added to the list. Since the State is generated from random numbers, it SHOULD be unique every time.
I have already added debug code to print out the state every time the radius server generates it. Out of a thousand requests you see atleast 4-5 requests with duplicate state info, the others have unique states. The debug above 'Added state ..." gets printed after a call to generate_state().
I am guessing this can cause other problems, not sure if this can result
in sending Access-Accept to users who should not be authenticated but it definitely rejects users who should be authenticated. Another problem I see which seems related to this is that I receive Access-Accept on the gateway for a valid user but the reply has an invalid message authenticator. I think this is probably because it finds a different handler instead, but not sure about that.
The calculation of the Message-Authenticator has nothing to do with EAP, or with the State. If the Message-Authenticator is invalid, then something is seriously wrong.
The reason I started debugging this problem is because I started getting Access-Reject without RADIUS_ATTR_MESSAGE_AUTHENTICATOR which is a seperate attribute in the radius message. I also observed that some reply's from the radius server had this field but it did not match the authenticator in the original request. Then I tried to link this to the problem I found and I think it is possible if we generate the same state. Assume that after the server sends the access challenge the radius server fails to insert the handler because there is already a duplicate and then before it gets rid of the duplicate handler the client replies. In this case the radius server will try to look for the handler and actually find it since the id, ip addr and state is the same but the identity is different. It can then use that context to reply to this request in which case the fields may not match. If the radius server had already replied to the duplicate handler then it will not find the handler that for our current request and send an Access-Reject. So whether we get an Access-Reject or a reply with invalid message authenticator depends on the timing of whether the radius server still has the duplicate context or not. Is that possible though ?
Is this a known bug ? If yes, is it fixed and in what release ? For now,
I am changing the eap_handler_cmp() function to compare the identity if the state is the same, since in our lab setup we also have a unique identity for each user, do you see a problem with doing this.
No. I would suggest in your situation also adding a comparison on the source port. That should work ard the problem.
I'm hesitant to add that comparison to the main code, though. It may cause other peoples networks to break.
Thats true, but how do we deal with this as a general solution ? If the product we build needs to sit in a network where they have deployed free radius we cannot modify their code and will need a solution from the freeradius community. I seems like we need to investigate the generate_state() function and see if we really generate random states. I have uploaded the log file you can download it at http://download.yousendit.com/D1B4E30A06784505 (its 3MB and I was not sure if I would get flamed for attaching it, it will be available for 14 days at this location). Here are the important line numbers Line 23472: Access-Request from 201myid@mynet.net Line 23521: A new state is generated for 210myid@myid.net and Access-Challenge is sent Line 27092: Access-Request from 245myid@mynet.net Line 27141: State generate for 245myid@mynet.net but rbtree_insert fails due to duplicate state (same as 201myid@mynet.net) Access-Challenge sent to 245myid@mynet.net Line 43911: Response to Access-Challenge from 201myid@mynet.net Line 43964: Freeing handler for 201myid@mynet.net and sending Access-Accept. Line 49522: Access-Request from 245myid@mynet.net, this is a response to the Access-Challenge Lin3 49569: eaplist_find() fails to find the handler for 245myid@mynet.net since we had already sent the reply to 201myid@mynet.net Thanks for your time, Vinay Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Vinay Wagh wrote
Which is why it took me some time to figure this out. What I did was added debug code in rbtree_insert to print out contents of the node if a duplicate node existed. In the logs I saw that the node had the same state but a different identity.
Ok. That shouldn't be happening. It may be an internal race condition in the server.
The reason I started debugging this problem is because I started getting Access-Reject without RADIUS_ATTR_MESSAGE_AUTHENTICATOR which is a seperate attribute in the radius message.
Yes... Please also use the common name "Message-Authenticator". RADIUS_ATTR_MESSAGE_AUTHENTICATOR is an implementation-specific name on your system.
I also observed that some reply's from the radius server had this field but it did not match the authenticator in the original request.
That statement makes no sense. The Message-Authenticator attribute is not supposed to be matched with anything. Are you saying that it fails validation on the client?
Then I tried to link this to the problem I found and I think it is possible if we generate the same state. Assume that after the server sends the access challenge the radius server fails to insert the handler because there is already a duplicate and then before it gets rid of the duplicate handler the client replies. In this case the radius server will try to look for the handler and actually find it since the id, ip addr and state is the same but the identity is different. It can then use that context to reply to this request in which case the fields may not match. If the radius server had already replied to the duplicate handler then it will not find the handler that for our current request and send an Access-Reject.
OK...
So whether we get an Access-Reject or a reply with invalid message authenticator depends on the timing of whether the radius server still has the duplicate context or not. Is that possible though ?
I can understand why it would send an Access-Reject. I don't understand why it would reply with an invalid Message-Authenticator. The calculation for Message-Authenticator is done in src/lib/radius.c, which is independent of any issues in rlm_eap.
Thats true, but how do we deal with this as a general solution ? If the product we build needs to sit in a network where they have deployed free radius we cannot modify their code and will need a solution from the freeradius community. I seems like we need to investigate the generate_state() function and see if we really generate random states.
If it's a bug in FreeRADIUS, it needs to be fixed. The fact that this is seen only for 1000 EAP packets/s indicates it might be a race condition. i.e. there are few systems currently handling 1000 EAP packets/s. Even in your test, I suspect it's doing EAP-MD5. If not, it's doing another non-TLS EAP method. If it is doing a TLS method, then either it has hardware acceleration, or you have *huge* amounts of CPU power available to the RADIUS server.
I have uploaded the log file you can download it at http://download.yousendit.com/D1B4E30A06784505 (its 3MB and I was not sure if I would get flamed for attaching it, it will be available for 14 days at this location). Here are the important line numbers
Ok. As a potential work-around, try editing src/include/libradius.h. Look for the lrad_randctx structure, and change every entry from: uint32_t ... to uint32_t volatile ... That *might* help. If it does, I'll commit the change. Alan DeKok.
participants (4)
-
Alan DeKok -
Igor Smitran -
tnt＠kalik.co.yu -
Vinay Wagh