FreeRADIUS can't make progress under certain load

[rihad]

On 09/10/2011 09:28 PM, Arran Cudbard-Bell wrote:
> On 10 Sep 2011, at 17:49, rihad wrote:
>
>> Hi, sometimes when a NAS with many users reboots FreeRADIUS is unable to cope with the number of incoming requests:
> How many users ?
>
Around 6-8k online.

>> Sat Sep 10 13:23:16 2011 : Auth: Login OK: [5702018] (from client 10.10.70.100 port 0)
>> Sat Sep 10 13:23:16 2011 : Error: Received conflicting packet from client 10.10.70.98 port 1645 - ID: 131 due to unfinished request 2814. Giving up on old request.
>> Sat Sep 10 13:23:16 2011 : Auth: Login OK: [5026706] (from client 10.10.70.38 port 0)
>> Sat Sep 10 13:23:16 2011 : Error: Received conflicting packet from client 10.10.70.93 port 1646 - ID: 117 due to unfinished request 1036. Giving up on old request.
>> Sat Sep 10 13:23:16 2011 : Auth: Login OK: [4925140] (from client 10.10.70.93 port 0)
>> Sat Sep 10 13:23:16 2011 : Error: Received conflicting packet from client 10.10.70.60 port 1645 - ID: 109 due to unfinished request 3370. Giving up on old request.
>> Sat Sep 10 13:23:16 2011 : Auth: Login OK: [4977364] (from client 10.10.70.38 port 0)
>> Sat Sep 10 13:23:16 2011 : Error: Received conflicting packet from client 10.10.70.93 port 1646 - ID: 118 due to unfinished request 1040. Giving up on old request.
>> Sat Sep 10 13:23:16 2011 : Error: Received conflicting packet from client 10.10.70.60 port 1645 - ID: 110 due to unfinished request 3373. Giving up on old request.
>> Sat Sep 10 13:23:16 2011 : Auth: Login OK: [4529464] (from client 10.10.70.28 port 0)
>> Sat Sep 10 13:23:16 2011 : Error: Received conflicting packet from client 10.10.70.98 port 1645 - ID: 132 due to unfinished request 2829. Giving up on old request.
>>
>> and ad infinitum. The duplicate requests come from PPPoE clients after they fail to receive a response within 5 seconds or so.
> Yes its retransmitting the packets because it assumed they were lost in transit.
>
>> The problem can then be solved by restarting the daemon once or twice and watching the errors go away. The fact that dropping all current work helped gave me this idea: wouldn't it be nice if FreeRADIUS dropped both the old _and_ the new request at the time it logged that "giving up" message?
> No... Because then it'd never respond to either request. In a normal network (not under overload condition) if two packets were not lost but delayed, then both reach the server at around the same time, the server would end up dropping both requests ...and there's a bunch of other reasons depending on the deployment scenario.
>
Don't you think 5 second delay is pretty unusual for a LAN. Yes, UDP 
acct packets sometimes get dropped and never reach the server, but the 
delay... It's much safer to sometimes drop 2 auth requests in a row for 
the reason you stated than allow the server to reach a point where it's 
unable to make further progress doing _current_ work.

>> That would at least allow it to make progress. BTW, I'm not sure why, but under comparable workloads openradius does not exhibit this problem.
> What are you doing with FreeRADIUS which is taking such a long time to complete? The log messages are a symptom not a cause.
>
That's hundreds if not thousands (if you count acct packets too) of 
requests all reaching the server nearly at the same time.

Tomorrow early in the morning I'll try the fix and see how it goes.


Thanks for all the info you gave.

>> Any suggestions?
>
> Fix whatever backend is getting overloaded and making FreeRADIUS wait for>  5 seconds to send a response...
>
> -Arran
>
> Arran Cudbard-Bell
> a.cudbardb at freeradius.org
>
> RADIUS - Waging war on ignorance and apathy one Access-Challenge at a time.
>
>