Accounting-Response taking too long
Hi all, Radius version : 2.2 (old, but reliable) Ive recently ran into a problem with our accounting Radius server which have seen smooth sailing for a long time. Basically every night at 8pm a stop request is generated for all users and then consequently a start to change the package assigned to the customer. Currently our Radius is serving 19000 customers. The problem we are facing is that the NAS which is basically a session and resource controller sends the accounting-requests but the accounting-responses are taking from around 7 to 10 seconds from the time the Acct-requests come in to be dispatched to the NAS as observed from wire shark, please see attachment. The same is reflected in the radius accounting logs. The problem that occurs due to this is that the NAS when it doesn't receive the response in the given time keeps retrying, sending accounting-requests again and again and eventually errors with retries expired and fails. Should I increase the ''max_requests'' in radiusd.conf, currently it set to 2048. The comments say it should be 256 multiplied by the number clients. Now by clients does it mean the actual number of customers in my case or the number of NASs that connect to the Radius server. (Apologies for the lack of understanding) Accounting responses usually take milliseconds but I don't know why it's taking so long in this case. If I could be guided in how to troubleshoot this or any assistance in this regard would be greatly appreciated. BR/Mustafa
On Mar 5, 2018, at 2:12 PM, mustafa mujahid <mustafa.mujahid@outlook.com> wrote:
Radius version : 2.2 (old, but reliable)
You should upgrade to 2.2.10.
Ive recently ran into a problem with our accounting Radius server which have seen smooth sailing for a long time. Basically every night at 8pm a stop request is generated for all users and then consequently a start to change the package assigned to the customer. Currently our Radius is serving 19000 customers. The problem we are facing is that the NAS which is basically a session and resource controller sends the accounting-requests but the accounting-responses are taking from around 7 to 10 seconds from the time the Acct-requests come in to be dispatched to the NAS as observed from wire shark, please see attachment.
The default configuration works, and doesn't take 10s to respond. So it must be something on your local system which is causing the problem.
The same is reflected in the radius accounting logs. The problem that occurs due to this is that the NAS when it doesn't receive the response in the given time keeps retrying, sending accounting-requests again and again and eventually errors with retries expired and fails.
That's what the NAS should do.
Should I increase the ''max_requests'' in radiusd.conf, currently it set to 2048. The comments say it should be 256 multiplied by the number clients. Now by clients does it mean the actual number of customers in my case or the number of NASs that connect to the Radius server. (Apologies for the lack of understanding)
The NASes. But changing that number won't help.
Accounting responses usually take milliseconds but I don't know why it's taking so long in this case. If I could be guided in how to troubleshoot this or any assistance in this regard would be greatly appreciated.
See this: https://wiki.freeradius.org/list-help You haven't said what you've configured FreeRADIUS to *do* with accounting packets, so it's difficult to help you. All we can say is that there is no magic "wait 10s" setting in the server. In general, delays are due to local issues. e.g. slow database, slow script your executing, etc. But you should ask a *good* question to get a good answer. Right now, vague questions get vague answers, which aren't overly helpful. Alan DeKok.
My apologies Alan, The server has absolutely nothing extra aside from the default configuration, no scripts or any type of automation at that. Its a completely default configuration. I have investigated the DB and plotted a latency graph with the help of the wire shark output. The graph is attached. I first thought it was the DB that was responding slowly but the graph shows only a few requests taking longer than 1 second to complete (y-axis is seconds). Now the issue occurs during the load time of 8pm, I would love to paste the debug output but the debug moves so fast that nothing can be tracked. I'm quite stuck and this has become an issue for us. Anything at all that could be recommended that I can check to identify the issue would be greatly appreciated. Network latency from NAS to Radius and Radius to DB is less than 1 ms. Thank you as always! BR/Mustafa On 06/03/2018 12:21 AM, Alan DeKok wrote: On Mar 5, 2018, at 2:12 PM, mustafa mujahid <mustafa.mujahid@outlook.com><mailto:mustafa.mujahid@outlook.com> wrote: Radius version : 2.2 (old, but reliable) You should upgrade to 2.2.10. Ive recently ran into a problem with our accounting Radius server which have seen smooth sailing for a long time. Basically every night at 8pm a stop request is generated for all users and then consequently a start to change the package assigned to the customer. Currently our Radius is serving 19000 customers. The problem we are facing is that the NAS which is basically a session and resource controller sends the accounting-requests but the accounting-responses are taking from around 7 to 10 seconds from the time the Acct-requests come in to be dispatched to the NAS as observed from wire shark, please see attachment. The default configuration works, and doesn't take 10s to respond. So it must be something on your local system which is causing the problem. The same is reflected in the radius accounting logs. The problem that occurs due to this is that the NAS when it doesn't receive the response in the given time keeps retrying, sending accounting-requests again and again and eventually errors with retries expired and fails. That's what the NAS should do. Should I increase the ''max_requests'' in radiusd.conf, currently it set to 2048. The comments say it should be 256 multiplied by the number clients. Now by clients does it mean the actual number of customers in my case or the number of NASs that connect to the Radius server. (Apologies for the lack of understanding) The NASes. But changing that number won't help. Accounting responses usually take milliseconds but I don't know why it's taking so long in this case. If I could be guided in how to troubleshoot this or any assistance in this regard would be greatly appreciated. See this: https://wiki.freeradius.org/list-help You haven't said what you've configured FreeRADIUS to *do* with accounting packets, so it's difficult to help you. All we can say is that there is no magic "wait 10s" setting in the server. In general, delays are due to local issues. e.g. slow database, slow script your executing, etc. But you should ask a *good* question to get a good answer. Right now, vague questions get vague answers, which aren't overly helpful. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On 8 Mar 2018, at 07:49, mustafa mujahid <mustafa.mujahid@outlook.com> wrote:
The server has absolutely nothing extra aside from the default configuration, no scripts or any type of automation at that. Its a completely default configuration.
FreeRADIUS doesn't come with a database by default, so that at least has been changed.
Anything at all that could be recommended that I can check to identify the issue would be greatly appreciated.
Please read https://wiki.freeradius.org/list-help in full, and provide the information requested. Regards, Adam Bishop gpg: E75B 1F92 6407 DFDF 9F1C BF10 C993 2504 6609 D460 jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. Jisc Services Limited is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under company number 2881024, VAT number GB 197 0632 86. The registered office is: One Castle Park, Tower Hill, Bristol BS2 0JA. T 0203 697 5800.
On Mar 8, 2018, at 2:49 AM, mustafa mujahid <mustafa.mujahid@outlook.com> wrote:
The server has absolutely nothing extra aside from the default configuration,
The default configuration doesn't contain anything which blocks for 8-10 seconds.
no scripts or any type of automation at that. Its a completely default configuration. I have investigated the DB and plotted a latency graph with the help of the wire shark output. The graph is attached. I first thought it was the DB that was responding slowly but the graph shows only a few requests taking longer than 1 second to complete (y-axis is seconds).
Well, it's almost always the DB. FreeRADIUS can do 50K packets/s with sub-millisecond latency on every packet. And if the DB takes 1s to respond, that's a big problem. It should always be much faster.
Now the issue occurs during the load time of 8pm, I would love to paste the debug output but the debug moves so fast that nothing can be tracked. I'm quite stuck and this has become an issue for us. Anything at all that could be recommended that I can check to identify the issue would be greatly appreciated. Network latency from NAS to Radius and Radius to DB is less than 1 ms.
Network latency doesn't matter. The issue is almost definitely the database. If the database is slow, it slows down FreeRADIUS. Alan DeKok.
hi, you need to turn your monitoring guns onto the SQL server - use any of the available SQL monitoring toolsets to eg plot request queue size, cache hits/fails etc. expect you'll find funky things at 20:00 - what DB engine are you using for your SQL tables etc? alan
participants (4)
-
Adam Bishop -
Alan Buxey -
Alan DeKok -
mustafa mujahid