Dear Friends, Can we do throttling of no of request in process queue at freeRadius? FreeRadius is depending on mysql, if mysql db connectivity is down for some time then GGSN will feed freeRadius. Radius server may run into the 'child hung' issue then we need to restart freeRadius. To avoid child hung and restart of system. Can we do following? 1) Apply a cap on freeRadius processing queue i.e. if backlog on processing queue is more than 50000 request then reject requests further request until it comes down less than 50000. It will avoid child hung issue. My understanding on child hung: 1) If backlog is more in processing queue then it will spend time on processing head of the queue. 2) messages are reaching age limit of unattended age limit at tail of the queue. 3) Due to that child hung is happening. -- Regards, Srithar Durairaj Alternate Mail I.D: srijeevadurai1@yahoo.co.in Mobile: +919886251852
On 2 Jul 2015, at 09:47, srithar jeevadurai <srijeevadurai1@gmail.com> wrote:
Dear Friends,
Can we do throttling of no of request in process queue at freeRadius?
FreeRadius is depending on mysql, if mysql db connectivity is down for some time then GGSN will feed freeRadius.
If the database is down adjust the query and connection timeouts to ensure the worker threads don't block for an excessively long period.
Radius server may run into the 'child hung' issue then we need to restart freeRadius.
To avoid child hung and restart of system. Can we do following?
If you're dealing with traffic spikes: Determine the optimal number of parallel inserts. Set the max connections in the SQL pool to be that number. Use a detail writer/reader to insert the excesss requests into a file based queue. accounting { sql { fail = 1 } if (fail) { detail } } Example of the detail reader/writer here: https://github.com/FreeRADIUS/freeradius-server/blob/v3.1.x/raddb/sites-avai...
1) Apply a cap on freeRadius processing queue i.e. if backlog on processing queue is more than 50000 request then reject requests further request until it comes down less than 50000.
and drop accounting data in the process?
It will avoid child hung issue.
Probably not.
My understanding on child hung:
Is incorrect.
1) If backlog is more in processing queue then it will spend time on processing head of the queue. 2) messages are reaching age limit of unattended age limit at tail of the queue. 3) Due to that child hung is happening.
Nope. It's happening because your server is performing blocking calls to an overloaded database, which are taking too long. This would be one of the advantages to moving to an asynchronous processing model. We could set a hard coded one second query timeout, and force everyone to fix their databases. -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
Hi Arran, Thanks a lot for your reply. Let me explain the actual scenario. 1) Radius server is sending request to an external process over tcp/ip socket for accounting (start/stop/interim) request. 2) Based on the external process response, radius (custom module) will reply to GGSN. 3) Some time if response is slow from external process then radius server is facing child hung error and we have to restart radius server for the same. 4) Since it is online system, we don;t want child hung happens actual intention is not to restart the system. 5) I believe that it is happening may be due to radius process queue is filled with request and getting aged out. Is there any throttling can be done at freeRadius to handle the same so that it will not run into child hung state/no need to restart radius server. I request you to provide your suggestion please. Regards, Srithar On Thu, Jul 2, 2015 at 7:36 PM, Arran Cudbard-Bell < a.cudbardb@freeradius.org> wrote:
On 2 Jul 2015, at 09:47, srithar jeevadurai <srijeevadurai1@gmail.com> wrote:
Dear Friends,
Can we do throttling of no of request in process queue at freeRadius?
FreeRadius is depending on mysql, if mysql db connectivity is down for some time then GGSN will feed freeRadius.
If the database is down adjust the query and connection timeouts to ensure the worker threads don't block for an excessively long period.
Radius server may run into the 'child hung' issue then we need to restart freeRadius.
To avoid child hung and restart of system. Can we do following?
If you're dealing with traffic spikes:
Determine the optimal number of parallel inserts.
Set the max connections in the SQL pool to be that number.
Use a detail writer/reader to insert the excesss requests into a file based queue.
accounting { sql { fail = 1 } if (fail) { detail } }
Example of the detail reader/writer here:
https://github.com/FreeRADIUS/freeradius-server/blob/v3.1.x/raddb/sites-avai...
1) Apply a cap on freeRadius processing queue i.e. if backlog on
processing
queue is more than 50000 request then reject requests further request until it comes down less than 50000.
and drop accounting data in the process?
It will avoid child hung issue.
Probably not.
My understanding on child hung:
Is incorrect.
1) If backlog is more in processing queue then it will spend time on processing head of the queue. 2) messages are reaching age limit of unattended age limit at tail of the queue. 3) Due to that child hung is happening.
Nope. It's happening because your server is performing blocking calls to an overloaded database, which are taking too long.
This would be one of the advantages to moving to an asynchronous processing model. We could set a hard coded one second query timeout, and force everyone to fix their databases.
-Arran
Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team
FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
-- Regards, Srithar Durairaj Alternate Mail I.D: srijeevadurai1@yahoo.co.in Mobile: +919886251852
On 2 Jul 2015, at 11:01, srithar jeevadurai <srijeevadurai1@gmail.com> wrote:
Hi Arran,
Thanks a lot for your reply. Let me explain the actual scenario.
1) Radius server is sending request to an external process over tcp/ip socket for accounting (start/stop/interim) request. 2) Based on the external process response, radius (custom module) will reply to GGSN.
3) Some time if response is slow from external process then radius server is facing child hung error and we have to restart radius server for the same.
Ok, well that's an issue. You should probably add a socket timeout in your module, so the server doesn't think the request has hung. If all the workers are blocked in a system call, without sane timeouts no matter what else you change it's going to break the server, and require it to be restarted.
4) Since it is online system, we don;t want child hung happens actual intention is not to restart the system.
Fix your module.
5) I believe that it is happening may be due to radius process queue is filled with request and getting aged out.
Even if that's the case, it's a symptom of the real issue, which is your custom module. -Arran
On Jul 2, 2015, at 11:01 AM, srithar jeevadurai <srijeevadurai1@gmail.com> wrote:
Thanks a lot for your reply. Let me explain the actual scenario.
1) Radius server is sending request to an external process over tcp/ip socket for accounting (start/stop/interim) request.
That's normal. It's what the SQL module uses to query the SQL databases.
2) Based on the external process response, radius (custom module) will reply to GGSN.
That too, is normal.
3) Some time if response is slow from external process then radius server is facing child hung error and we have to restart radius server for the same.
Not exactly.... if the response takes *forever*, then the child thread will be blocked, because it's waiting for the reply from the TCP connection. Here, "forever" means anything over 5 seconds. If your back end can't respond to a TCP request in 5 seconds, it's broken. Fix it. No amount of poking FreeRADIUS will make the back-end faster.
4) Since it is online system, we don;t want child hung happens actual intention is not to restart the system.
Or, you can close the blocked TCP connections. Or, re-start the back-end you're using. Which will close the blocked TCP connections.
5) I believe that it is happening may be due to radius process queue is filled with request and getting aged out.
No. The "child is hung" message comes after the request has been pulled out of the queue. If the queue was full, you would see errors saying "queue is full". If the request ages out, you would see errors saying that, too. The problem here is that your module is destroying the server. Fix your module. Fix the back-end so it takes less than 0.1s to respond.
Is there any throttling can be done at freeRadius to handle the same so that it will not run into child hung state/no need to restart radius server.
For your situation, no. Any throttling will just hide the problem, and make it worse in the long run.
I request you to provide your suggestion please.
Fix your back-end so that it can handle the load. No amount of poking FreeRADIUS will fix this problem. Alan DeKok.
participants (3)
-
Alan DeKok -
Arran Cudbard-Bell -
srithar jeevadurai