How does freeradius calculate the number of required threads?
Hi group, I'm trying to figure out optimal configuration for our radius-proxy farm. atm the farm is handling about 10-15 req/sec per device with the following config (two servers, load-balanced using an slb) thread pool { start_servers = 20 max_servers = 120 min_spare_servers = 20 max_spare_servers = 70 max_requests_per_server = 300 } perl { {...} max_clones = 100 start_clones = 20 min_spare_clones = 20 max_spare_clones = 50 cleanup_delay = 5 max_request_per_clone = 100 } With this configuration both servers run between 25 and 35 threads. Everything is fine (except for huge memory consumption), but when i try to point another device to the load balanced ip (device creates about 15-20req/sec (auth+acct)) freeradius starts spawning threads untill it hits the limit (both freeradius and rlm_perl) and then it locks. I suppose the problems are caused by the fact that the number of requests increases so suddenly, but I have no other way of switching the device over. Is there any way I can influence that behaviour? I suspect that about 50-60 threads would do the job, but if i drop the limit to 60 it just locks quicker. Any ideas how to fix that? regards pshemko
"Pshem Kowalczyk" <pshem.k@gmail.com> wrote:
I'm trying to figure out optimal configuration for our radius-proxy farm. atm the farm is handling about 10-15 req/sec per device with the following config (two servers, load-balanced using an slb)
A normal RADIUS server should be able to handle 100's to 1000's of requests per second. 10-15/s is pretty small.
With this configuration both servers run between 25 and 35 threads. Everything is fine (except for huge memory consumption), but when i try to point another device to the load balanced ip (device creates about 15-20req/sec (auth+acct)) freeradius starts spawning threads untill it hits the limit (both freeradius and rlm_perl)
The server spawns more threads when all of the current threads are busy. It looks to me like your Perl scripts are taking WAY too long to process a request. i.e. If they take 1/10 of a second, then your server can handle 10 requests/s.
I suspect that about 50-60 threads would do the job, but if i drop the limit to 60 it just locks quicker. Any ideas how to fix that?
The problem isn't the number of threads. It's that each request is taking too long to process. Find out why. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
On 10/29/06, Alan DeKok <aland@deployingradius.com> wrote: {cut}
The server spawns more threads when all of the current threads are busy. It looks to me like your Perl scripts are taking WAY too long to process a request. i.e. If they take 1/10 of a second, then your server can handle 10 requests/s.
I suspect that about 50-60 threads would do the job, but if i drop the limit to 60 it just locks quicker. Any ideas how to fix that?
The problem isn't the number of threads. It's that each request is taking too long to process. Find out why.
I'll have a look into that. The question is - how can I effectively measure the time that a single request takes? I suspect that the problem might be caused by the fact, that some information from every accounting packet is stored in the database (even though the load on the db machine is low, the number of simultanious request might be causing some problems). My next question is - what should be the realation between the number of freeradius threads and rlm_perl threads? Almost every single request requires some perl help - should there be a match - 1 to 1 beetween them? kind regards pshemko
On 10/29/06, Alan DeKok <aland@deployingradius.com> wrote: {cut}
The server spawns more threads when all of the current threads are busy. It looks to me like your Perl scripts are taking WAY too long to process a request. i.e. If they take 1/10 of a second, then your server can handle 10 requests/s.
{cut}
The problem isn't the number of threads. It's that each request is taking too long to process. Find out why.
You were right (as usual) :-), the configuration of the proxy was a bit flawed, and instead of caching the destination after figuring out what to do with the packet the perl script was 'forgetting' all the maths, so every single packet had to be individually analysed. That was taking the load (and processing time) through the roof . After getting that under control system behaves now much better (not perfect, but that's rather problem of tunning). I have one more question, though, - assuming that almost every single packet request requires some perl intervention - what should be the ratio of freeradius threads to rlm_perl threads (at least which number should be higher, or should they be equal)? kind regards pshemko
"Pshem Kowalczyk" <pshem.k@gmail.com> wrote:
I have one more question, though, - assuming that almost every single packet request requires some perl intervention - what should be the ratio of freeradius threads to rlm_perl threads (at least which number should be higher, or should they be equal)?
They can be equal. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Another update I've measured the time required by a single function under light load (5-7req/sec) - on average it takes ~ 0.01 to 0.015 to process authorize, pre-acct and less then half of that for pre/post proxy. Under heavier load the boxes still spiral out of control (but only after some time) and eventually freeradius disappears of the list of processes in a puff of white smoke, without leaving any trails in the logs (except for complains about max of thread running). I have no idea what causes that, but the fact that it doesn't happen straightaway suggests some memory problems (but not lack of it, the boxes have 4GB of RAM). I think I hit a wall :-( and I don't think rlm_perl is going to make it to the next phase of the project. regards pshemko
participants (2)
-
Alan DeKok -
Pshem Kowalczyk