thread pool and requests queueing
Hi! What may be a reason for freeradius idle thread does not processing new request from the queue immediately? thread pool { start_servers = 256 max_servers = 256 min_spare_servers = 256 max_spare_servers = 256 max_queue_size = 262144 max_requests_per_server = 0 } As you see, I have lots of threads and my logs show they are mostly idle (each thread logs its internal OS pthread id when post_auth is started). However, this what happens here: 1. freeradius obtains new request R and successfully queues it. I added radlog() call to event.c, received_request() just before "return 1;" to make sure. 2. Eventually, some thread starts to process this request but between moment 1 and this moment this thread processes 2-3 another requests. So, the request R was waiting in the queue meantime in spite of presence many other idle threads. This is big problem for us as sometimes my perl code NEEDS to insert 2 seconds delay before it returns RLM_MODULE_NOTFOUND to the dhcp module so that DHCPNAK is sent with 2 seconds delay. This way we keep broken DHCP clients from abusing our DHCP relays with tons of requests per second. These broken clients keep repeating DHCPDISCOVER/DHCPREQUEST at high rate if they get DHCPNAK quickly and we cannot get rid of them all. With this delay, they patiently wait for answer this giving us some kind of "request throttling". How can I force radiusd to use some idle thread at once to minimize queueing delay? Eugene
Eugene Grosbein wrote:
What may be a reason for freeradius idle thread does not processing new request from the queue immediately?
An idle thread should always process *a* request from the queue. But it will process the oldest one. Not a new one.
As you see, I have lots of threads and my logs show they are mostly idle (each thread logs its internal OS pthread id when post_auth is started).
That doesn't mean the thread is idle.
However, this what happens here:
1. freeradius obtains new request R and successfully queues it. I added radlog() call to event.c, received_request() just before "return 1;" to make sure.
2. Eventually, some thread starts to process this request but between moment 1 and this moment this thread processes 2-3 another requests. So, the request R was waiting in the queue meantime in spite of presence many other idle threads.
Hmm... that shouldn't happen.
This is big problem for us as sometimes my perl code NEEDS to insert 2 seconds delay before it returns RLM_MODULE_NOTFOUND to the dhcp module so that DHCPNAK is sent with 2 seconds delay. This way we keep broken DHCP clients from abusing our DHCP relays with tons of requests per second.
That is the source of your issues. It would have been polite to say that at the start. The threads MUST NOT BLOCK FOR SECONDS. The entire server is designed with that assumption. If you're deliberately breaking the server, the all bets are off as to what happens. If you need to send a delayed reply, you'll need to modify the server core. It has timers for delayed Access-Rejects. You'll need to do something similar for these packets.
These broken clients keep repeating DHCPDISCOVER/DHCPREQUEST at high rate if they get DHCPNAK quickly and we cannot get rid of them all. With this delay, they patiently wait for answer this giving us some kind of "request throttling".
That's a good intention, but implemented incorrectly.
How can I force radiusd to use some idle thread at once to minimize queueing delay?
Step one is to fix your threading issue. DO NOT have the threads block for 2 seconds. It is a terrible, terrible, design. It is destroying the server. As for the idle thread issue, I have no idea. All I know is that it works for me. Maybe something you're doing is breaking the server. Alan DeKok.
On 11.04.2014 18:31, Alan DeKok wrote:
How can I force radiusd to use some idle thread at once to minimize queueing delay?
Step one is to fix your threading issue. DO NOT have the threads block for 2 seconds. It is a terrible, terrible, design. It is destroying the server.
Why does it destroy the server? I can have any large number of lightweight threads to compensate for busyness of sleeping threads. I use FreeBSD having sysctl kern.threads.max_threads_per_proc=1500 by default and may increase it to arbitrary high value.
As for the idle thread issue, I have no idea. All I know is that it works for me. Maybe something you're doing is breaking the server.
I'll try to dig this problem deeper then.
On 11.04.2014 18:31, Alan DeKok wrote:
As you see, I have lots of threads and my logs show they are mostly idle (each thread logs its internal OS pthread id when post_auth is started).
That doesn't mean the thread is idle.
Hmm, yes, now I see it. I've added logging of thread_pool.num_queued value (btw, this is very important to be able to see as in the logs as it changes) and now I see that 256 threads just not enough for this configuration and load as thread_pool.num_queued spikes upto about 800. I'm back to 1000 threads and this keeps thread_pool.num_queued under 2 for now. Thank you for explanations and patience. I'm afraid that changing freeradius core is out of my skills and limited by lack of familiarity with the code. Eugene Grosbein
Eugene Grosbein wrote:
Hmm, yes, now I see it. I've added logging of thread_pool.num_queued value (btw, this is very important to be able to see as in the logs as it changes) and now I see that 256 threads just not enough for this configuration and load as thread_pool.num_queued spikes upto about 800.
Yes. Having the thread blocked for 2 seconds is destroying the server.
I'm back to 1000 threads and this keeps thread_pool.num_queued under 2 for now.
That's good.
Thank you for explanations and patience.
It's what I do. Some people complain I'm mean, but they get out what they put in.
I'm afraid that changing freeradius core is out of my skills and limited by lack of familiarity with the code.
Try the v3.0.x branch from github on Monday. The server already has a reject delay for Access-Reject packets. Making it a bit more generic is fairly trivial. Alan DeKok.
Try the v3.0.x branch from github on Monday. The server already has a reject delay for Access-Reject packets. Making it a bit more generic is fairly trivial.
If it is fairly trivial for you, could you please take it at feature request? :-) I mean, to extent this for delaying dhcp replies so that perl code running under rlm_perl can return needed delay to the core? I would be happy to test 3.0 from github under our severe load then. And add an option to note significant changes of thread_pool.num_queued to the log, too. For example, log this value if it has increased or decreased with configurable number of percents comparing previously logged value. Eugene Grosbein
participants (2)
-
Alan DeKok -
Eugene Grosbein