thread pool and requests queueing

[Alan DeKok]

Eugene Grosbein wrote:
> What may be a reason for freeradius idle thread does not processing
> new request from the queue immediately?

  An idle thread should always process *a* request from the queue.  But
it will process the oldest one.  Not a new one.

> As you see, I have lots of threads and my logs show they are mostly idle
> (each thread logs its internal OS pthread id when post_auth is started).

  That doesn't mean the thread is idle.

> However, this what happens here:
> 
> 1. freeradius obtains new request R and successfully queues it.
> I added radlog() call to event.c, received_request() just before "return 1;"
> to make sure.
> 
> 2. Eventually, some thread starts to process this request but between moment 1
> and this moment this thread processes 2-3 another requests. So, the request R
> was waiting in the queue meantime in spite of presence many other idle threads.

  Hmm... that shouldn't happen.

> This is big problem for us as sometimes my perl code NEEDS to insert
> 2 seconds delay before it returns RLM_MODULE_NOTFOUND to the dhcp module
> so that DHCPNAK is sent with 2 seconds delay. This way we keep broken DHCP clients
> from abusing our DHCP relays with tons of requests per second.

  That is the source of your issues.  It would have been polite to say
that at the start.

  The threads MUST NOT BLOCK FOR SECONDS.  The entire server is designed
with that assumption.  If you're deliberately breaking the server, the
all bets are off as to what happens.

  If you need to send a delayed reply, you'll need to modify the server
core.  It has timers for delayed Access-Rejects.  You'll need to do
something similar for these packets.

> These broken clients keep repeating DHCPDISCOVER/DHCPREQUEST
> at high rate if they get DHCPNAK quickly and we cannot get rid of them all.
> With this delay, they patiently wait for answer this giving us
> some kind of "request throttling".

  That's a good intention, but implemented incorrectly.

> How can I force radiusd to use some idle thread at once to minimize queueing delay?

  Step one is to fix your threading issue.  DO NOT have the threads
block for 2 seconds.  It is a terrible, terrible, design.  It is
destroying the server.

  As for the idle thread issue, I have no idea.  All I know is that it
works for me.  Maybe something you're doing is breaking the server.

  Alan DeKok.