2.2.0 & dhcp: regression

[Phil Mayers]

On 12/07/13 11:55, Eugene Grosbein wrote:
> On 12.07.2013 17:38, Phil Mayers wrote:
>> On 12/07/13 11:17, Eugene Grosbein wrote:
>>>
>>> Please help. We need at least 1000 concurrent threads to deal with the load here.
>>
>> 1000 threads is a crazy number. Can you explain why you think you need
>> that many? Are you doing very slow logic/lookups or something?
>
> Our database is powerful enough to deal with so many requests.
> We may easily get that many requests and want to be able to process
> them in parallel without needless queueing.

With respect, this is a pretty basic logic.

The figure of merit here is offered load in terms of request/sec, and 
the average/max processing time per-request.

If you have 1000 request/sec and each request takes 1 millisecond to 
process, a single thread is sufficient.

If you have 100,000 request/sec and each request takes 10 milliseconds, 
then you need 100 threads.

If your database is "so powerful" it shouldn't be taking too long, so 
unless you have a truly enormous number of request/sec, you don't need 
1000 threads.

If you really do have that many request/sec, you probably should look at 
some form of load balancing, rather than having enormous thread pool. 
The thread pool performance will not scale linearly - various server 
internal data structures are locked, and you will probably run into lock 
contention at high thread counts.

I assert that 1000 (posix, shared-memory) threads is always the wrong 
answer to pretty much *any* problem ;o)

>
>> Anyway, the problem is almost certainly system ulimits. I don't know why
>> it's different under 2.2 to 2.1, but I would look into ulimit.
>
> ulimit for files is over 11000 here. And radiusd successfully opens more
> than 1024 files. It just breaks afterwards.

It could be the use of select() then. If rlm_perl opens FDs 5-1023, then 
the radius server needs to open some sockets, it will get FD #1024, 
which might cause select to complain.

Try running the server under "strace" and see if you can see where it 
goes wrong, and what it calls just before it does.