[4.0.x] radiusd process CPU spikes at 400% when trying to do DHCP concurrently

[Jorge Pereira]

Nicolas,

Can you share the exactly steps to reproduce it?
--
Jorge Pereira
jpereira at networkradius.com



> On 19 Dec 2019, at 14:59, Chaigneau, Nicolas via Freeradius-Devel <freeradius-devel at lists.freeradius.org> wrote:
> 
> 
> Thanks for the pointers!
> 
> 
> So I went for the "blame a commit" dichotomous approach.
> Think I found it (commit from November 24, "Always return +1 pending event if there's a timer ready"):
> 
> https://github.com/FreeRADIUS/freeradius-server/commit/7c2b992cc79c5c2cdd2863eb6d91ffb9559dd0a9
> 
> 
> Technically I narrowed it down to 2 commits, but I don't think ab3dfc768f2eeeeb5a53eba2944bff984b7543c5 ("Formatting") can be the culprit. :)
> 
> 
> 
> Regards,
> Nicolas.
> 
> 
>> On Dec 19, 2019, at 10:38 AM, Chaigneau, Nicolas via Freeradius-Devel <freeradius-devel at lists.freeradius.org> wrote:
>>> 
>>> Using FreeRADIUS 4.0.x (HEAD from today), I've tried to set up a dummy (very dumb) DHCP server, which offers random IP addresses (unlang only).
>>> This is with a build in non developer mode (configured with --disable-developer), with default configuration (except for the dummy DHCP virtual server - see file attached).
>>> 
>>> 
>>> Sending a few DHCP Discover packets (about 100 is enough) concurrently:
>>> 
>>> The radiusd process CPU spikes up to 400% (8 CPU are available), and then stays here forever.
>>> If the packets are sent sequentially there is no problem.
>>> 
>>> 
>>> Something's definitely wrong, but I don't know how to look into this...
>> 
>>  We've seen that in our tests, and it's been difficult to track down.  :(
>> 
>>  If you install gperftools, it has a profiler which may help.  Just re-run configure / make, and the server will be built with the performance tools.
>> 
>>  Then, do:
>> 
>> env CPUPROFILE=/tmp/freeradius.prof CPUPROFILESIGNAL=12 freeradius ...
>> killall -12 freeradius
>>  ... wait ...
>> killall -12 freeradius
>> pprof --text /path/to/freeradius /tmp/freeradius.prof
>> 
>>  or
>> 
>> pprof --callgrind /path/to/freeradius /tmp/freeradius.prof
>> 
>> 
>>  The profile should tell you where all of the CPU is going.
>> 
>>  *Why* it's there is a different story. :(
>> 
>>> I can reproduce this behaviour easily, so I can test things if you want me to.
>> 
>>  What would be enormously useful is either some profile output as above, or tracking it down to a particular commit range.  i.e. does it work with code from a month ago?  If so, then it could be tracked down to one commit or one set of commits.
>> 
>>  We're also looking at this internally, but it's been difficult for us to reproduce.
>> 
>>  Alan DeKok.
>> 
>> 
> This message contains information that may be privileged or confidential and is the property of the Capgemini Group. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message.
> 
> 
> -
> List info/subscribe/unsubscribe? See http://www.freeradius.org/list/devel.html