[4.0.x] radiusd process CPU spikes at 400% when trying to do DHCP concurrently

[Alan DeKok]

On Dec 19, 2019, at 10:38 AM, Chaigneau, Nicolas via Freeradius-Devel <freeradius-devel at lists.freeradius.org> wrote:
> 
> Using FreeRADIUS 4.0.x (HEAD from today), I've tried to set up a dummy (very dumb) DHCP server, which offers random IP addresses (unlang only).
> This is with a build in non developer mode (configured with --disable-developer), with default configuration (except for the dummy DHCP virtual server - see file attached).
> 
> 
> Sending a few DHCP Discover packets (about 100 is enough) concurrently:
> 
> The radiusd process CPU spikes up to 400% (8 CPU are available), and then stays here forever.
> If the packets are sent sequentially there is no problem.
> 
> 
> Something's definitely wrong, but I don't know how to look into this...

  We've seen that in our tests, and it's been difficult to track down.  :(

  If you install gperftools, it has a profiler which may help.  Just re-run configure / make, and the server will be built with the performance tools.

  Then, do:

env CPUPROFILE=/tmp/freeradius.prof CPUPROFILESIGNAL=12 freeradius ...
killall -12 freeradius
  ... wait ...
killall -12 freeradius
pprof --text /path/to/freeradius /tmp/freeradius.prof

  or

pprof --callgrind /path/to/freeradius /tmp/freeradius.prof


  The profile should tell you where all of the CPU is going.

  *Why* it's there is a different story. :(

> I can reproduce this behaviour easily, so I can test things if you want me to.

  What would be enormously useful is either some profile output as above, or tracking it down to a particular commit range.  i.e. does it work with code from a month ago?  If so, then it could be tracked down to one commit or one set of commits.

  We're also looking at this internally, but it's been difficult for us to reproduce.

  Alan DeKok.