Waking up in 999995.9 seconds.

[Duncan Carter]

> Date: Mon, 9 Jul 2018 12:16:06 -0400
> From: Alan DeKok <aland at deployingradius.com>
> To: FreeRadius users mailing list
> 	<freeradius-users at lists.freeradius.org>
> Subject: Re: Waking up in 999995.9 seconds.
> Message-ID: <7DB29C1F-AA12-495D-817F-793873EF1354 at deployingradius.com>
> Content-Type: text/plain; charset=us-ascii
> 
> On Jul 9, 2018, at 9:38 AM, Duncan Carter <duncan.carter at comms365.com> wrote:
> > We've got about 20 freeradius servers running for different customers, all have been happily running for months, the only config change I'm aware of is an additional interface on a new IP that's not currently in use.
> > 
> > Last week, we started getting the following error coming up in the logs:
> > 
> >                Error: Dropping request (1025 is too many): from client admin port 58950 - ID: 81 ...
> 
>   Increase "max_requests", and check why the database is too slow.

There's no database involved.  Clients, users and logs are all flat files.  We have a separate application that processes the log files.
We wondered if that's locking the log files somehow, so we tested turning off the app and it made no difference.

> 
> > We've been restarting radiusd to get the service working again, but I've had a chance to run radiusd in debug mode today, and this stands out in the log files:
> > tail -f radiusd_-X_debug.log | grep Waking
> > Waking up in 0.3 seconds.
> > Waking up in 999994.6 seconds.
> > Waking up in 1999991.9 seconds.
> > Waking up in 0.3 seconds.
> > Waking up in 4.6 seconds.
> > Waking up in 3.9 seconds.
> > Waking up in 999995.9 seconds.
> > Waking up in 1999991.9 seconds.
> > Waking up in 3997746.1 seconds.
> > 
> > Looking at that, it's only a matter of time before the 1024 threads/max_requests are asleep.

Running it in debug mode overnight, it did indeed fall over when all of the threads were 'Waking up in 99999x.x seconds'.

> > 
> > Can someone tell me why it's going to sleep for so long?
> 
>   Probably an issue with an older version of the server.
> 
>   Upgrade to 3.0.17.

There's a feature we use that's not included with the newer versions, my colleague who was dealing with isn’t in the office at the moment but I'll find out what that was and test it on one side (dual DC).

> 
>   Alan Dekok.

Duncan Carter.