Re: Waking up in 999995.9 seconds.
Date: Mon, 9 Jul 2018 12:16:06 -0400 From: Alan DeKok <aland@deployingradius.com> To: FreeRadius users mailing list <freeradius-users@lists.freeradius.org> Subject: Re: Waking up in 999995.9 seconds. Message-ID: <7DB29C1F-AA12-495D-817F-793873EF1354@deployingradius.com> Content-Type: text/plain; charset=us-ascii
On Jul 9, 2018, at 9:38 AM, Duncan Carter <duncan.carter@comms365.com> wrote:
We've got about 20 freeradius servers running for different customers, all have been happily running for months, the only config change I'm aware of is an additional interface on a new IP that's not currently in use.
Last week, we started getting the following error coming up in the logs:
Error: Dropping request (1025 is too many): from client admin port 58950 - ID: 81 ...
Increase "max_requests", and check why the database is too slow.
There's no database involved. Clients, users and logs are all flat files. We have a separate application that processes the log files. We wondered if that's locking the log files somehow, so we tested turning off the app and it made no difference.
We've been restarting radiusd to get the service working again, but I've had a chance to run radiusd in debug mode today, and this stands out in the log files: tail -f radiusd_-X_debug.log | grep Waking Waking up in 0.3 seconds. Waking up in 999994.6 seconds. Waking up in 1999991.9 seconds. Waking up in 0.3 seconds. Waking up in 4.6 seconds. Waking up in 3.9 seconds. Waking up in 999995.9 seconds. Waking up in 1999991.9 seconds. Waking up in 3997746.1 seconds.
Looking at that, it's only a matter of time before the 1024 threads/max_requests are asleep.
Running it in debug mode overnight, it did indeed fall over when all of the threads were 'Waking up in 99999x.x seconds'.
Can someone tell me why it's going to sleep for so long?
Probably an issue with an older version of the server.
Upgrade to 3.0.17.
There's a feature we use that's not included with the newer versions, my colleague who was dealing with isn’t in the office at the moment but I'll find out what that was and test it on one side (dual DC).
Alan Dekok.
Duncan Carter.
On Jul 10, 2018, at 4:49 AM, Duncan Carter <duncan.carter@comms365.com> wrote:e database is too slow.
There's no database involved. Clients, users and logs are all flat files.
Well, maybe it really is getting 1024 simultaneous packets.
There's a feature we use that's not included with the newer versions,
Which is what? There have been very few useful features removed from the server over the years. Alan DeKok.
There's no database involved. Clients, users and logs are all flat files. Well, maybe it really is getting 1024 simultaneous packets. Simultaneous? No, it's getting about 2500 packets (access and accounting) in 3-4 hours, then crashing.
There's a feature we use that's not included with the newer versions, Which is what? There have been very few useful features removed from the server over the years. Found my colleagues notes, it relates to these issues: https://github.com/FreeRADIUS/freeradius-server/issues/1131 https://github.com/FreeRADIUS/freeradius-server/issues/1384 We use: log { ... requests = ${logdir}/radiusd-%{%{Virtual-Server}:-DEFAULT}-%Y%m%d.log ... Which is apparently depreciated, but I can't identify why. We then process those log files to populate a MongoDB.
I really want to know why the "Waking up in" time jumps from 0.x, to 3.x, to 4.x, to 9999x.x to 3978xxx.x seconds? Would you like me to send a complete debug log to you Alan? Many thanks in advance,
its very likely a bug thats already fixed in much newer versions. older versions of the server may have bugs, unless proved that its not fixed in latest release, little point in doing further diagnosis. alan
On Jul 10, 2018, at 5:56 PM, Duncan Carter <duncan.carter@comms365.com> wrote:
There's no database involved. Clients, users and logs are all flat files. Well, maybe it really is getting 1024 simultaneous packets. Simultaneous? No, it's getting about 2500 packets (access and accounting) in 3-4 hours, then crashing.
Well, upgrade. You haven't said what version you're using. But it's old. Very old.
There's a feature we use that's not included with the newer versions, Which is what? There have been very few useful features removed from the server over the years. Found my colleagues notes, it relates to these issues: https://github.com/FreeRADIUS/freeradius-server/issues/1131 https://github.com/FreeRADIUS/freeradius-server/issues/1384 We use: log { ... requests = ${logdir}/radiusd-%{%{Virtual-Server}:-DEFAULT}-%Y%m%d.log ... Which is apparently depreciated, but I can't identify why. We then process those log files to populate a MongoDB.
It was too much of a PITA to manage. You can update the "log" section in 3.0.17 to include the virtual server. Then, post-process the logs as you do today.
I really want to know why the "Waking up in" time jumps from 0.x, to 3.x, to 4.x, to 9999x.x to 3978xxx.x seconds?
A bug which was found and fixed years ago.
Would you like me to send a complete debug log to you Alan?
No. I'm not going to spend time on a bug which has already been fixed. Alan DeKok.
participants (3)
-
Alan Buxey -
Alan DeKok -
Duncan Carter