RE: anyone know what actually causes this? "FATAL! Server is too busyto process requests"
Hi Tony, I've run into this problem in the past. What version of freeRADIUS are you running? Like you I found that it appears more often when proxying requests to a home server - I guess the reqeusts sit in the queue longer waiting for a reply. Alan was kind enough to supply a patch within hours for 1.1.0 and later. It was bug #331. However, we are still using 1.0.1, which couldn't take advantange of that patch. I had to make some small bandaid changes to request_dequeue, which reduced the change of this happening, at the expense of dropping some requests. Let me know if you need those patches. regards, Mike
-----Original Message----- From: freeradius-users-bounces+michael.mitchell=team.telstra.com@lis ts.freeradius.org [mailto:freeradius-users-bounces+michael.mitchell=team.telstra .com@lists.freeradius.org] On Behalf Of Tony Redstone Sent: Friday, 19 May 2006 12:03 AM To: freeradius-users@lists.freeradius.org Subject: anyone know what actually causes this? "FATAL! Server is too busyto process requests"
we occasionally get these errors in our logs: Thu May 18 09:31:05 2006 : Error: FATAL! Server is too busy to process requests
and the server dies. I've found the core in src/main/threads.c that spits out this message but it's not clear to me under what circumstances this would/should happen. It happens on several boxes ranging from redhat 8.0 to fc4, some processes are talking to backend SQL databases some aren't (in fact, it's our plain radius passthrough/routing proxy servers with file only based config that seem to experience this issue the most).
does anyone know what really causes this and how to fix it? we're serving around 50k DSL users with a nominal authentication load of around 200-300 requests per minute (this is for authentication only, accounting is handled by separate processes which don't experience the same problem). peak load can run into several thousands per minute when recovering from planned maintenance on part of the DSL network.
Tony
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Hi Michael, On 5/18/06, Mitchell, Michael J <Michael.Mitchell@team.telstra.com> wrote:
Hi Tony,
I've run into this problem in the past. What version of freeRADIUS are you running?
until recently we were running 1.0.5 but we've just upgraded to 1.1.1 and we see the same issue (which doesn't surprise me since the relevant threads.c code doesn't appear to have changed).
Like you I found that it appears more often when proxying requests to a home server - I guess the reqeusts sit in the queue longer waiting for a reply.
Alan was kind enough to supply a patch within hours for 1.1.0 and later. It was bug #331.
looks quite relevant, I'm going to try to understand the patch and then perhaps apply the patch on one of our servers to see if it helps.
However, we are still using 1.0.1, which couldn't take advantange of that patch. I had to make some small bandaid changes to request_dequeue, which reduced the change of this happening, at the expense of dropping some requests. Let me know if you need those patches.
this also sounds interesting, under extreme conditions, when, in our case, BT dropped around 30k users simultaneously during one maintenance window (SDH MUX software changes), we had to resort to forcibly killing and restarting our freeradius proxies every 5s in order to get users back online. it would handle requests for the first 2-3s after being started and then pegged at 100% cpu and served no further requests. any patches which touch the request load are of definite interest to us. I would appreciate it if you could forward them to me? Tony
regards, Mike
-----Original Message----- From: freeradius-users-bounces+michael.mitchell=team.telstra.com@lis ts.freeradius.org [mailto:freeradius-users-bounces+michael.mitchell=team.telstra .com@lists.freeradius.org] On Behalf Of Tony Redstone Sent: Friday, 19 May 2006 12:03 AM To: freeradius-users@lists.freeradius.org Subject: anyone know what actually causes this? "FATAL! Server is too busyto process requests"
we occasionally get these errors in our logs: Thu May 18 09:31:05 2006 : Error: FATAL! Server is too busy to process requests
and the server dies. I've found the core in src/main/threads.c that spits out this message but it's not clear to me under what circumstances this would/should happen. It happens on several boxes ranging from redhat 8.0 to fc4, some processes are talking to backend SQL databases some aren't (in fact, it's our plain radius passthrough/routing proxy servers with file only based config that seem to experience this issue the most).
does anyone know what really causes this and how to fix it? we're serving around 50k DSL users with a nominal authentication load of around 200-300 requests per minute (this is for authentication only, accounting is handled by separate processes which don't experience the same problem). peak load can run into several thousands per minute when recovering from planned maintenance on part of the DSL network.
Tony
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Tony Redstone wrote:
Hi Michael,
On 5/18/06, Mitchell, Michael J <Michael.Mitchell@team.telstra.com> wrote:
this also sounds interesting, under extreme conditions, when, in our case, BT dropped around 30k users simultaneously during one maintenance window (SDH MUX software changes), we had to resort to forcibly killing and restarting our freeradius proxies every 5s in order to get users back online. it would handle requests for the first 2-3s after being started and then pegged at 100% cpu and served no further requests. any patches which touch the request load are of definite interest to us. I would appreciate it if you could forward them to me?
Tony
For my last major upgrade push, lasting some few months, I put in quite a bunch of patches for the system I am currently running -- iirc, thread requests and dying after the "thundering herd" was something I tried to deal with. Currently the last thing I attempted to address was preventing core dumps after mysql database becomes "locked". YMMV but you can have a look at it at http://www.jmaimon.com/freeradius specfically from the debian repository, grab the source package and look at the packages I am using today right now. Joe
participants (3)
-
Joe Maimon -
Mitchell, Michael J -
Tony Redstone