FATAL: Thread create failed: Cannot allocate memory
We had one of our MAC-auth radius server instances hang up with this error at about 0200 this morning. That server receives pretty heavy load, and it's bursty, so we see this a couple of times a day: The maximum number of threads (32) are active, cannot spawn new thread to handle request ...but it does not cause problems. An inability to create a new thread is an entirely different matter though; it implies <max threads are running, the server tried to create a new one, and the OS couldn't allocate a thread. Any ideas how to resolve this? Version is FreeRadius 1.1.6 (only reason we haven't upgraded is change control, it's due shortly)
Phil Mayers wrote:
We had one of our MAC-auth radius server instances hang up with this error at about 0200 this morning.
Ouch.
That server receives pretty heavy load, and it's bursty, so we see this a couple of times a day:
The maximum number of threads (32) are active, cannot spawn new thread to handle request
That shouldn't be a problem. The request will just get queued.
...but it does not cause problems. An inability to create a new thread is an entirely different matter though; it implies <max threads are running, the server tried to create a new one, and the OS couldn't allocate a thread.
Any ideas how to resolve this? Version is FreeRadius 1.1.6 (only reason we haven't upgraded is change control, it's due shortly)
Set all of the thread information to the same numbers: start_servers = 32 max_servers = 32 min_spare_servers = 0 max_spare_servers = 32 That way threads won't be created, but they also won't be deleted. I suspect it's the deletion of threads that is causing the problem. i.e. delete/create/delete/create/.../panic !
On Fri, 2007-10-12 at 14:55 +0200, Alan DeKok wrote:
Phil Mayers wrote:
We had one of our MAC-auth radius server instances hang up with this error at about 0200 this morning.
Ouch.
That server receives pretty heavy load, and it's bursty, so we see this a couple of times a day:
The maximum number of threads (32) are active, cannot spawn new thread to handle request
That shouldn't be a problem. The request will just get queued.
Indeed. It does not seem to cause problems.
...but it does not cause problems. An inability to create a new thread is an entirely different matter though; it implies <max threads are running, the server tried to create a new one, and the OS couldn't allocate a thread.
Any ideas how to resolve this? Version is FreeRadius 1.1.6 (only reason we haven't upgraded is change control, it's due shortly)
Set all of the thread information to the same numbers:
start_servers = 32 max_servers = 32 min_spare_servers = 0 max_spare_servers = 32
That way threads won't be created, but they also won't be deleted. I suspect it's the deletion of threads that is causing the problem. i.e. delete/create/delete/create/.../panic !
We just had a repeat of the on the *other* server. Given the relative loads, uptimes of the processes, and burst nature of the load, I am wondering if there is some limit on the total number of thread creates over the lifetime of a process (e.g. 2^16, 2^24). Since the load is bursty, I suspect with the default settings the pool would have been resizing frequently. (For info, OS is Linux 2.6.9, RHEL4 kernel -22.0.1ELsmp, glibc 2.3.4 RPM release 2.16) Anyway, I've implemented this suggestion and we'll see how things go. It seems likely fixing the thread pool size would be trouble-free.
Hi,
We had one of our MAC-auth radius server instances hang up with this error at about 0200 this morning.
That server receives pretty heavy load, and it's bursty, so we see this a couple of times a day:
The maximum number of threads (32) are active, cannot spawn new thread to handle request
...but it does not cause problems. An inability to create a new thread is an entirely different matter though; it implies <max threads are running, the server tried to create a new one, and the OS couldn't allocate a thread.
Any ideas how to resolve this? Version is FreeRadius 1.1.6 (only reason we haven't upgraded is change control, it's due shortly)
we recently had a similar issue after migrating to using FR for VMPS handling. the system may deal with many hundreds of requests per second as with VMPS the switch re-auths all ports at exactly the same time. with 48 port switches this gets interesting. anyway. the issue was that we had the following config radiusd.conf max_servers = X experimental.conf - perl, max_clones = Y where X != Y this is a big problem and you get the above mentioned errors. you ALSO get the error when X = Y and the load/demand is very high. in this case the radius thread appears to be trying to launch a new PERL instace before the old ones have gone. anyway, a rapid increase of the values helped straight away. as did a proper optimization of the DB to get much much faster PERL code. what you have to ensure in these cases is you dont see these ones: Fri Feb 11 16:00:11 2006 : Error: Discarding duplicate request from client BLAH port 49464 - ID: 10313 due to unfinished request 6 as although seemingly okay the client isnt getting an answer from its requests. alan
participants (3)
-
A.L.M.Buxey＠lboro.ac.uk -
Alan DeKok -
Phil Mayers