On Fri, 2007-10-12 at 14:55 +0200, Alan DeKok wrote:
Phil Mayers wrote:
We had one of our MAC-auth radius server instances hang up with this error at about 0200 this morning.
Ouch.
That server receives pretty heavy load, and it's bursty, so we see this a couple of times a day:
The maximum number of threads (32) are active, cannot spawn new thread to handle request
That shouldn't be a problem. The request will just get queued.
Indeed. It does not seem to cause problems.
...but it does not cause problems. An inability to create a new thread is an entirely different matter though; it implies <max threads are running, the server tried to create a new one, and the OS couldn't allocate a thread.
Any ideas how to resolve this? Version is FreeRadius 1.1.6 (only reason we haven't upgraded is change control, it's due shortly)
Set all of the thread information to the same numbers:
start_servers = 32 max_servers = 32 min_spare_servers = 0 max_spare_servers = 32
That way threads won't be created, but they also won't be deleted. I suspect it's the deletion of threads that is causing the problem. i.e. delete/create/delete/create/.../panic !
We just had a repeat of the on the *other* server. Given the relative loads, uptimes of the processes, and burst nature of the load, I am wondering if there is some limit on the total number of thread creates over the lifetime of a process (e.g. 2^16, 2^24). Since the load is bursty, I suspect with the default settings the pool would have been resizing frequently. (For info, OS is Linux 2.6.9, RHEL4 kernel -22.0.1ELsmp, glibc 2.3.4 RPM release 2.16) Anyway, I've implemented this suggestion and we'll see how things go. It seems likely fixing the thread pool size would be trouble-free.