FATAL: Thread create failed: Cannot allocate memory

[Phil Mayers]

On Fri, 2007-10-12 at 14:55 +0200, Alan DeKok wrote:
> Phil Mayers wrote:
> > We had one of our MAC-auth radius server instances hang up with this
> > error at about 0200 this morning.
> 
>   Ouch.
> 
> > That server receives pretty heavy load, and it's bursty, so we see this
> > a couple of times a day:
> > 
> > The maximum number of threads (32) are active, cannot spawn new thread
> > to handle request
> 
>   That shouldn't be a problem.  The request will just get queued.

Indeed. It does not seem to cause problems.

> 
> > ...but it does not cause problems. An inability to create a new thread
> > is an entirely different matter though; it implies <max threads are
> > running, the server tried to create a new one, and the OS couldn't
> > allocate a thread.
> > 
> > Any ideas how to resolve this? Version is FreeRadius 1.1.6 (only reason
> > we haven't upgraded is change control, it's due shortly)
> 
>   Set all of the thread information to the same numbers:
> 
>   start_servers = 32
>   max_servers = 32
>   min_spare_servers = 0
>   max_spare_servers = 32
> 
>   That way threads won't be created, but they also won't be deleted.  I
> suspect it's the deletion of threads that is causing the problem.  i.e.
> delete/create/delete/create/.../panic !

We just had a repeat of the on the *other* server. Given the relative
loads, uptimes of the processes, and burst nature of the load, I am
wondering if there is some limit on the total number of thread creates
over the lifetime of a process (e.g. 2^16, 2^24). Since the load is
bursty, I suspect with the default settings the pool would have been
resizing frequently.

(For info, OS is Linux 2.6.9, RHEL4 kernel -22.0.1ELsmp, glibc 2.3.4 RPM
release 2.16)

Anyway, I've implemented this suggestion and we'll see how things go. It
seems likely fixing the thread pool size would be trouble-free.