Radius crashing with "Failed adding event handler for socket: Too many readers"

[Alan DeKok]

On Mar 26, 2018, at 6:00 PM, Campbell McKenzie <Campbell.McKenzie at aarnet.edu.au> wrote:
> 
> I'm running Free Radius 3.0.16 on Redhat 7 and the about once or 10 times a day the service will "crash" (Exit with error code 1) with the error message "Failed adding event handler for socket: Too many readers".
> 
> The only messages prior to Radius crashing are:
> 
> Mar 19 21:37:34 radius1.acme.com radiusd[11344]: ... adding new socket proxy address 202.AA.BB.CC port 57158
> Mar 19 21:37:34 radius1.acme.com radiusd[11344]: ... adding new socket proxy address 202.AA.BB.CC port 57158
> Mar 19 21:37:34 radius1.acme.com radiusd[11344]:Failed adding event handler for socket: Too many readers

  It doesn't crash, it just exists because it doesn't know what else to do.

> Our configuration consists of about 157 home_servers (mostly running in auth+acct) and about 276 clients and two upstream proxies. (Eduroam).
> 
> It seems like a new socket is created for each home_server (auth) + another socket for accounting and with such a high(?) number of home_servers we are running out of sockets (possibly).

  Yes.

> Is there a limit to the number of home_servers and/or clients?

  There's no limit to the number of home servers and/or clients.  I've tested FR with 100K of each.

  There *is* a limit to the number of sockets.  One way to hit that limit for proxy sockets is to specify a "src_ipaddr" in the home server.   Each src_ipaddr means at least one more open socket to send packets.  In addition, high packet rates means that the server opens multiple sockets from the same src_ipaddr, up to 32 per src_ipaddr.

  So if you have 8 src_ipaddr listed, and lots of packets, you can hit 256 sockets pretty quickly.  And that's the limit being enforced with the "Too many readers" error.

  I've pushed a few fixes:

* bump up the maximum number of sockets to 512, which will help a bit
* don't exit on reaching the limit, just close the socket we tried to open

  That will help, but won't fix the underlying issue.  The better fix for you is to reduce the number of "src_ipaddr" in your configuration.  May guess is you have 4-5.  When you add various other sockets that the server uses, you reach the limit of 256 sockets pretty quickly.

  We're fixing this in v4 through the simple expedient of re-designing and re-writing all of the relevant code. :(  The bad news is that it's a lot of work.  The good news is that we're removing all limits on sockets.  So the server should be able to handle 10K connections without an issue.

  Alan DeKok.