Radius crashing with "Failed adding event handler for socket: Too many readers"
Hello, I'm running Free Radius 3.0.16 on Redhat 7 and the about once or 10 times a day the service will "crash" (Exit with error code 1) with the error message "Failed adding event handler for socket: Too many readers". The only messages prior to Radius crashing are: Mar 19 21:37:34 radius1.acme.com radiusd[11344]: ... adding new socket proxy address 202.AA.BB.CC port 57158 Mar 19 21:37:34 radius1.acme.com radiusd[11344]: ... adding new socket proxy address 202.AA.BB.CC port 57158 Mar 19 21:37:34 radius1.acme.com radiusd[11344]:Failed adding event handler for socket: Too many readers Then systemd picks up the failure: Mar 19 21:37:34 radius1.acme.com systemd:radiusd.service: main process exited, code=exited, status=1/FAILURE Mar 19 21:37:34 radius1.acme.com systemd:Unit radiusd.service entered failed state. Mar 19 21:37:34 radius1.acme.com systemd:radiusd.service failed. Our configuration consists of about 157 home_servers (mostly running in auth+acct) and about 276 clients and two upstream proxies. (Eduroam). It seems like a new socket is created for each home_server (auth) + another socket for accounting and with such a high(?) number of home_servers we are running out of sockets (possibly). Is there a limit to the number of home_servers and/or clients? Kind regards, Campbell Campbell McKenzie AARNet Pty Ltd address: Tenancy 1, Ground Floor, 143 Coronation Drive, Milton, QLD, 4064, Australia postal: PO Box 1787, Milton, Qld, 4064 w. www.aarnet.edu.au
On Mar 26, 2018, at 6:00 PM, Campbell McKenzie <Campbell.McKenzie@aarnet.edu.au> wrote:
I'm running Free Radius 3.0.16 on Redhat 7 and the about once or 10 times a day the service will "crash" (Exit with error code 1) with the error message "Failed adding event handler for socket: Too many readers".
The only messages prior to Radius crashing are:
Mar 19 21:37:34 radius1.acme.com radiusd[11344]: ... adding new socket proxy address 202.AA.BB.CC port 57158 Mar 19 21:37:34 radius1.acme.com radiusd[11344]: ... adding new socket proxy address 202.AA.BB.CC port 57158 Mar 19 21:37:34 radius1.acme.com radiusd[11344]:Failed adding event handler for socket: Too many readers
It doesn't crash, it just exists because it doesn't know what else to do.
Our configuration consists of about 157 home_servers (mostly running in auth+acct) and about 276 clients and two upstream proxies. (Eduroam).
It seems like a new socket is created for each home_server (auth) + another socket for accounting and with such a high(?) number of home_servers we are running out of sockets (possibly).
Yes.
Is there a limit to the number of home_servers and/or clients?
There's no limit to the number of home servers and/or clients. I've tested FR with 100K of each. There *is* a limit to the number of sockets. One way to hit that limit for proxy sockets is to specify a "src_ipaddr" in the home server. Each src_ipaddr means at least one more open socket to send packets. In addition, high packet rates means that the server opens multiple sockets from the same src_ipaddr, up to 32 per src_ipaddr. So if you have 8 src_ipaddr listed, and lots of packets, you can hit 256 sockets pretty quickly. And that's the limit being enforced with the "Too many readers" error. I've pushed a few fixes: * bump up the maximum number of sockets to 512, which will help a bit * don't exit on reaching the limit, just close the socket we tried to open That will help, but won't fix the underlying issue. The better fix for you is to reduce the number of "src_ipaddr" in your configuration. May guess is you have 4-5. When you add various other sockets that the server uses, you reach the limit of 256 sockets pretty quickly. We're fixing this in v4 through the simple expedient of re-designing and re-writing all of the relevant code. :( The bad news is that it's a lot of work. The good news is that we're removing all limits on sockets. So the server should be able to handle 10K connections without an issue. Alan DeKok.
That will help, but won't fix the underlying issue. The better fix for you is to reduce the number of "src_ipaddr" in your configuration. May guess is you have 4-5. When you add various other sockets that the server uses, you reach the limit of 256 sockets pretty quickly.
We had src_ipaddr specified for all 150+ home_servers - hence why this issue was appearing. Unfortunately due to the dummy interface (anycast) (any0) we had the specify the source ip for various reasons. Ultimately I removed all the src_ipaddr directives and added an iptables SNAT rule to modify all the outbound packets for dest ports 1812/1813. So far it seems to be working, hopefully it doesn't run out of sockets this way.... Thanks for your help Alan. --Campbell
On Mar 28, 2018, at 1:13 AM, Campbell McKenzie <Campbell.McKenzie@aarnet.edu.au> wrote:
We had src_ipaddr specified for all 150+ home_servers - hence why this issue was appearing.
Yes, that would do it.
Unfortunately due to the dummy interface (anycast) (any0) we had the specify the source ip for various reasons.
Ultimately I removed all the src_ipaddr directives and added an iptables SNAT rule to modify all the outbound packets for dest ports 1812/1813. So far it seems to be working, hopefully it doesn't run out of sockets this way....
It won't.
Thanks for your help Alan.
You're welcome. It's what I do... Alan DeKok.
Hello All, Upon using the packages from packages.networkradius.com (3.0.16) We are still getting the issue where the radius server will exit(1) with the last logged line being: " radiusd[13056]:Failed adding event handler for socket: Too many readers" We are no longer using "src_ipaddr" as previously mentioned... but still the same error as when we used it previously? The only thing I can think of next is weird interactions between Freeradius and dummy interfaces (/usr/sbin/ip link add any0 type dummy) producing similar behavior to "src_ipaddr" in regards to UDP sockets? Any thoughts on this? Or how I can continue to troubleshoot? Thanks Cam
On Apr 29, 2018, at 11:58 PM, Campbell McKenzie <Campbell.McKenzie@aarnet.edu.au> wrote:
Upon using the packages from packages.networkradius.com (3.0.16) We are still getting the issue where the radius server will exit(1) with the last logged line being: " radiusd[13056]:Failed adding event handler for socket: Too many readers"
We are no longer using "src_ipaddr" as previously mentioned... but still the same error as when we used it previously?
It depends on what your configuration does. If it worked before and now doesn't work, what changed? The v3.0.16 on the server is exactly the same source that would be installed from anywhere else.w
The only thing I can think of next is weird interactions between Freeradius and dummy interfaces (/usr/sbin/ip link add any0 type dummy) producing similar behavior to "src_ipaddr" in regards to UDP sockets?
FreeRADIUS doesn't care about ethernet interfaces.
Any thoughts on this? Or how I can continue to troubleshoot?
Use 3.0.17, as it has some issues fixed. And, describe what you're doing. Normal configurations don't run into this, so you're doing something unusual. Alan DeKok.
participants (2)
-
Alan DeKok -
Campbell McKenzie