FR 3.0.21 : Failed adding proxy socket: Too many open sockets
Hi, We are upgrading our eduroam proxy server to FR 3.0.21. We have ~550 clients and home_server. When we start FR, it seems that the number of udp socket keep increases, we have a lot of log lines like : Info: ... adding new socket proxy address * port 50740 Until this one : Error: Failed adding proxy socket: Too many open sockets Then, some proxied requests are still working fine but other ones (new ones which need a new socket ?) are rejected : (365476) Tue Aug 18 15:05:46 2020: ERROR: proxy: Failed allocating Id for proxied request With netstat or lsof, udp sockets are opened but not so many : $ lsof -p <freeradius_pid> | grep UDP | wc -l 259 Any idea how to solve this issue ? Regards, Arnaud Lauriou
On Aug 18, 2020, at 9:19 AM, Arnaud LAURIOU <arnaud.lauriou@renater.fr> wrote:
We are upgrading our eduroam proxy server to FR 3.0.21. We have ~550 clients and home_server.
When we start FR, it seems that the number of udp socket keep increases, we have a lot of log lines like : Info: ... adding new socket proxy address * port 50740
Until this one : Error: Failed adding proxy socket: Too many open sockets
Yes, it only supports 256 proxy sockets. Are you setting a unique src_ipaddr for each home server? Because that will definitely increase the number of sockets used. There's no issue with 1000 home servers using the same socket. RADIUS allows it, and FreeRADIUS has been tested with it.
Then, some proxied requests are still working fine but other ones (new ones which need a new socket ?) are rejected : (365476) Tue Aug 18 15:05:46 2020: ERROR: proxy: Failed allocating Id for proxied request
With netstat or lsof, udp sockets are opened but not so many : $ lsof -p <freeradius_pid> | grep UDP | wc -l 259
That's still a lot of sockets.
Any idea how to solve this issue ?
a) allow home servers to share src_ipaddr, which means that they can share sockets b) edit src/lib/packet.c, and change MAX_SOCKETS from 256 to 1024, or maybe 2048. It just has to be a power of 2. Alan DeKok.
Thank's for your reply. On 8/18/20 3:25 PM, Alan DeKok wrote:
Any idea how to solve this issue ? a) allow home servers to share src_ipaddr, which means that they can share sockets Just a precision, I added the same src_ipaddr for each home_server but several udp sockets are still created : Tue Aug 18 16:53:02 2020 : Info: ... adding new socket proxy address 194.57.7.78 port 39252 Tue Aug 18 16:53:51 2020 : Info: ... adding new socket proxy address 194.57.7.78 port 43055 Tue Aug 18 16:54:48 2020 : Info: ... adding new socket proxy address 194.57.7.78 port 59876 Tue Aug 18 17:01:21 2020 : Info: ... adding new socket proxy address 194.57.7.78 port 38393
Is there something else to do to share this src_ipaddr ? Regards, Arnaud Lauriou
On Aug 18, 2020, at 11:03 AM, Arnaud LAURIOU <arnaud.lauriou@renater.fr> wrote:
Just a precision, I added the same src_ipaddr for each home_server but several udp sockets are still created : Tue Aug 18 16:53:02 2020 : Info: ... adding new socket proxy address 194.57.7.78 port 39252 Tue Aug 18 16:53:51 2020 : Info: ... adding new socket proxy address 194.57.7.78 port 43055 Tue Aug 18 16:54:48 2020 : Info: ... adding new socket proxy address 194.57.7.78 port 59876 Tue Aug 18 17:01:21 2020 : Info: ... adding new socket proxy address 194.57.7.78 port 38393
Is there something else to do to share this src_ipaddr ?
No. If you have not set src_ipaddr before, then it adding it now won't change things. I suspect that one of your home servers is down. Limitations in RADIUS mean that the server can only send 256 packets on one socket before it needs to open a new socket. So if a home server is down or slow, it might very well use 256 packets & 256 sockets, just to talk to one home server. You may have to edit src/lib/packet.c as I suggested before. But even that may not help if one of the home servers is down. Alan DeKok.
On 8/18/20 5:07 PM, Alan DeKok wrote:
I suspect that one of your home servers is down. Limitations in RADIUS mean that the server can only send 256 packets on one socket before it needs to open a new socket. So if a home server is down or slow, it might very well use 256 packets & 256 sockets, just to talk to one home server.
You may have to edit src/lib/packet.c as I suggested before. But even that may not help if one of the home servers is down.
Well, I may have found home server guilty, a broken one. I'm investigating ... But : - I don't understand why we didn't have this issue with our current FR 2.1 proxy using the same home servers list and having 500 open udp sockets. - It's difficult to prevent from all home servers down/slow or broken. Is there a way to mitigate this issue ? Close udp sockets ? It seems that once this limit is reached the current sockets are never closed and all new clients requests are rejected. Regards, Arnaud Lauriou
On Aug 19, 2020, at 5:50 AM, Arnaud LAURIOU <arnaud.lauriou@renater.fr> wrote:
Well, I may have found home server guilty, a broken one. I'm investigating ... But : - I don't understand why we didn't have this issue with our current FR 2.1 proxy using the same home servers list and having 500 open udp sockets.
v2 is different from v3.
- It's difficult to prevent from all home servers down/slow or broken.
Is there a way to mitigate this issue ?
Read proxy.conf. Look for "max_outstanding". This is documented.
Close udp sockets ? It seems that once this limit is reached the current sockets are never closed and all new clients requests are rejected.
Likely because it's still trying to proxy to the home server, as it still thinks that the home server is alive. If you fill up the proxy with bad packets, then bad things happen. Update the configuration to limit the number of packets proxied to that home server, and bad things won't happen. Alan DeKok.
On 8/19/20 1:50 PM, Alan DeKok wrote:
- It's difficult to prevent from all home servers down/slow or broken.
Is there a way to mitigate this issue ? Read proxy.conf. Look for "max_outstanding". This is documented.
Ok, thanks Alan for pointing out that counter. I have a few questions for which I haven't found answer yet : Is there good tuning practices ? Is this counter reset ? For example : - If a home_server is temporarly dead, is his value reset when it comes alive again ? - If a home_server reaches this counter limit, how long will it be unused ? Regards, Arnaud Lauriou
On Aug 20, 2020, at 10:46 AM, Arnaud LAURIOU <arnaud.lauriou@renater.fr> wrote:
I have a few questions for which I haven't found answer yet : Is there good tuning practices ?
Set it to a value which is reasonable. What "reasonable" means depends on your situation. i.e. a busy server may *normally* have thousands of packets outstanding. A small server may normally have 10 packets outstanding.
Is this counter reset ? For example : - If a home_server is temporarly dead, is his value reset when it comes alive again ?
If the home server is dead, then it has no outstanding packets. And therefore the outstanding packet counter is zero.
- If a home_server reaches this counter limit, how long will it be unused ?
Until it either responds to packets (which lowers the outstanding counter), or it is marked as dead (which sets the counter to zero). i.e. it does what makes sense. Alan DeKok.
Add to this that you should care about latency. No point queuing 1000's of extra requests if it means that the queue is too long to process... That is the time to process everything in the queue > timeout of clients / devices. (Only option if that’s the case is more capacity or optimisation wherever the bottleneck is). To get AAA tuning right the only option is test it with realistic requests and request rates and see what you get. Get into the habit of doing this for every change and you won't get surprised by sudden performance deltas associated with what on the face of it are trivial configuration updates. Also beware the hockey stick performance case found by graphing request rate vs latency. See a huge latency delta for a tiny delta in request rate and you need to be careful if you are running anywhere near the limit. Alister. On 20/08/2020, 15:55, "Freeradius-Users on behalf of Alan DeKok" <freeradius-users-bounces+alister.winfield=sky.uk@lists.freeradius.org on behalf of aland@deployingradius.com> wrote: On Aug 20, 2020, at 10:46 AM, Arnaud LAURIOU <arnaud.lauriou@renater.fr> wrote: > I have a few questions for which I haven't found answer yet : > Is there good tuning practices ? Set it to a value which is reasonable. What "reasonable" means depends on your situation. i.e. a busy server may *normally* have thousands of packets outstanding. A small server may normally have 10 packets outstanding. > Is this counter reset ? For example : > - If a home_server is temporarly dead, is his value reset when it comes alive again ? If the home server is dead, then it has no outstanding packets. And therefore the outstanding packet counter is zero. > - If a home_server reaches this counter limit, how long will it be unused ? Until it either responds to packets (which lowers the outstanding counter), or it is marked as dead (which sets the counter to zero). i.e. it does what makes sense. Alan DeKok. - List info/subscribe/unsubscribe? See https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.freeradius.org%2Flist%2Fusers.html&data=02%7C01%7Calister.winfield%40sky.uk%7C6e6a50a7ffa54150d27a08d845190583%7C68b865d5cf184b2b82a4a4eddb9c5237%7C0%7C1%7C637335321057136541&sdata=WAxelrAW%2F07e%2Fckmj3nbZIVAkwH0%2FAyJFOoG%2BLf7p14%3D&reserved=0 -------------------------------------------------------------------- This email is from an external source. Please do not open attachments or click links from an unknown or suspicious origin. Phishing attempts can be reported by using the report message button in Outlook or sending them as an attachment to phishing@sky.uk. Thank you -------------------------------------------------------------------- Information in this email including any attachments may be privileged, confidential and is intended exclusively for the addressee. The views expressed may not be official policy, but the personal views of the originator. If you have received it in error, please notify the sender by return e-mail and delete it from your system. You should not reproduce, distribute, store, retransmit, use or disclose its contents to anyone. Please note we reserve the right to monitor all e-mail communication through our internal and external networks. SKY and the SKY marks are trademarks of Sky Limited and Sky International AG and are used under licence. Sky UK Limited (Registration No. 2906991), Sky-In-Home Service Limited (Registration No. 2067075), Sky Subscribers Services Limited (Registration No. 2340150) and Sky CP Limited (Registration No. 9513259) are direct or indirect subsidiaries of Sky Limited (Registration No. 2247735). All of the companies mentioned in this paragraph are incorporated in England and Wales and share the same registered office at Grant Way, Isleworth, Middlesex TW7 5QD
participants (3)
-
Alan DeKok -
Arnaud LAURIOU -
Winfield, Alister