Re: Proxy issue - Failing proxied request for user error
Found it. I checked `cat /proc/net/udp` and could see some packets being dropped in the UDP buffer queue; and whilst monitoring the dropped packets we're seeing from the clients, the numbers were going up in line with this. So I increased the `net.core.rmem` and `net.core.wmem` values a little on the server via sysctl, to increase the default UDP buffer size which has stopped the timeouts from occurring. I guess out the box the server has am appropriate value for the UDP buffer queue but when using FreeRADIUS it fills it up if there's enough traffic (and its all UDP) and then the kernel starts dropping packets. Whist I appreciate it's not FreeRADIUS related, is there any advice around the best sysctl values to use for a large FreeRADIUS server? Has anyone come across this before? Thanks J
Sigh basic network tuning on Linux. Yes and on high rate systems I've even managed to wrap the default NIC's ring-buffer mostly due to the kernel not processing enough per-interrupt. (Took a while to work out how you can get malformed packets in userland that definitely should have failed the checksums). On how big.... that depends. If what you are really seeing is short bursts of high rates and the server can easily keep up feel free to make the buffer HUGE. Latency might go up when the burst arrives but assuming it's all processed before timeouts start occurring you are good. If you are really starved for resource then increasing the buffers is bad news and might mean you process nothing before timeout essentially creating a total service failure rather than handling some percentage of requests. (This is especially true for DHCP, so much so that random drops to keep the queue short actually works better than answering everything when overloaded). Alister On 18/07/2019, 06:52, "Freeradius-Users on behalf of James Wood" <freeradius-users-bounces+alister.winfield=sky.uk@lists.freeradius.org on behalf of james.wood@purplewifi.com> wrote: Found it. I checked `cat /proc/net/udp` and could see some packets being dropped in the UDP buffer queue; and whilst monitoring the dropped packets we're seeing from the clients, the numbers were going up in line with this. So I increased the `net.core.rmem` and `net.core.wmem` values a little on the server via sysctl, to increase the default UDP buffer size which has stopped the timeouts from occurring. I guess out the box the server has am appropriate value for the UDP buffer queue but when using FreeRADIUS it fills it up if there's enough traffic (and its all UDP) and then the kernel starts dropping packets. Whist I appreciate it's not FreeRADIUS related, is there any advice around the best sysctl values to use for a large FreeRADIUS server? Has anyone come across this before? Thanks J - List info/subscribe/unsubscribe? See https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.freeradius.org%2Flist%2Fusers.html&data=02%7C01%7Calister.winfield%40sky.uk%7C671716d463054c78e42a08d70b442e11%7C68b865d5cf184b2b82a4a4eddb9c5237%7C0%7C0%7C636990259784198618&sdata=caG8ueU%2FpTo1CqI9d%2BJwyPKd%2BLqobLyXIGgW17ZxO38%3D&reserved=0 -------------------------------------------------------------------- This email is from an external source. Please do not open attachments or click links from an unknown or suspicious origin. Phishing attempts can be reported by sending them to phishing@sky.uk as attachments. Thank you -------------------------------------------------------------------- Information in this email including any attachments may be privileged, confidential and is intended exclusively for the addressee. The views expressed may not be official policy, but the personal views of the originator. If you have received it in error, please notify the sender by return e-mail and delete it from your system. You should not reproduce, distribute, store, retransmit, use or disclose its contents to anyone. Please note we reserve the right to monitor all e-mail communication through our internal and external networks. SKY and the SKY marks are trademarks of Sky Limited and Sky International AG and are used under licence. Sky UK Limited (Registration No. 2906991), Sky-In-Home Service Limited (Registration No. 2067075), Sky Subscribers Services Limited (Registration No. 2340150) and Sky CP Limited (Registration No. 9513259) are direct or indirect subsidiaries of Sky Limited (Registration No. 2247735). All of the companies mentioned in this paragraph are incorporated in England and Wales and share the same registered office at Grant Way, Isleworth, Middlesex TW7 5QD
On Jul 18, 2019, at 4:21 AM, Winfield, Alister via Freeradius-Users <freeradius-users@lists.freeradius.org> wrote:
If what you are really seeing is short bursts of high rates and the server can easily keep up feel free to make the buffer HUGE. Latency might go up when the burst arrives but assuming it's all processed before timeouts start occurring you are good. If you are really starved for resource then increasing the buffers is bad news and might mean you process nothing before timeout essentially creating a total service failure rather than handling some percentage of requests. (This is especially true for DHCP, so much so that random drops to keep the queue short actually works better than answering everything when overloaded).
Agreed. As a side note, we're using v4 for DHCP with Redis Cluster. An *out of the box* configuration gets 8K DHCP DORAs per second. With a bit of tuning and more CPU, we're getting higher rates than that. The new ISC DHCP server gets less than that with its *in-memory* database! When SQL is used, ISC is getting ~500/s, where we're getting sustained 2K+ to PostgreSQL. I'm not aware of any other DHCP server with this level of performance. :) Alan DeKok.
On Jul 18, 2019, at 1:52 AM, James Wood <james.wood@purplewifi.com> wrote:
I checked `cat /proc/net/udp` and could see some packets being dropped in the UDP buffer queue; and whilst monitoring the dropped packets we're seeing from the clients, the numbers were going up in line with this.
Ah, yes. That issue is cropping up more and more lately. It looks like networks are getting "burstier", which causes issues.
So I increased the `net.core.rmem` and `net.core.wmem` values a little on the server via sysctl, to increase the default UDP buffer size which has stopped the timeouts from occurring.
I guess out the box the server has am appropriate value for the UDP buffer queue but when using FreeRADIUS it fills it up if there's enough traffic (and its all UDP) and then the kernel starts dropping packets.
IIRC, the default size in Linux is 64K. Which should be enough for hundreds of RADIUS packets. If the server is processing packets normally, it should be able to process 1000's of packets per second.
Whist I appreciate it's not FreeRADIUS related, is there any advice around the best sysctl values to use for a large FreeRADIUS server? Has anyone come across this before?
It's becoming more common. It's not really clear why. You can also change this setting in FreeRADIUS. At least in recent releases. See "recv_buff" setting in sites-enabled/default: # # Set the default UDP receive buffer size. In most cases, # the default values set by the kernel are fine. However, in # some cases the NASes will send large packets, and many of # them at a time. It is then possible to overflow the # buffer, causing the kernel to drop packets before they # reach FreeRADIUS. Increasing the size of the buffer will # avoid these packet drops. # # recv_buff = 65536 Alan DeKok.
Hi Alan Thanks for the info about recv_buff What exactly does this do? Does it need to be set to allow more UDP requests and thus prevent dropped packets, even if I've increased the kernel udp values as described? Is 65536 the default even though its commented out by default? What if I set it to 0, is that no limit?
Is there a way in FreeRADIUS to tell if the buffers are full? radmin or similar? Thank you J
On Jul 21, 2019, at 4:53 AM, James Wood <james.wood@purplewifi.com> wrote:
Thanks for the info about recv_buff
What exactly does this do?
It sets the receive buffer on a per-application basis.
Does it need to be set to allow more UDP requests and thus prevent dropped packets, even if I've increased the kernel udp values as described?
It sets the same thing, but on a per application basis.
Is 65536 the default even though its commented out by default?
What if I set it to 0, is that no limit?
For both, see your OS documentation. FreeRADIUS is just setting a value in the kernel. FreeRADIUS doesn't define what those values do.
Is there a way in FreeRADIUS to tell if the buffers are full? radmin or similar?
No. There is no dependable way to query how much of the buffer is used. Linux exports /proc/net/udp with some statistics. But to read that, the application has to parse large amounts of text. Other operating systems don't even have that. Since RAM is *much* cheaper than human salaries, your best bet is to set the recv_buff to a large value, and leave it at that. Alan DeKok.
participants (3)
-
Alan DeKok -
James Wood -
Winfield, Alister