On Tue, Dec 08, 2009 at 03:43:14PM +0100, Alan DeKok wrote:
Garber, Neal wrote:
This limit is around 8K packets in 2.1.x, and will be 64K packets in 2.2.x. So if you're getting 500 packets/s for a home server, 16s after it goes down, all 8k "slots" will be used.
In 2.1.x, there are a limited number of UDP sockets that can be used for proxied traffic. This number is limited to 32, in src/lib/packet.c, macro MAX_SOCKETS.
Due to RADIUS limitations, it can only have 256 packets outstanding for any combination of (src/ds IP/port). So each socket can send 256 packets to every home server.
Packets are added to the "outstanding" list when proxied, and removed a short time after a response is received from the home server. If no response is received from the home server, the packets are removed 30s after they were received.
Once a packet has been removed from the "outstanding" list, its place can be used by a new packet that is proxied using the same socket/id to the same home server.
Which reminds me - the other day I had a situation where a NAS was rebooted and ~300 users immediately tried to reconnect and authenticated over a FreeRADIUS 2.0.4 server, which in turn tried to authenticate them over its two home_servers set up as fail-over, but neither of them with status_check. Sadly, this started failing horribly - it seemed to overload the primary home_server, entering a peculiar pattern - condensed for readability and some private info obfuscation: 1 Auth: Login OK: 10 Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812 1 Auth: Login OK: 2 Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812 1 Auth: Login incorrect (Home Server says so): 4 Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812 1 Auth: Login OK: 4 Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812 1 Auth: Login OK: 2 Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812 2 Auth: Login OK: 2 Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812 1 Auth: Login OK: 11 Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812 1 Auth: Login OK: 86 Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812 1 Proxy: No outstanding request was found for proxy reply from home server home_server_ip_5 port 1812 - ID 87 19 Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812 1 Proxy: No outstanding request was found for proxy reply from home server home_server_ip_5 port 1812 - ID 252 7 Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812 1 Error: Received Access-Accept packet from client home_server_ip_5 port 1812 with invalid signature (err=2)! (Shared secret is incorrect.) Dropping packet without response. 5 Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812 1 Error: Received Access-Accept packet from client home_server_ip_5 port 1812 with invalid signature (err=2)! (Shared secret is incorrect.) Dropping packet without response. 33 Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812 1 Proxy: No outstanding request was found for proxy reply from home server home_server_ip_5 port 1812 - ID 61 7 Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812 1 Proxy: No outstanding request was found for proxy reply from home server home_server_ip_5 port 1812 - ID 120 4 Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812 1 Proxy: No outstanding request was found for proxy reply from home server home_server_ip_5 port 1812 - ID 112 6 Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812 1 Error: Received Access-Accept packet from client home_server_ip_5 port 1812 with invalid signature (err=2)! (Shared secret is incorrect.) Dropping packet without response. 2 Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812 1 Error: Received Access-Accept packet from client home_server_ip_5 port 1812 with invalid signature (err=2)! (Shared secret is incorrect.) Dropping packet without response. 20 Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812 1 Proxy: No outstanding request was found for proxy reply from home server home_server_ip_5 port 1812 - ID 161 12 Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812 1 Error: Received Access-Accept packet from client home_server_ip_5 port 1812 with invalid signature (err=2)! (Shared secret is incorrect.) Dropping packet without response. 4 Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812 1 Proxy: No outstanding request was found for proxy reply from home server home_server_ip_5 port 1812 - ID 243 11 Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812 1 Proxy: No outstanding request was found for proxy reply from home server home_server_ip_5 port 1812 - ID 203 1 Proxy: No outstanding request was found for proxy reply from home server home_server_ip_5 port 1812 - ID 176 1 Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812 1 Proxy: No outstanding request was found for proxy reply from home server home_server_ip_5 port 1812 - ID 6 10 Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812 1 Error: Received Access-Accept packet from client home_server_ip_5 port 1812 with invalid signature (err=2)! (Shared secret is incorrect.) Dropping packet without response. 3 Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812 1 Proxy: No outstanding request was found for proxy reply from home server home_server_ip_5 port 1812 - ID 248 2 Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812 1 Proxy: No outstanding request was found for proxy reply from home server home_server_ip_5 port 1812 - ID 69 [...] This went on for a while, before I was alerted about the outage, came in and started checking what is going on - using radtest showed that the primary home_server was indeed ignoring my requests, but the secondary one was replying as if there was nothing going on. So I tried the poor man's solution - I shuffled them manually, restarted FreeRADIUS, and then it started authenticating them, before it seemingly DoSed that one and entered a very similar pattern of brokenness. I did the manual shuffle a few more times, by which time most users were connected, and the problem worked itself out. Can any conclusions be drawn from this? I send over the detailed logs if necessary. -- 2. That which causes joy or happiness.