Pre-release of Version 2.1.8

[Josip Rodin]

On Tue, Dec 08, 2009 at 03:43:14PM +0100, Alan DeKok wrote:
> Garber, Neal wrote:
> >> This limit is around 8K packets in 2.1.x, and will be 64K packets in
> >> 2.2.x.  So if you're getting 500 packets/s for a home server, 16s after
> >> it goes down, all 8k "slots" will be used.
> 
>   In 2.1.x, there are a limited number of UDP sockets that can be used
> for proxied traffic.  This number is limited to 32, in src/lib/packet.c,
>  macro MAX_SOCKETS.
> 
>   Due to RADIUS limitations, it can only have 256 packets outstanding
> for any combination of (src/ds IP/port).  So each socket can send 256
> packets to every home server.
> 
>   Packets are added to the "outstanding" list when proxied, and removed
> a short time after a response is received from the home server.  If no
> response is received from the home server, the packets are removed 30s
> after they were received.
> 
>   Once a packet has been removed from the "outstanding" list, its place
> can be used by a new packet that is proxied using the same socket/id to
> the same home server.

Which reminds me - the other day I had a situation where a NAS was rebooted
and ~300 users immediately tried to reconnect and authenticated over a
FreeRADIUS 2.0.4 server, which in turn tried to authenticate them over its
two home_servers set up as fail-over, but neither of them with status_check.

Sadly, this started failing horribly - it seemed to overload the primary
home_server, entering a peculiar pattern - condensed for readability and
some private info obfuscation:

      1  Auth: Login OK: 
     10  Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812
      1  Auth: Login OK: 
      2  Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812
      1  Auth: Login incorrect (Home Server says so): 
      4  Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812
      1  Auth: Login OK: 
      4  Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812
      1  Auth: Login OK: 
      2  Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812
      2  Auth: Login OK: 
      2  Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812
      1  Auth: Login OK: 
     11  Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812
      1  Auth: Login OK: 
     86  Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812
      1  Proxy: No outstanding request was found for proxy reply from home server home_server_ip_5 port 1812 - ID 87
     19  Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812
      1  Proxy: No outstanding request was found for proxy reply from home server home_server_ip_5 port 1812 - ID 252
      7  Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812
      1  Error: Received Access-Accept packet from client home_server_ip_5 port 1812 with invalid signature (err=2)!  (Shared secret is incorrect.) Dropping packet without response.
      5  Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812
      1  Error: Received Access-Accept packet from client home_server_ip_5 port 1812 with invalid signature (err=2)!  (Shared secret is incorrect.) Dropping packet without response.
     33  Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812
      1  Proxy: No outstanding request was found for proxy reply from home server home_server_ip_5 port 1812 - ID 61
      7  Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812
      1  Proxy: No outstanding request was found for proxy reply from home server home_server_ip_5 port 1812 - ID 120
      4  Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812
      1  Proxy: No outstanding request was found for proxy reply from home server home_server_ip_5 port 1812 - ID 112
      6  Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812
      1  Error: Received Access-Accept packet from client home_server_ip_5 port 1812 with invalid signature (err=2)!  (Shared secret is incorrect.) Dropping packet without response.
      2  Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812
      1  Error: Received Access-Accept packet from client home_server_ip_5 port 1812 with invalid signature (err=2)!  (Shared secret is incorrect.) Dropping packet without response.
     20  Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812
      1  Proxy: No outstanding request was found for proxy reply from home server home_server_ip_5 port 1812 - ID 161
     12  Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812
      1  Error: Received Access-Accept packet from client home_server_ip_5 port 1812 with invalid signature (err=2)!  (Shared secret is incorrect.) Dropping packet without response.
      4  Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812
      1  Proxy: No outstanding request was found for proxy reply from home server home_server_ip_5 port 1812 - ID 243
     11  Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812
      1  Proxy: No outstanding request was found for proxy reply from home server home_server_ip_5 port 1812 - ID 203
      1  Proxy: No outstanding request was found for proxy reply from home server home_server_ip_5 port 1812 - ID 176
      1  Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812
      1  Proxy: No outstanding request was found for proxy reply from home server home_server_ip_5 port 1812 - ID 6
     10  Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812
      1  Error: Received Access-Accept packet from client home_server_ip_5 port 1812 with invalid signature (err=2)!  (Shared secret is incorrect.) Dropping packet without response.
      3  Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812
      1  Proxy: No outstanding request was found for proxy reply from home server home_server_ip_5 port 1812 - ID 248
      2  Error: Rejecting request <number> due to lack of any response from home server home_server_ip_5 port 1812
      1  Proxy: No outstanding request was found for proxy reply from home server home_server_ip_5 port 1812 - ID 69
[...]

This went on for a while, before I was alerted about the outage, came in and
started checking what is going on - using radtest showed that the primary
home_server was indeed ignoring my requests, but the secondary one was
replying as if there was nothing going on.

So I tried the poor man's solution - I shuffled them manually, restarted
FreeRADIUS, and then it started authenticating them, before it seemingly
DoSed that one and entered a very similar pattern of brokenness. I did the
manual shuffle a few more times, by which time most users were connected,
and the problem worked itself out.

Can any conclusions be drawn from this? I send over the detailed logs if
necessary.

-- 
     2. That which causes joy or happiness.