Ignoring duplicate packet

[Ammad Ali]

Hi Alan,

I have below config in sql mod.

         pool {
                 min = 50
                 max = ${thread[pool].max_servers}
                 spare = 10
                 uses = 0
                 lifetime = 0
                 idle_timeout = 60
         }

I have below config in radiusd.conf

max_request_time = 30
cleanup_delay = 10
max_requests = 81920
hostname_lookups = no

thread pool {
         start_servers = 64
         max_servers = 256
         min_spare_servers = 50
         max_spare_servers = 100
         max_requests_per_server = 300
         auto_limit_acct = yes
}


Sometime I get these messages.

Wed Oct 14 04:30:00 2020 : Warning: rlm_sql (sql): 60 of 60 connections 
in use.  You probably need to increase "spare"
Wed Oct 14 04:30:00 2020 : Error: rlm_sql (sql): Cannot open new 
connection, connection spawning already in progress
Wed Oct 14 04:30:00 2020 : Warning: rlm_sql (sql): 60 of 60 connections 
in use.  You probably need to increase "spare"
Wed Oct 14 04:30:00 2020 : Error: rlm_sql (sql): Cannot open new 
connection, connection spawning already in progress
Wed Oct 14 04:30:00 2020 : Warning: rlm_sql (sql): 60 of 60 connections 
in use.  You probably need to increase "spare"
Wed Oct 14 04:30:00 2020 : Error: rlm_sql (sql): Cannot open new 
connection, connection spawning already in progress
Wed Oct 14 04:30:00 2020 : Warning: rlm_sql (sql): 60 of 60 connections 
in use.  You probably need to increase "spare"
Wed Oct 14 04:30:00 2020 : Error: rlm_sql (sql): Cannot open new 
connection, connection spawning already in progress
Wed Oct 14 04:30:00 2020 : Warning: rlm_sql (sql): 61 of 61 connections 
in use.  You probably need to increase "spare"

And these

Wed Oct 14 04:29:53 2020 : Info: rlm_sql (sql): Opening additional 
connection (209059)
Wed Oct 14 04:29:54 2020 : Info: rlm_sql (sql): 0 of 44 connections in 
use.  Need more spares
Wed Oct 14 04:29:54 2020 : Info: rlm_sql (sql): Opening additional 
connection (209060)
Wed Oct 14 04:29:55 2020 : Info: rlm_sql (sql): 6 of 45 connections in 
use.  Need more spares
Wed Oct 14 04:29:55 2020 : Info: rlm_sql (sql): Opening additional 
connection (209061)
Wed Oct 14 04:29:55 2020 : Info: rlm_sql (sql): Closing connection 
(209028): Hit idle_timeout, was idle for 61 seconds
Wed Oct 14 04:29:55 2020 : Warning: rlm_sql (sql): You probably need to 
lower "min"
Wed Oct 14 04:29:55 2020 : Info: rlm_sql (sql): Closing connection 
(209030): Hit idle_timeout, was idle for 61 seconds
Wed Oct 14 04:29:55 2020 : Info: rlm_sql (sql): Closing connection 
(209022): Hit idle_timeout, was idle for 61 seconds
Wed Oct 14 04:29:55 2020 : Info: rlm_sql (sql): Closing connection 
(209023): Hit idle_timeout, was idle for 61 seconds

I can see at DB that connections are connected from freeradius. zero 
connection might means that its currently no connection is being used.

But below errors are continuous.

Wed Oct 14 14:00:23 2020 : Error: (34868042) Ignoring duplicate packet 
from client BRAS port 50044 - ID: 9 due to unfinished request in 
component <core> module
Wed Oct 14 14:00:23 2020 : Error: (34868065) Ignoring duplicate packet 
from client BRAS port 56885 - ID: 56 due to unfinished request in 
component <core> module

I have added below tweaking in sysctl.conf for TCP and UDP tweaking.

# allow testing with buffers up to 128MB
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
# increase the length of the processor input queue
net.core.netdev_max_backlog = 250000

# increase Linux autotuning TCP buffer limit to 64MB
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864

# recommended default congestion control is htcp
net.ipv4.tcp_congestion_control=htcp

net.ipv4.udp_rmem_min=131072
net.ipv4.udp_wmem_min=131072

I have checked by turning off the firewall on host.

I have checked by tcpdump for 10mins. Approx 75% of auth request and 78% 
of accounting requests are being responded by freeradius.

I have tuned mysql to its optimal level as well but didn't fixed the 
problem.

The system have 200k connected users with 30mins of interm update.

Need your advise how to troubleshoot this problem further as the DB and 
freeradius are on same node (duel 2640v4 processors + 256GB RAM + 800GB SSD)

Ammad Ali
On 10/13/2020 6:21 PM, Alan DeKok wrote:
> On Oct 13, 2020, at 9:03 AM, Ammad Ali <ammad.ali at rapidcompute.com> wrote:
>> My database is working fine. I am running 4 radius servers using same database. But surprisingly the radius process running same node with the database I am getting the below errors alot ignoring duplicate errors.
>    *Something* is blocking the RADIUS server.  And *something* is making it have *zero* connections to the database.
>
>    It's really simple.  I've been doing this for about 20 years.  I've seen those errors well over a thousand times at this point.  It's almost always something *external* blocking the RADIUS server.
>
>    So you can believe me and fix the problem.  Or, believe that your system is somehow unique, and you've run into an issue never seen before.
>
>    Alan DeKok.
>
>
> -
> List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html