Ignoring duplicate packet

[Alan DeKok]

On Oct 14, 2020, at 5:05 AM, Ammad Ali <ammad.ali at rapidcompute.com> wrote:
> I have below config in sql mod.
> ...
> I have below config in radiusd.conf

  None of that matters.

> Sometime I get these messages.
> 
> Wed Oct 14 04:30:00 2020 : Warning: rlm_sql (sql): 60 of 60 connections in use.  You probably need to increase "spare"

  The database is slow.

> I can see at DB that connections are connected from freeradius. zero connection might means that its currently no connection is being used.

  Yes.

> But below errors are continuous.
> 
> Wed Oct 14 14:00:23 2020 : Error: (34868042) Ignoring duplicate packet from client BRAS port 50044 - ID: 9 due to unfinished request in component <core> module
> Wed Oct 14 14:00:23 2020 : Error: (34868065) Ignoring duplicate packet from client BRAS port 56885 - ID: 56 due to unfinished request in component <core> module

  Yes.  Because the database is slow.  FreeRADIUS is waiting for a response from the database, and the database is taking forever to respond.

  There are still packets arriving at FreeRADIUS while the database is slow.  Eventually, FreeRADIUS can't make progress.  And the NAS retransmits.

> I have added below tweaking in sysctl.conf for TCP and UDP tweaking.

  That's nice, but it may or may not help.

> I have checked by tcpdump for 10mins. Approx 75% of auth request and 78% of accounting requests are being responded by freeradius.

  Because the database is slow, and is blocking the server.

> I have tuned mysql to its optimal level as well but didn't fixed the problem.

  It's clearly not optimal if it can't reply to FreeRADIUS in a timely manner.  

> The system have 200k connected users with 30mins of interm update.

  That should be about ~100 accounting packets per second.  Any reasonable MySQL system should be able to keep up.

  And if you configure FreeRADIUS to just "ack" all accounting packets, it can do 40K+ packets per second.  So FreeRADIUS isn't the issue.

> Need your advise how to troubleshoot this problem further as the DB and freeradius are on same node (duel 2640v4 processors + 256GB RAM + 800GB SSD)

  This isn't a FreeRADIUS issue.  It's a database issue.  We can give only limited advice for databases, because we didn't write them, and we're not responsible for them.

  The issue here is that you're stuck on the logs coming from FreeRADIUS.  Those logs are not the *cause* of the problem.  They're the *side effect* of the problem.

  You need to track down *why* the database is slow, and fix it.  Look at the slow query log for MySQL.  There's likely queries which take 5-10s to finish.

  Alan DeKok.