Seeing dhcp packet drops in latest master

[Alan DeKok]

On Aug 13, 2020, at 9:11 AM, Chinnapaiyan, Nagamani <Nagamani.Chinnapaiyan at viasat.com> wrote:
> 
> We are seeing packet drops frequently in latest 4.x version(#8c2e4122572a12f43d4ec294d3258a2f0da07b1d). We are not doing scale test, just ~10 DORAs. If I run some 10 testcases(10 DORAs each test case), it fails randomly at any one of the tests. The same testcases were passing around one month before.

  Weird.

  We've made some changes recently for how DHCPv4 packets are processed.  But there's nothing which should block.

> seems packet is delivered to DHCP server, but radiusd is not responding and there are no other error logs in that time.
> 
> I tried my best to corner out the problem. But, could not tackle the exact use case as it is happening randomly.
> 
> I suspected these two logs,
> 
> ldap - Closing connection (3): Hit idle_timeout, was idle for 207.534324944s
> 
> redis_ippool - Closing connection (5): Hit idle_timeout, was idle for 212.217780777s
> 
> I tried tweaking that parameter from 60, to 600 then to infinite(0) in redis, redis_ippool and ldap. That didnt help.

  There are two approaches to tracking this down:

1 - checkout the previous code until you find out which revision has made the change

2 - disable various modules until you see which one is making the change

  We've been doing testing internally in v4, and seeing many hundreds of DORA per second, even to SQL databases.

  Alan DeKok.