FreeRadius DHCP Performance

[Alan DeKok]

On Jan 28, 2016, at 12:24 PM, O'Connell, Ryan <ryan.oconnell at viasat.com> wrote:
> 
> Curious if any has ballpark figures for any testing they have done with a "stand alone” FreeRadius DHCP server (3.0.10).  I have setup a host with a local instance of MySQL (FR & MySQL on same Cent7 host) as well as FR pointing to a “physical” MySQL server with a massive amount of resources and write capability via a SAN.  Irrespective of this, I seem to be topping out at 3 req/sec which seems strange.

  How is MySQL configured?  Which DB back-end are you using?  i.e. MyISAM, etc. How many IPs are in the pool?  Is the pool indexed?

>  When the server starts to fail, I start seeing errors like:
> 
> Sat Jan 23 12:19:51 2016 : ERROR: (60870) dhcp_sqlippool: ERROR: rlm_sql_mysql: ERROR 1205 (Lock wait timeout exceeded; try restarting transaction): HY000
> Sat Jan 23 12:19:51 2016 : WARNING: (60870) WARNING: Module rlm_sqlippool became unblocked for request 60870

  Because the SQL server is taking too long to respond.

> Consistent with the errors in the radius log, the MySQL log has errors similar to this.
> 
> 2016-01-23 12:19:52 7270 [ERROR] /usr/sbin/mysqld: Sort aborted: Lock wait timeout exceeded; try restarting transaction
> 2016-01-23 12:19:52 7270 [ERROR] /usr/sbin/mysqld: Lock wait timeout exceeded; try restarting transaction

  Which probably means that the query is locking the entire table, and later queries get delayed / blocked.

> I’ve tried adjusting the number of servers up and down from tens to hundreds (I know this isn’t a solution per se but just commenting that this has been done) while also adjusting my.cnf and sysctl (open files / processes) but the behavior is the same.    I have also tried moving the MySQL data to a ramdisk since this is just a test but irrespective of whether a high I/) SAN or ramdisk is used, the performance is the same.  Has anyone achieved better performance than this?  I found comments from the past talking about the mac2ip performance being really high but I can’t seem to figure out what I’m missing to make dhcp_sqlippool go faster.  I’m using “dhcperf” as the test but have also tried other simulators.  Here’s the server startup.  There is no selection logic so just a single pool.

  The issue is in MySQL.  For whatever reason, it can't handle the load.  How to fix that is largely up to MySQL...

  We've tested the v3.1.x HEAD with Redis as an IP pool back-end.  It's benchmarked at 10K IPs per second per redis node.  i.e. a cluster with 4 nodes can do 40K IP assignments per second.

  I'm not aware of many *commercial* DHCP servers which can do that.

  Alan DeKok.