FreeRadius DHCP Performance
Curious if any has ballpark figures for any testing they have done with a "stand alone” FreeRadius DHCP server (3.0.10). I have setup a host with a local instance of MySQL (FR & MySQL on same Cent7 host) as well as FR pointing to a “physical” MySQL server with a massive amount of resources and write capability via a SAN. Irrespective of this, I seem to be topping out at 3 req/sec which seems strange. When the server starts to fail, I start seeing errors like: Sat Jan 23 12:19:51 2016 : ERROR: (60870) dhcp_sqlippool: ERROR: rlm_sql_mysql: ERROR 1205 (Lock wait timeout exceeded; try restarting transaction): HY000 Sat Jan 23 12:19:51 2016 : WARNING: (60870) WARNING: Module rlm_sqlippool became unblocked for request 60870 Consistent with the errors in the radius log, the MySQL log has errors similar to this. 2016-01-23 12:19:52 7270 [ERROR] /usr/sbin/mysqld: Sort aborted: Lock wait timeout exceeded; try restarting transaction 2016-01-23 12:19:52 7270 [ERROR] /usr/sbin/mysqld: Lock wait timeout exceeded; try restarting transaction I’ve tried adjusting the number of servers up and down from tens to hundreds (I know this isn’t a solution per se but just commenting that this has been done) while also adjusting my.cnf and sysctl (open files / processes) but the behavior is the same. I have also tried moving the MySQL data to a ramdisk since this is just a test but irrespective of whether a high I/) SAN or ramdisk is used, the performance is the same. Has anyone achieved better performance than this? I found comments from the past talking about the mac2ip performance being really high but I can’t seem to figure out what I’m missing to make dhcp_sqlippool go faster. I’m using “dhcperf” as the test but have also tried other simulators. Here’s the server startup. There is no selection logic so just a single pool. update control { &Pool-Name := “test" } [root@dhcp01 raddb]# /usr/sbin/radiusd -X Copyright (C) 1999-2015 The FreeRADIUS server project and contributors There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE You may redistribute copies of FreeRADIUS under the terms of the GNU General Public License For more information about these matters, see the file named COPYRIGHT Starting - reading configuration files ... including dictionary file /usr/share/freeradius/dictionary including dictionary file /usr/share/freeradius/dictionary.dhcp including dictionary file /usr/share/freeradius/dictionary.vqp including dictionary file /etc/raddb/dictionary including configuration file /etc/raddb/radiusd.conf including configuration file /etc/raddb/clients.conf including files in directory /etc/raddb/mods-enabled/ including configuration file /etc/raddb/mods-enabled/always including configuration file /etc/raddb/mods-enabled/dhcp including configuration file /etc/raddb/mods-enabled/dhcp_sqlippool including configuration file /etc/raddb/mods-config/sql/ippool-dhcp/mysql/queries.conf including configuration file /etc/raddb/mods-enabled/sql including configuration file /etc/raddb/mods-config/sql/main/mysql/queries.conf including files in directory /etc/raddb/policy.d/ including configuration file /etc/raddb/policy.d/control including configuration file /etc/raddb/policy.d/debug including configuration file /etc/raddb/policy.d/filter including configuration file /etc/raddb/policy.d/dhcp including files in directory /etc/raddb/sites-enabled/ including configuration file /etc/raddb/sites-enabled/dhcp main { security { user = "radiusd" group = "radiusd" allow_core_dumps = no } name = "radiusd" prefix = "/usr" localstatedir = "/var" logdir = "/var/log/radius" run_dir = "/var/run/radiusd" } main { name = "radiusd" prefix = "/usr" localstatedir = "/var" sbindir = "/usr/sbin" logdir = "/var/log/radius" run_dir = "/var/run/radiusd" libdir = "/usr/lib64/freeradius" radacctdir = "/var/log/radius/radacct" hostname_lookups = no max_request_time = 30 cleanup_delay = 3 max_requests = 100000 pidfile = "/var/run/radiusd/radiusd.pid" checkrad = "/usr/sbin/checkrad" debug_level = 0 proxy_requests = yes log { stripped_names = no auth = no auth_badpass = no auth_goodpass = no colourise = yes msg_denied = "You are already logged in - access denied" } resources { } security { max_attributes = 200 reject_delay = 1.000000 status_server = yes allow_vulnerable_openssl = "no" } } radiusd: #### Loading Realms and Home Servers #### radiusd: #### Loading Clients #### client localhost { ipaddr = 127.0.0.1 require_message_authenticator = no secret = <<< secret >>> nas_type = "other" proto = "*" limit { max_connections = 16 lifetime = 0 idle_timeout = 30 } } client localhost_ipv6 { ipv6addr = ::1 require_message_authenticator = no secret = <<< secret >>> limit { max_connections = 16 lifetime = 0 idle_timeout = 30 } } Debugger not attached radiusd: #### Instantiating modules #### # Loaded module rlm_always # Loading module "reject" from file /etc/raddb/mods-enabled/always always reject { rcode = "reject" simulcount = 0 mpp = no } # Loading module "fail" from file /etc/raddb/mods-enabled/always always fail { rcode = "fail" simulcount = 0 mpp = no } # Loading module "ok" from file /etc/raddb/mods-enabled/always always ok { rcode = "ok" simulcount = 0 mpp = no } # Loading module "handled" from file /etc/raddb/mods-enabled/always always handled { rcode = "handled" simulcount = 0 mpp = no } # Loading module "invalid" from file /etc/raddb/mods-enabled/always always invalid { rcode = "invalid" simulcount = 0 mpp = no } # Loading module "userlock" from file /etc/raddb/mods-enabled/always always userlock { rcode = "userlock" simulcount = 0 mpp = no } # Loading module "notfound" from file /etc/raddb/mods-enabled/always always notfound { rcode = "notfound" simulcount = 0 mpp = no } # Loading module "noop" from file /etc/raddb/mods-enabled/always always noop { rcode = "noop" simulcount = 0 mpp = no } # Loading module "updated" from file /etc/raddb/mods-enabled/always always updated { rcode = "updated" simulcount = 0 mpp = no } # Loaded module rlm_dhcp # Loading module "dhcp" from file /etc/raddb/mods-enabled/dhcp # Loaded module rlm_sqlippool # Loading module "dhcp_sqlippool" from file /etc/raddb/mods-enabled/dhcp_sqlippool sqlippool dhcp_sqlippool { sql_module_instance = "sql" lease_duration = 7200 pool_name = "" default_pool = "main_pool" allocate_begin = "START TRANSACTION" allocate_clear = "UPDATE radippool SET nasipaddress = '', pool_key = 0, callingstationid = '', username = '', expiry_time = NOW() WHERE expiry_time <= NOW() - INTERVAL 1 SECOND" allocate_find = "SELECT framedipaddress FROM radippool WHERE pool_name = '%{control:Pool-Name}' AND (callingstationid = '%{Calling-Station-Id}' or callingstationid = '') ORDER BY (callingstationid <> '%{Calling-Station-Id}'), expiry_time LIMIT 1 FOR UPDATE" allocate_update = "UPDATE radippool SET nasipaddress = '%{NAS-IP-Address}', pool_key = '%{Calling-Station-Id}', callingstationid = '%{Calling-Station-Id}', username = '%{User-Name}', expiry_time = NOW() + INTERVAL 7200 SECOND WHERE framedipaddress = '%I'" allocate_commit = "COMMIT" pool_check = "" start_begin = "START TRANSACTION" start_update = "UPDATE radippool SET expiry_time = NOW() + INTERVAL 7200 SECOND WHERE nasipaddress = '%{NAS-IP-Address}' AND pool_key = '%{Calling-Station-Id}' AND username = '%{User-Name}' AND callingstationid = '%{Calling-Station-Id}' AND framedipaddress = '%{Framed-IP-Address}'" start_commit = "COMMIT" alive_begin = "START TRANSACTION" alive_update = "UPDATE radippool SET expiry_time = NOW() + INTERVAL 7200 SECOND WHERE nasipaddress = '%{Nas-IP-Address}' AND pool_key = '%{Calling-Station-Id}' AND username = '%{User-Name}' AND callingstationid = '%{Calling-Station-Id}' AND framedipaddress = '%{Framed-IP-Address}'" alive_commit = "COMMIT" stop_begin = "START TRANSACTION" stop_clear = "UPDATE radippool SET nasipaddress = '', pool_key = 0, callingstationid = '', username = '', expiry_time = NOW() WHERE nasipaddress = '%{Nas-IP-Address}' AND pool_key = '%{Calling-Station-Id}' AND username = '%{User-Name}' AND callingstationid = '%{Calling-Station-Id}' AND framedipaddress = '%{Framed-IP-Address}'" stop_commit = "COMMIT" on_begin = "START TRANSACTION" on_clear = "UPDATE radippool SET nasipaddress = '', pool_key = 0, callingstationid = '', username = '', expiry_time = NOW() WHERE nasipaddress = '%{Nas-IP-Address}'" on_commit = "COMMIT" off_begin = "START TRANSACTION" off_clear = "UPDATE radippool SET nasipaddress = '', pool_key = 0, callingstationid = '', username = '', expiry_time = NOW() WHERE nasipaddress = '%{Nas-IP-Address}'" off_commit = "COMMIT" messages { } } # Loaded module rlm_sql # Loading module "sql" from file /etc/raddb/mods-enabled/sql sql { driver = "rlm_sql_mysql" server = "localhost" port = 3306 login = "radius" password = <<< secret >>> radius_db = "radius" read_groups = yes read_profiles = yes read_clients = no delete_stale_sessions = yes sql_user_name = "%{%{Stripped-User-Name}:-%{%{User-Name}:-DEFAULT}}" default_user_profile = "" client_query = "SELECT id, nasname, shortname, type, secret, server FROM nas" authorize_check_query = "SELECT id, username, attribute, value, op FROM radcheck WHERE username = '%{SQL-User-Name}' ORDER BY id" authorize_reply_query = "SELECT id, username, attribute, value, op FROM radreply WHERE username = '%{SQL-User-Name}' ORDER BY id" authorize_group_check_query = "SELECT id, groupname, attribute, Value, op FROM radgroupcheck WHERE groupname = '%{SQL-Group}' ORDER BY id" authorize_group_reply_query = "SELECT id, groupname, attribute, value, op FROM radgroupreply WHERE groupname = '%{SQL-Group}' ORDER BY id" group_membership_query = "SELECT groupname FROM radusergroup WHERE username = '%{SQL-User-Name}' OR (NOT EXISTS (select 1 from radreply where username='%{SQL-User-Name}') AND username='DEFAULT-SF') ORDER BY priority" simul_verify_query = "SELECT radacctid, acctsessionid, username, nasipaddress, nasportid, framedipaddress, callingstationid, framedprotocol FROM radacct WHERE username = '%{SQL-User-Name}' AND acctstoptime IS NULL" safe_characters = "@abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789.-_: /" accounting { reference = ".query" type { accounting-on { query = "UPDATE radacct SET acctstoptime = FROM_UNIXTIME(%{integer:Event-Timestamp}), acctsessiontime = '%{integer:Event-Timestamp}' - UNIX_TIMESTAMP(acctstarttime), acctterminatecause = '%{%{Acct-Terminate-Cause}:-NAS-Reboot}' WHERE acctstoptime IS NULL AND nasipaddress = '%{NAS-IP-Address}' AND acctstarttime <= FROM_UNIXTIME(%{integer:Event-Timestamp})" } accounting-off { query = "UPDATE radacct SET acctstoptime = FROM_UNIXTIME(%{integer:Event-Timestamp}), acctsessiontime = '%{integer:Event-Timestamp}' - UNIX_TIMESTAMP(acctstarttime), acctterminatecause = '%{%{Acct-Terminate-Cause}:-NAS-Reboot}' WHERE acctstoptime IS NULL AND nasipaddress = '%{NAS-IP-Address}' AND acctstarttime <= FROM_UNIXTIME(%{integer:Event-Timestamp})" } start { query = "INSERT INTO radacct (acctsessionid, acctuniqueid, username, realm, nasipaddress, nasportid, nasporttype, acctstarttime, acctupdatetime, acctstoptime, acctsessiontime, acctauthentic, connectinfo_start, connectinfo_stop, acctinputoctets, acctoutputoctets, calledstationid, callingstationid, acctterminatecause, servicetype, framedprotocol, framedipaddress) VALUES ('%{Acct-Session-Id}', '%{Acct-Unique-Session-Id}', '%{SQL-User-Name}', '%{Realm}', '%{NAS-IP-Address}', '%{%{NAS-Port-ID}:-%{NAS-Port}}', '%{NAS-Port-Type}', FROM_UNIXTIME(%{integer:Event-Timestamp}), FROM_UNIXTIME(%{integer:Event-Timestamp}), NULL, '0', '%{Acct-Authentic}', '%{Connect-Info}', '', '0', '0', '%{Called-Station-Id}', '%{Calling-Station-Id}', '', '%{Service-Type}', '%{Framed-Protocol}', '%{Framed-IP-Address}')" } interim-update { query = "UPDATE radacct SET acctupdatetime = (@acctupdatetime_old:=acctupdatetime), acctupdatetime = FROM_UNIXTIME(%{integer:Event-Timestamp}), acctinterval = %{integer:Event-Timestamp} - UNIX_TIMESTAMP(@acctupdatetime_old), framedipaddress = '%{Framed-IP-Address}', acctsessiontime = %{%{Acct-Session-Time}:-NULL}, acctinputoctets = '%{%{Acct-Input-Gigawords}:-0}' << 32 | '%{%{Acct-Input-Octets}:-0}', acctoutputoctets = '%{%{Acct-Output-Gigawords}:-0}' << 32 | '%{%{Acct-Output-Octets}:-0}' WHERE AcctUniqueId = '%{Acct-Unique-Session-Id}' AND username = '%{SQL-User-Name}' AND nasipaddress = '%{NAS-IP-Address}'" } stop { query = "UPDATE radacct SET acctstoptime = FROM_UNIXTIME(%{integer:Event-Timestamp}), acctsessiontime = %{%{Acct-Session-Time}:-NULL}, acctinputoctets = '%{%{Acct-Input-Gigawords}:-0}' << 32 | '%{%{Acct-Input-Octets}:-0}', acctoutputoctets = '%{%{Acct-Output-Gigawords}:-0}' << 32 | '%{%{Acct-Output-Octets}:-0}', acctterminatecause = '%{Acct-Terminate-Cause}', connectinfo_stop = '%{Connect-Info}' WHERE AcctUniqueId = '%{Acct-Unique-Session-Id}'" } } } post-auth { reference = ".query" query = "INSERT INTO radpostauth (username, pass, reply, authdate) VALUES ( '%{SQL-User-Name}', '%{%{User-Password}:-%{Chap-Password}}', '%{reply:Packet-Type}', '%S')" } } rlm_sql (sql): Driver rlm_sql_mysql (module rlm_sql_mysql) loaded and linked Creating attribute SQL-Group instantiate { } modules { # Instantiating module "reject" from file /etc/raddb/mods-enabled/always # Instantiating module "fail" from file /etc/raddb/mods-enabled/always # Instantiating module "ok" from file /etc/raddb/mods-enabled/always # Instantiating module "handled" from file /etc/raddb/mods-enabled/always # Instantiating module "invalid" from file /etc/raddb/mods-enabled/always # Instantiating module "userlock" from file /etc/raddb/mods-enabled/always # Instantiating module "notfound" from file /etc/raddb/mods-enabled/always # Instantiating module "noop" from file /etc/raddb/mods-enabled/always # Instantiating module "updated" from file /etc/raddb/mods-enabled/always # Instantiating module "dhcp_sqlippool" from file /etc/raddb/mods-enabled/dhcp_sqlippool # Instantiating module "sql" from file /etc/raddb/mods-enabled/sql rlm_sql_mysql: libmysql version: 5.6.27 mysql { tls { } warnings = "auto" } rlm_sql (sql): Attempting to connect to database "radius" rlm_sql (sql): Initialising connection pool pool { start = 50 min = 15 max = 64 spare = 35 uses = 0 lifetime = 0 cleanup_interval = 30 idle_timeout = 60 retry_delay = 30 spread = no } rlm_sql (sql): Opening additional connection (0), 1 of 64 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (1), 1 of 63 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (2), 1 of 62 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (3), 1 of 61 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (4), 1 of 60 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (5), 1 of 59 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (6), 1 of 58 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (7), 1 of 57 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (8), 1 of 56 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (9), 1 of 55 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (10), 1 of 54 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (11), 1 of 53 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (12), 1 of 52 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (13), 1 of 51 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (14), 1 of 50 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (15), 1 of 49 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (16), 1 of 48 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (17), 1 of 47 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (18), 1 of 46 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (19), 1 of 45 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (20), 1 of 44 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (21), 1 of 43 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (22), 1 of 42 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (23), 1 of 41 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (24), 1 of 40 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (25), 1 of 39 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (26), 1 of 38 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (27), 1 of 37 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (28), 1 of 36 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (29), 1 of 35 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (30), 1 of 34 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (31), 1 of 33 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (32), 1 of 32 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (33), 1 of 31 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (34), 1 of 30 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (35), 1 of 29 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (36), 1 of 28 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (37), 1 of 27 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (38), 1 of 26 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (39), 1 of 25 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (40), 1 of 24 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (41), 1 of 23 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (42), 1 of 22 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (43), 1 of 21 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (44), 1 of 20 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (45), 1 of 19 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (46), 1 of 18 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (47), 1 of 17 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (48), 1 of 16 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 rlm_sql (sql): Opening additional connection (49), 1 of 15 pending slots used rlm_sql_mysql: Starting connect to MySQL server rlm_sql_mysql: Connected to database 'radius' on Localhost via UNIX socket, server version 5.6.27, protocol version 10 } # modules radiusd: #### Loading Virtual Servers #### server { # from file /etc/raddb/radiusd.conf } # server server dhcp { # from file /etc/raddb/sites-enabled/dhcp # Loading dhcp DHCP-Discover {...} # Loading dhcp DHCP-Request {...} # Loading dhcp DHCP-Decline {...} # Loading dhcp DHCP-Inform {...} # Loading dhcp DHCP-Release {...} # Loading dhcp DHCP-Lease-Query {...} } # server dhcp radiusd: #### Opening IP addresses and Ports #### listen { type = "dhcp" ipaddr = 0.0.0.0 port = 67 src_ipaddr = 10.88.198.13 } Listening on dhcp interface eth1.3118 address * port 67 bound to server dhcp Ready to process requests
On Jan 28, 2016, at 12:24 PM, O'Connell, Ryan <ryan.oconnell@viasat.com> wrote:
Curious if any has ballpark figures for any testing they have done with a "stand alone” FreeRadius DHCP server (3.0.10). I have setup a host with a local instance of MySQL (FR & MySQL on same Cent7 host) as well as FR pointing to a “physical” MySQL server with a massive amount of resources and write capability via a SAN. Irrespective of this, I seem to be topping out at 3 req/sec which seems strange.
How is MySQL configured? Which DB back-end are you using? i.e. MyISAM, etc. How many IPs are in the pool? Is the pool indexed?
When the server starts to fail, I start seeing errors like:
Sat Jan 23 12:19:51 2016 : ERROR: (60870) dhcp_sqlippool: ERROR: rlm_sql_mysql: ERROR 1205 (Lock wait timeout exceeded; try restarting transaction): HY000 Sat Jan 23 12:19:51 2016 : WARNING: (60870) WARNING: Module rlm_sqlippool became unblocked for request 60870
Because the SQL server is taking too long to respond.
Consistent with the errors in the radius log, the MySQL log has errors similar to this.
2016-01-23 12:19:52 7270 [ERROR] /usr/sbin/mysqld: Sort aborted: Lock wait timeout exceeded; try restarting transaction 2016-01-23 12:19:52 7270 [ERROR] /usr/sbin/mysqld: Lock wait timeout exceeded; try restarting transaction
Which probably means that the query is locking the entire table, and later queries get delayed / blocked.
I’ve tried adjusting the number of servers up and down from tens to hundreds (I know this isn’t a solution per se but just commenting that this has been done) while also adjusting my.cnf and sysctl (open files / processes) but the behavior is the same. I have also tried moving the MySQL data to a ramdisk since this is just a test but irrespective of whether a high I/) SAN or ramdisk is used, the performance is the same. Has anyone achieved better performance than this? I found comments from the past talking about the mac2ip performance being really high but I can’t seem to figure out what I’m missing to make dhcp_sqlippool go faster. I’m using “dhcperf” as the test but have also tried other simulators. Here’s the server startup. There is no selection logic so just a single pool.
The issue is in MySQL. For whatever reason, it can't handle the load. How to fix that is largely up to MySQL... We've tested the v3.1.x HEAD with Redis as an IP pool back-end. It's benchmarked at 10K IPs per second per redis node. i.e. a cluster with 4 nodes can do 40K IP assignments per second. I'm not aware of many *commercial* DHCP servers which can do that. Alan DeKok.
Thanks Alan. I am using INNODB and default indexes (default schema from mods-config). I put 65k ip’s in the pool so that I could let the test run for a bit but that’s definitely contributing to the performance bottleneck. If I drop the number to 5k I can do a bit more (5/sec). As a next step, I added an index to include the pool_name but unfortunately the results are the same. ---- from “show create table radius.radippool” ---- radippool | CREATE TABLE `radippool` ( `id` int(11) unsigned NOT NULL AUTO_INCREMENT, `pool_name` varchar(30) NOT NULL, `framedipaddress` varchar(15) NOT NULL DEFAULT '', `nasipaddress` varchar(15) NOT NULL DEFAULT '', `calledstationid` varchar(30) NOT NULL, `callingstationid` varchar(30) NOT NULL, `expiry_time` datetime DEFAULT NULL, `username` varchar(64) NOT NULL DEFAULT '', `pool_key` varchar(30) NOT NULL, PRIMARY KEY (`id`), KEY `radippool_poolname_expire` (`pool_name`,`expiry_time`), KEY `framedipaddress` (`framedipaddress`), KEY `radippool_nasip_poolkey_ipaddress` (`nasipaddress`,`pool_key`,`framedipaddress`), KEY `radippool_nasip_poolkey_ipaddress_pool_name` (`nasipaddress`,`pool_key`,`framedipaddress`,`pool_name`) ) ENGINE=InnoDB AUTO_INCREMENT=152091 DEFAULT CHARSET=latin1 | Even with some improvements to MySQL it sounds like Redis is a much better option so I’ll explore that instead. Thanks again for your feedback. -Ryan On 1/28/16, 9:34 AM, "Freeradius-Users on behalf of Alan DeKok" <freeradius-users-bounces+ryan.oconnell=viasat.com@lists.freeradius.org on behalf of aland@deployingradius.com> wrote:
On Jan 28, 2016, at 12:24 PM, O'Connell, Ryan <ryan.oconnell@viasat.com> wrote:
Curious if any has ballpark figures for any testing they have done with a "stand alone” FreeRadius DHCP server (3.0.10). I have setup a host with a local instance of MySQL (FR & MySQL on same Cent7 host) as well as FR pointing to a “physical” MySQL server with a massive amount of resources and write capability via a SAN. Irrespective of this, I seem to be topping out at 3 req/sec which seems strange.
How is MySQL configured? Which DB back-end are you using? i.e. MyISAM, etc. How many IPs are in the pool? Is the pool indexed?
When the server starts to fail, I start seeing errors like:
Sat Jan 23 12:19:51 2016 : ERROR: (60870) dhcp_sqlippool: ERROR: rlm_sql_mysql: ERROR 1205 (Lock wait timeout exceeded; try restarting transaction): HY000 Sat Jan 23 12:19:51 2016 : WARNING: (60870) WARNING: Module rlm_sqlippool became unblocked for request 60870
Because the SQL server is taking too long to respond.
Consistent with the errors in the radius log, the MySQL log has errors similar to this.
2016-01-23 12:19:52 7270 [ERROR] /usr/sbin/mysqld: Sort aborted: Lock wait timeout exceeded; try restarting transaction 2016-01-23 12:19:52 7270 [ERROR] /usr/sbin/mysqld: Lock wait timeout exceeded; try restarting transaction
Which probably means that the query is locking the entire table, and later queries get delayed / blocked.
I’ve tried adjusting the number of servers up and down from tens to hundreds (I know this isn’t a solution per se but just commenting that this has been done) while also adjusting my.cnf and sysctl (open files / processes) but the behavior is the same. I have also tried moving the MySQL data to a ramdisk since this is just a test but irrespective of whether a high I/) SAN or ramdisk is used, the performance is the same. Has anyone achieved better performance than this? I found comments from the past talking about the mac2ip performance being really high but I can’t seem to figure out what I’m missing to make dhcp_sqlippool go faster. I’m using “dhcperf” as the test but have also tried other simulators. Here’s the server startup. There is no selection logic so just a single pool.
The issue is in MySQL. For whatever reason, it can't handle the load. How to fix that is largely up to MySQL...
We've tested the v3.1.x HEAD with Redis as an IP pool back-end. It's benchmarked at 10K IPs per second per redis node. i.e. a cluster with 4 nodes can do 40K IP assignments per second.
I'm not aware of many *commercial* DHCP servers which can do that.
Alan DeKok.
- List info/subscribe/unsubscribe? See https://urldefense.proofpoint.com/v2/url?u=http-3A__www.freeradius.org_list_...
On Jan 28, 2016, at 1:14 PM, O'Connell, Ryan <ryan.oconnell@viasat.com> wrote:
Thanks Alan. I am using INNODB and default indexes (default schema from mods-config). I put 65k ip’s in the pool so that I could let the test run for a bit but that’s definitely contributing to the performance bottleneck. If I drop the number to 5k I can do a bit more (5/sec).
That's low. You know if you run the server in debug mode it becomes single threaded right?
As a next step, I added an index to include the pool_name but unfortunately the results are the same.
If you're not running in debug mode, it's probably roundtrip time, or I/O latency that's the cause of the poor performance. FreeRADIUS is largely synchronous, so latency between FR and the database really matters.
---- from “show create table radius.radippool” ---- radippool | CREATE TABLE `radippool` ( `id` int(11) unsigned NOT NULL AUTO_INCREMENT, `pool_name` varchar(30) NOT NULL, `framedipaddress` varchar(15) NOT NULL DEFAULT '', `nasipaddress` varchar(15) NOT NULL DEFAULT '', `calledstationid` varchar(30) NOT NULL, `callingstationid` varchar(30) NOT NULL, `expiry_time` datetime DEFAULT NULL, `username` varchar(64) NOT NULL DEFAULT '', `pool_key` varchar(30) NOT NULL, PRIMARY KEY (`id`), KEY `radippool_poolname_expire` (`pool_name`,`expiry_time`), KEY `framedipaddress` (`framedipaddress`), KEY `radippool_nasip_poolkey_ipaddress` (`nasipaddress`,`pool_key`,`framedipaddress`), KEY `radippool_nasip_poolkey_ipaddress_pool_name` (`nasipaddress`,`pool_key`,`framedipaddress`,`pool_name`) ) ENGINE=InnoDB AUTO_INCREMENT=152091 DEFAULT CHARSET=latin1 |
Even with some improvements to MySQL it sounds like Redis is a much better option so I’ll explore that instead. Thanks again for your feedback.
It'll scale to millions of lease allocations a second if you have enough Redis/FreeRADIUS servers and a front end load balancer :) -Arran
Run an explain in SQL using the query that you are using in freeradius to see what the time is and why the request takes a long time. Because there aren't many complexities here a NoSQL variant probably is the way to go anyway. alan
participants (4)
-
Alan Buxey -
Alan DeKok -
Arran Cudbard-Bell -
O'Connell, Ryan