Slow "update radacct ..." query
Hi I know that this maybe is not FreeRadius related question... I have some problems with freeradius and mysql as a database. radacct table is InnoDB and update queries (interim-updates) are terrible slow (10 - 50 sec). Interim-updates are triggered every 8 minutes so it isn't under heavy load. (max 500 users online @ same time) After I wrote a small benchmark script for testing this kind of queries (10 000 queries "UPDATE radacct SET ... " with random integer values for acctsessiontime, acctinputoctets and acctoutputoctets) I can't reproduce this problem. My script takes 10 000 random rows from radacct table (table has almost 600 000 rows) and it is finished in 130-180 seconds (executed 5 times and all results are between 130 and 180 seconds). That means 0,013-0,018s/query (average) Slow query log is empty after this benchm. script. I checked radacct table later and I have exaclty 10 000 updated rows so the script is OK. Any ideas how to trace this problem ? FreeRadius is 2.1.6 and the same problem I had with FR1.1.7 and FR2.1.3
Hi,
I have some problems with freeradius and mysql as a database. radacct table is InnoDB and update queries (interim-updates) are terrible slow (10 - 50 sec). Interim-updates are triggered every 8 minutes so it isn't under heavy load. (max 500 users online @ same time)
After I wrote a small benchmark script for testing this kind of queries (10 000 queries "UPDATE radacct SET ... " with random integer values for acctsessiontime, acctinputoctets and acctoutputoctets) I can't reproduce this problem. My script takes 10 000 random rows from radacct table (table has almost 600 000 rows) and it is finished in 130-180 seconds (executed 5 times and all results are between 130 and 180 seconds). That means 0,013-0,018s/query (average)
Slow query log is empty after this benchm. script. I checked radacct table later and I have exaclty 10 000 updated rows so the script is OK.
mysql isnt too hot - even with innoDB. postgres can have a quicker throughput on such operations....however, you can speedmysql up a little further by the addition (and removal!) of indexes for the table. do some 'explain' commands instead of straight 'update' to see what the operation does and why it takes so long - usually its having to go through far too many records. you should have this down to sub-seconds after some work. you may need to play with /etc/my.cnf to give yourself much larger buffers and key space etc. you should have some templates somewhere on your filesystem eg my-large.cnf (locate my-large) to give you some idea of play values. alan
Marinko Tarlac wrote:
I have some problems with freeradius and mysql as a database. radacct table is InnoDB and update queries (interim-updates) are terrible slow (10 - 50 sec). Interim-updates are triggered every 8 minutes so it isn't under heavy load. (max 500 users online @ same time)
The MySQL database is slow.
After I wrote a small benchmark script for testing this kind of queries (10 000 queries "UPDATE radacct SET ... " with random integer values for acctsessiontime, acctinputoctets and acctoutputoctets) I can't reproduce this problem. My script takes 10 000 random rows from radacct table (table has almost 600 000 rows) and it is finished in 130-180 seconds (executed 5 times and all results are between 130 and 180 seconds). That means 0,013-0,018s/query (average)
Have you checked the sqltrace file? If not (or if it doesn't exist), enable it, and use it.
Slow query log is empty after this benchm. script. I checked radacct table later and I have exaclty 10 000 updated rows so the script is OK.
Any ideas how to trace this problem ?
My guess is that you're not receiving accounting start packets. In this case, the SQL module tries to do an update, fails, and then does an insert. It's likely that your test is updating *existing* radacct rows. Try changing it to do 10,000 inserts. Odds are it will be slow. In this case, there really isn't a whole log of magic on the FreeRADIUS side. It does SQL queries by calling MySQL functions. Once execution has been passed to the MySQL function, it has really very little to do with FreeRADIUS. Look at the sqltrace file, and try re-playing the queries there into a *fresh* database. i.e. wipe the radacct database, and let it run for a while. Then wipe the database *again*, and re-play the sqltrace file. Odds are that replaying it with the MySQL command-line client will be just as slow as you saw with FreeRADIUS. Alan DeKok.
@ Alan DeKok, A.L.M. Buxey ---- Thanks for your time Alan DeKok wrote:
My guess is that you're not receiving accounting start packets. In this case, the SQL module tries to do an update, fails, and then does an insert.
After some investigation, I found that accounting start packets was received before update packets and session was already in database before the update happen. (AcctStartTime < MySQL time in slow query log) Also, after few tests I have: - 10 000 row updates which doesn't exist was done in avg. 4 seconds (tested 3 times) - 10 000 inserts in less then 9 seconds (avg. tested 3 times) my.cnf is a combination of my-huge.cnf and optimization tips I get with: - tunning-primer.sh (http://www.day32.com/MySQL/tuning-primer.sh), - mytop, - mysqlreport (http://hackmysql.com/mysqlreport) and - mysqltuner.pl (http://wiki.mysqltuner.com/MySQLTuner) Similar results I get with default my-huge.cnf
It's likely that your test is updating *existing* radacct rows. Try changing it to do 10,000 inserts. Odds are it will be slow.
In this case, there really isn't a whole log of magic on the FreeRADIUS side. It does SQL queries by calling MySQL functions. Once execution has been passed to the MySQL function, it has really very little to do with FreeRADIUS.
Look at the sqltrace file, and try re-playing the queries there into a *fresh* database. i.e. wipe the radacct database, and let it run for a while. Then wipe the database *again*, and re-play the sqltrace file.
Odds are that replaying it with the MySQL command-line client will be just as slow as you saw with FreeRADIUS.
I'll try this and that is the reason why I installed FR2.1.6. Another option is to update MySQL to latest available version 5.0.x (in 5.0 generation) because there are some indication that 5.0.45 has some bugs with cache and InnoDB storage engine...
Marinko Tarlac wrote:
After some investigation, I found that accounting start packets was received before update packets and session was already in database before the update happen. (AcctStartTime < MySQL time in slow query log) Also, after few tests I have: - 10 000 row updates which doesn't exist was done in avg. 4 seconds (tested 3 times) - 10 000 inserts in less then 9 seconds (avg. tested 3 times)
Hmm... that would seem to indicate that MySQL is OK.
Odds are that replaying it with the MySQL command-line client will be just as slow as you saw with FreeRADIUS.
I'll try this and that is the reason why I installed FR2.1.6.
Thanks. That will help us track it down. If replaying the sqltrace file is very fast, then there's some kind of magic going on... Most people find MySQL sufficiently fast, so it's hard to understand why some people run into issues.
Another option is to update MySQL to latest available version 5.0.x (in 5.0 generation) because there are some indication that 5.0.45 has some bugs with cache and InnoDB storage engine...
Maybe... Alan DeKok.
participants (3)
-
A.L.M.Buxey@lboro.ac.uk -
Alan DeKok -
Marinko Tarlac