Hi Alan, On Tue, Nov 10, 2009 at 5:35 PM, Alan DeKok <aland@deployingradius.com>wrote:

Dinh Pham Cong wrote:

...

start_servers = 1024 max_servers = 3000 max_spare_servers = 3000

That's a little high.

I saw that when 1000 client was created at the same time and send requests to Radius server concurrently there were about 2000+ radiusd threads were created too. I thought that 1000 concurrent client could be translated to more than 1000 packets/s. I have done another load tests with smaller concurrent clients: 100, 200, 500. However, Radiusd can not handle more than 1000 request/s (total requests/total time used to sent all the requests). For example Radiusd can handle 20 000 request in more than 20 seconds, 40 000 requests in more than 43 seconds.

My sql.conf

...

# number of sql connections to make to server num_sql_socks = 1000

Remember: 1000 packets/s does *not* mean 1000 threads or 1000 SQL sockets. The server should be able to do 1000 packets/s with a much smaller number of threads && sockets. i.e. 50/50.

I thought that num_sql_socks = 1000 meaned that Radiusd would create a connection pool of 1000 MySQL connections.

...

I developed a Java client that can create 1000 concurrent threads that connect and send accounting requests (start and stop) to Radius server.

See also "radperf" (http://networkradius.com/radperf.html). It can do load testing and user login simulations.

Thanks I will try.

...

However, after sending for 20 or 30 seconds, radius server generated a lot of error messages and crashed

Error messages

Tue Nov 10 16:14:34 2009 : Error: Discarding duplicate request from client tester1 port 41442 - ID: 134 due to unfinished request 2436

This is likely because your SQL database is too slow.

Have you checked to see if the SQL database can handle 1000 inserts/s?

...

Crashed

Nov 10 15:55:35 Mobile6 kernel: radiusd[22242]: segfault at 00000000000000c8 rip 0000003daee5306b rsp 00002aae5d82a780 error 4

See doc/bugs for instructions on producing useful crash reports. We can then use these to fix bugs.

...

Nov 10 15:57:29 Mobile6 kernel: radiusd[30941]: segfault at 00000000000000c8 rip 0000003daee5306b rsp 00002aaee56de180 error 4 Nov 10 16:14:36 Mobile kernel: radiusd[10788]: segfault at 00000000000000c8 rip 0000003daee5306b rsp 00002aac913de780 error 4

What do I need to do to enable FreeRadius server to meet more than 1000 accounting request (INSERT AND DELETE) for a second?

Use a database that can handle 1000 requests/s.

I have done separate load tests of INSERT and DELETE on radacct table (using MEMORY engine) and found that MySQL 5.0 can handle about 6000 - 8700 qps. I have used mysqlslap to do that. Thanks Dinh

Am Dienstag, 10. November 2009 13:02:50 schrieb Alisson:

I have the same problem with Freebsd + radius and Mysql 5. server hp core 2 duo, 2 gb ram

we have 1000 connections concurrent... and we have the same error "discarding packet'

Alan Dekok sayd to me that the problem is the same "slow database"

I'm don't found a solution yet

What about load balancing the SQL queryies? And make them asynchronous? Greetings, -- Dr. Michael Schwartzkopff MultiNET Services GmbH Addresse: Bretonischer Ring 7; 85630 Grasbrunn; Germany Tel: +49 - 89 - 45 69 11 0 Fax: +49 - 89 - 45 69 11 21 mob: +49 - 174 - 343 28 75 mail: misch@multinet.de web: www.multinet.de Sitz der Gesellschaft: 85630 Grasbrunn Registergericht: Amtsgericht München HRB 114375 Geschäftsführer: Günter Jurgeneit, Hubert Martens --- PGP Fingerprint: F919 3919 FF12 ED5A 2801 DEA6 AA77 57A4 EDD8 979B Skype: misch42

Dinh Pham Cong wrote:

I saw that when 1000 client was created at the same time and send requests to Radius server concurrently there were about 2000+ radiusd threads were created too. I thought that 1000 concurrent client could be translated to more than 1000 packets/s.

Maybe. It depends on a number of factors.

I have done another load tests with smaller concurrent clients: 100, 200, 500. However, Radiusd can not handle more than 1000 request/s (total requests/total time used to sent all the requests). For example Radiusd can handle 20 000 request in more than 20 seconds, 40 000 requests in more than 43 seconds.

That is usually because the back-end database can't handle more than those loads. Try replacing the "sql" entry in the "accounting" section with "ok". This means it won't log to the DB, but it will run faster. You SHOULD see it handle many 1000's of accounting requests/s. This means that FreeRADIUS can handle that load all by itself. The problem would then be the interaction between FreeRADIUS && MySQL.

I thought that num_sql_socks = 1000 meaned that Radiusd would create a connection pool of 1000 MySQL connections.

Yes. But you do NOT need 1000 MySQL connections to do 1000 packets/s. I've seen systems with 50 mySQL connections getting 2K packets/s.

See also "radperf" (http://networkradius.com/radperf.html). It can do load testing and user login simulations.

Thanks I will try.

Let me know of any issues you find.

I have done separate load tests of INSERT and DELETE on radacct table (using MEMORY engine) and found that MySQL 5.0 can handle about 6000 - 8700 qps. I have used mysqlslap to do that.

FreeRADIUS does a mix of selects, inserts and updates for accounting. It may be that the DB can only handle 1000 transactions/s with that mix. See also "sqltrace" config in sql.conf. You can try the following tests: - start off with a clean (i.e. empty) DB - enable sqltrace - run 10's of 1000's of packets through the server - wipe the DB - use a command-line SQL client to re-play SQL queries from the sqltrace file The performance of the last step SHOULD be within a factor of 2 of the performance of the RADIUS server. The performance will be higher than a real RADIUS server, because the RADIUS server has to send && receive packets over the network. The CLI client just has to read a file. Alan DeKok.

Hi Alan, I have attached a graph created by Munin that shows how MySQL works under FreeRadius load test http://img4.imageshack.us/img4/7204/mysqlqueriesday.png This test was conducted last night. I used 2 16-core Dell servers, FastEthernet 100Mb to run 2 test clients to test 2 Radius servers. Each client created 100 concurrent threads to simulate 100 concurrent users and send 10 000 000 start request and 10 000 000 stop each thread. Radius server and MySQL are installed in a 16-core Dell server too. I expect that the test can take several hours in the night. At some point of times in the very early this morning, Munin found that there was very high load in our MySQL server: 11 000 INSERT query/s and 11 000 DELETE query/s. It was because one of Radius server crashed and all the traffic was forwarded to the rest by the load balancer. It means that MySQL is not slow. There may be a bug in FreeRadius that makes it impossible to work in very high load environment: thread deadlock or kind of race condition??? Have someone got into this problem? Thanks Dinh On Tue, Nov 10, 2009 at 10:50 PM, Dinh Pham Cong <dinhpc@vega.com.vn> wrote:

Hi Alan,

On Tue, Nov 10, 2009 at 8:15 PM, Alan DeKok <aland@deployingradius.com>wrote:

...

FreeRADIUS does a mix of selects, inserts and updates for accounting. It may be that the DB can only handle 1000 transactions/s with that mix.

See also "sqltrace" config in sql.conf. You can try the following tests:

- start off with a clean (i.e. empty) DB - enable sqltrace - run 10's of 1000's of packets through the server - wipe the DB - use a command-line SQL client to re-play SQL queries from the sqltrace file

I have enabled sqltrace and found that for accounting purpose, there was only a single query made into MySQL for accounting stop or start: An insert for start and a delete for stop. For single INSERT and/or DELETE I have used mysqlslap for stress testing and found that MYSQL can handle 6000 - 8700 qps for 2000 concurrent clients. Therefore I double that MySQL is not a bottleneck. There might be something wrong with FreeRadius that made it not scalable when the load is high. When FreeRadius crashed there were only 1200 - 1400 radiusd threads as Munin recorded it.

The way it crashed is strange too. No fatal error in radius.log. There is a single kernel log found in /var/log/messages. Is it stable in 64 bit OS and SMP ?

Thanks

Dinh

Hi Alan, On Tue, Nov 10, 2009 at 11:25 PM, Alan DeKok <aland@deployingradius.com>wrote:

But if you follow the instructions in doc/bugs, you will be able to give us the data we need to FIX the crash.

Thanks Alan. I am reviewing my system to see if there is any wrong in our system settings that can cause FreeRadius crashed under high load. If I can not find something wrong caused by us, I will fill a bug report. So far, nothing is found. Thanks Dinh

Hi Michael, Yes I have enabled MySQL slow query log for all queries that take more than 1s for execution. I have found 2 slow queries only (INSERT) in one server and no slow query on the other. Because I installed MySQL using a rpm package so I can not patch it with microslow patch to enable logging queries that takes less than a second (e.x 0.5 second). Thanks, Dinh On Wed, Nov 11, 2009 at 12:22 AM, Michael da Silva Pereira < michael@tradepage.co.za> wrote:

Hi Dinh,

max_request_time is 10 seconds, have you switched on mysql slow query / non-index lookups logging ? (in my.cnf '#log_slow_queries = /var/log/mysql/mysql-slow.log' ). If the log file has nothing in it during the testing great, then MySQL is happy. If you have queries taking longer than '#long_query_time' there might be a problem.

Regards, Michael

Dinh Pham Cong wrote:

Thanks Alan. I am reviewing my system to see if there is any wrong in our system settings that can cause FreeRadius crashed under high load.

Think about that for a second. Is there really a configuration setting that says "crash under high load" ? I don't think so.

If I can not find something wrong caused by us, I will fill a bug report. So far, nothing is found.

All of the time you spend investigating things is WASTED. The ONLY thing that will help is to follow the instructions in doc/bugs. Follow the instructions in doc/bugs, or stop posting messages on this list. Alan DeKok.