FreeRadius crashed on accounting load tests with 1000 concurrent clients

[Dinh Pham Cong]

Hi Alan,

I have attached a graph created by Munin that shows how MySQL works under
FreeRadius load test

http://img4.imageshack.us/img4/7204/mysqlqueriesday.png

This test was conducted last night. I used 2 16-core Dell servers,
FastEthernet 100Mb to run 2 test clients to test 2 Radius servers. Each
client created 100 concurrent threads to simulate 100 concurrent users and
send 10 000 000 start request and 10 000 000 stop each thread. Radius server
and MySQL are installed in a 16-core Dell server too. I expect that the test
can take several hours in the night. At some point of times in the very
early this morning, Munin found that there was very high load in our MySQL
server: 11 000 INSERT query/s and 11 000 DELETE query/s. It was because one
of Radius server crashed and all the traffic was forwarded to the rest by
the load balancer.

It means that MySQL is not slow. There may be a bug in FreeRadius that makes
it impossible to work in very high load environment: thread deadlock or kind
of race condition??? Have someone got into this problem?

Thanks

Dinh

On Tue, Nov 10, 2009 at 10:50 PM, Dinh Pham Cong <dinhpc at vega.com.vn> wrote:

> Hi Alan,
>
>
> On Tue, Nov 10, 2009 at 8:15 PM, Alan DeKok <aland at deployingradius.com>wrote:
>
>>
>>  FreeRADIUS does a mix of selects, inserts and updates for accounting.
>>  It may be that the DB can only handle 1000 transactions/s with that mix.
>>
>>  See also "sqltrace" config in sql.conf.  You can try the following tests:
>>
>> - start off with a clean (i.e. empty) DB
>> - enable sqltrace
>> - run 10's of 1000's of packets through the server
>> - wipe the DB
>> - use a command-line SQL client to re-play SQL queries from the sqltrace
>> file
>>
>
>
> I have enabled sqltrace and found that for accounting purpose, there was
> only a single query made into MySQL for accounting stop or start: An insert
> for start and a delete for stop. For single INSERT and/or DELETE I have used
> mysqlslap for stress testing and found that MYSQL can handle 6000 - 8700 qps
> for 2000 concurrent clients. Therefore I double that MySQL is not a
> bottleneck. There might be something wrong with FreeRadius that made it not
> scalable when the load is high. When FreeRadius crashed there were only 1200
> - 1400 radiusd threads as Munin recorded it.
>
> The way it crashed is strange too. No fatal error in radius.log. There is a
> single kernel log found in /var/log/messages. Is it stable in 64 bit OS and
> SMP ?
>
> Thanks
>
>
> Dinh
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freeradius.org/pipermail/freeradius-users/attachments/20091110/6a5e0384/attachment.html>