FreeRadius crashed on accounting load tests with 1000 concurrent clients

[Dinh Pham Cong]

Hi Alan,

On Tue, Nov 10, 2009 at 5:35 PM, Alan DeKok <aland at deployingradius.com>wrote:

> Dinh Pham Cong wrote:
> > start_servers = 1024
> > max_servers = 3000
> > max_spare_servers = 3000
>
>   That's a little high.
>
>
I saw that when 1000 client was created at the same time and send requests
to Radius server concurrently there were about 2000+ radiusd threads were
created too. I thought that 1000 concurrent client could be translated to
more than 1000 packets/s.

I have done another load tests with smaller concurrent clients: 100, 200,
500. However, Radiusd can not handle more than 1000 request/s (total
requests/total time used to sent all the requests). For example Radiusd can
handle 20 000 request in more than 20 seconds, 40 000 requests in more than
43 seconds.

> My sql.conf
> >
> > # number of sql connections to make to server
> > num_sql_socks = 1000
>
>   Remember: 1000 packets/s does *not* mean 1000 threads or 1000 SQL
> sockets.  The server should be able to do 1000 packets/s with a much
> smaller number of threads && sockets.  i.e. 50/50.
>

I thought that num_sql_socks = 1000 meaned that Radiusd would create a
connection pool of 1000 MySQL connections.


> > I developed a Java client that can create 1000 concurrent threads that
> > connect and send accounting requests (start and stop) to Radius server.
>
>   See also "radperf" (http://networkradius.com/radperf.html).  It can do
> load testing and user login simulations.
>
>
Thanks I will try.


> > However, after sending for 20 or 30 seconds, radius server generated a
> > lot of error messages and crashed
> >
> > Error messages
> >
> > Tue Nov 10 16:14:34 2009 : Error: Discarding duplicate request from
> > client tester1 port 41442 - ID: 134 due to unfinished request 2436
>
>   This is likely because your SQL database is too slow.
>
>  Have you checked to see if the SQL database can handle 1000 inserts/s?
>
> > Crashed
> >
> > Nov 10 15:55:35 Mobile6 kernel: radiusd[22242]: segfault at
> > 00000000000000c8 rip 0000003daee5306b rsp 00002aae5d82a780 error 4
>
>   See doc/bugs for instructions on producing useful crash reports.  We
> can then use these to fix bugs.
>
> > Nov 10 15:57:29 Mobile6 kernel: radiusd[30941]: segfault at
> > 00000000000000c8 rip 0000003daee5306b rsp 00002aaee56de180 error 4
> > Nov 10 16:14:36 Mobile kernel: radiusd[10788]: segfault at
> > 00000000000000c8 rip 0000003daee5306b rsp 00002aac913de780 error 4
> >
> > What do I need to do to enable FreeRadius server to meet more than 1000
> > accounting request (INSERT AND DELETE) for a second?
>
>   Use a database that can handle 1000 requests/s.
>

I have done separate load tests of INSERT and DELETE on radacct table (using
MEMORY engine) and found that MySQL 5.0 can handle about 6000 - 8700 qps. I
have used mysqlslap to do that.


Thanks

Dinh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freeradius.org/pipermail/freeradius-users/attachments/20091110/f2e899c0/attachment.html>