Performance or locking issue with rlm_detail

[Alan DeKok]

On Mar 30, 2017, at 8:41 AM, Aleš Rygl <ales at rygl.net> wrote:.
> 
> 
> The spikes can reach about 6-8 kqps in case there is a network outage.

  Yeah, that's a lot for any DB.

  We typically do a load-balancer and sets of RADIUS/MySQL to handle this load.  It's much easier to configure, and much easier to scale.

>>  It would likely be better to write to SQL directly, and then to the detail
>> files as a backup.
> 
> I will try it - once more time. The DB is pretty fast nevertheless I am affraid about the performance as it will be bound just to DB.

  I don't see how writing to detail files and *then* SQL is any faster.  It should be slower, TBH.

  See raddb/sites-available/buffered-sql for examples.

> I have tried to increase the number of queues to 48 and reduced the load factor to 90. Unfortunately I looks like I am running out of something somewhere else...
> 
> Thu Mar 30 14:00:29 2017 : ERROR: (9093117) detail.mobile: ERROR: Couldn't open file /var/log/freeradius/radacct/detail.mobile_v3/queue-6/detail-2017033014: Too many different filenames
> Thu Mar 30 14:00:29 2017 : ERROR: (9093118) detail.mobile: ERROR: Couldn't open file /var/log/freeradius/radacct/detail.mobile_v3/queue-0/detail-2017033014: Too many different filenames
> 
> Could I save some resources by limiting max_servers or any other parameters?

  No.  To fix that message, you'll need to edit the source and recompile:

src/modules/rlm_detail/rlm_detail.c:	inst->ef = exfile_init(inst, 256, 30, inst->locking);

  Change the 256 to 1024 or higher.  I suppose that could be configurable, but 99.99% of people will never need it.

>>  For the future, we're re-architecting v4 to avoid this problem by design. 
>> The writes to SQL can be queued internally in an async fashion.  That
>> allows for high sustained throughput with minimal contention.  We're also
>> re-working the detail handling in a similar fashion.
> 
> Is it already available for testing?

  The v4.0.x branch is on github.  But the re-architecture isn't finished.

  Alan DeKok.