Hi All,
From the radmin command, I can see the following
radmin> stats socket 10.255.251.2 1812 requests 645589 responses 641483 accepts 359013 rejects 282470 challenges 0 dup 0 invalid 0 malformed 0 bad_authenticator 0 dropped 3110 unknown_types 0 last_packet 1402555332 elapsed.1us 0 elapsed.10us 0 elapsed.100us 0 elapsed.1ms 0 elapsed.10ms 229378 elapsed.100ms 384248 elapsed.1s 27768 elapsed.10s 86 Now given my specific application that I use Radius for, the elapsed times are OK for the majority of requests (the sub 100ms range is fine). I just have a quick question and a thought perhaps. Under what circumstances will FR "drop" a request (i.e. dropped 3110). Also, would it be possible to be able to get stats for the thread pool through radmin as well? I've been experiencing very weird, intermittent and seemingly random timeouts from FR, and after increasing (doubling) the size of my thread pool, most of my issues seemed to have gone away. Surely, those kind of stats in terms of the thread pool will not only be very helpful, but it will be absolutely beneficial in terms of debugging performance issues? Also as a last note, maybe something like a "slow query" log option can be added as well, where we can log auth/acct requests to a while in instances where FR takes longer than a specific amount of time to process? Say, if it takes longer than 100ms to process a request, log the request to a specific file? Not only will it again help to troubleshoot and isolate performance issues, but it could possibly also mean that the entire server does not need to be run in debug mode to identify requests which takes long to process. Just a thought, and just my 2c :-) Thanks, Chris.
On 12 Jun 2014, at 07:59, Chris Knipe <savage@savage.za.org> wrote:
Hi All,
From the radmin command, I can see the following
radmin> stats socket 10.255.251.2 1812 requests 645589 responses 641483 accepts 359013 rejects 282470 challenges 0 dup 0 invalid 0 malformed 0 bad_authenticator 0 dropped 3110 unknown_types 0 last_packet 1402555332 elapsed.1us 0 elapsed.10us 0 elapsed.100us 0 elapsed.1ms 0 elapsed.10ms 229378 elapsed.100ms 384248 elapsed.1s 27768 elapsed.10s 86
Now given my specific application that I use Radius for, the elapsed times are OK for the majority of requests (the sub 100ms range is fine). I just have a quick question and a thought perhaps.
Under what circumstances will FR "drop" a request (i.e. dropped 3110). Also, would it be possible to be able to get stats for the thread pool through radmin as well?
When they take an excessively long time to complete, when the request queue is full, when there's already a request in the request queue matching the incoming request which has not yet been processed. There may be more, Alan D will know.
I've been experiencing very weird, intermittent and seemingly random timeouts from FR, and after increasing (doubling) the size of my thread pool, most of my issues seemed to have gone away.
Check your database. FreeRADIUS worker threads block on database queries, if your database takes a long time to respond all the workers can block, and cause the request queue to overflow.
Surely, those kind of stats in terms of the thread pool will not only be very helpful, but it will be absolutely beneficial in terms of debugging performance issues?
There are already events logged when a module take excessively long the process a request.
Also as a last note, maybe something like a "slow query" log option can be added as well, where we can log auth/acct requests to a while in instances where FR takes longer than a specific amount of time to process? Say, if it takes longer than 100ms to process a request, log the request to a specific file? Not only will it again help to troubleshoot and isolate performance issues, but it could possibly also mean that the entire server does not need to be run in debug mode to identify requests which takes long to process.
Use v3.0.x radsniff, it already has the capability to print out the requests which have timed out. -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS Development Team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
Chris Knipe wrote:
Now given my specific application that I use Radius for, the elapsed times are OK for the majority of requests (the sub 100ms range is fine).
100ms for a request? That's terrible. It indicates something wrong with your configuration. If you put an entry into the "users" file in the default config, the server can do ~5K auths/s. That's 2 microseconds each, or 50K times faster.
Under what circumstances will FR "drop" a request (i.e. dropped 3110).
When a request is blocked, and the NAS transmits a new request using the same src/dst IP/port and RADIUS ID.
Also, would it be possible to be able to get stats for the thread pool through radmin as well?
Sure. Send a patch. :)
I've been experiencing very weird, intermittent and seemingly random timeouts from FR, and after increasing (doubling) the size of my thread pool, most of my issues seemed to have gone away. Surely, those kind of stats in terms of the thread pool will not only be very helpful, but it will be absolutely beneficial in terms of debugging performance issues?
Not really. Increasing the size of the thread pool is putting a band-aid on the problem. It won't fix anything.
Also as a last note, maybe something like a "slow query" log option can be added as well, where we can log auth/acct requests to a while in instances where FR takes longer than a specific amount of time to process?
It's possible, but it would slow down the server a fair bit. It wouldn't be a "slow query" log, so much as a "slow module" log. And it would almost always show: - SQL - EAP - exec
Say, if it takes longer than 100ms to process a request, log the request to a specific file? Not only will it again help to troubleshoot and isolate performance issues, but it could possibly also mean that the entire server does not need to be run in debug mode to identify requests which takes long to process.
That's not really necessary. The default configuration isn't slow. So... what did you change? That's the source of the slowness. Alan DeKok.
On Thu, Jun 12, 2014 at 3:14 PM, Alan DeKok <aland@deployingradius.com> wrote:
Chris Knipe wrote:
Now given my specific application that I use Radius for, the elapsed times are OK for the majority of requests (the sub 100ms range is fine).
100ms for a request? That's terrible. It indicates something wrong with your configuration.
If you put an entry into the "users" file in the default config, the server can do ~5K auths/s. That's 2 microseconds each, or 50K times faster.
Yes. Unfortunately, I need to SSH/Telnet into a load balancer, and set up certain parameters for each unique session. This isn't traditional PPP based authentication, this is actually application level authentication and application load balancing. :-) Sub 100ms is great for what I authenticate and the process involved in terms of authenticating. Radius is and can be used for more than just traditional PPP authentication, yes?
Under what circumstances will FR "drop" a request (i.e. dropped 3110).
When a request is blocked, and the NAS transmits a new request using the same src/dst IP/port and RADIUS ID.
Interesting, considering that the application wasn't programmed to do ANY re-transmissions. It does do re-transmissions now though, but those stats were pulled prior to re-transmissions being introduced inside the application. Would a request not also be dropped if
Also, would it be possible to be able to get stats for the thread pool through radmin as well?
Sure. Send a patch. :)
If I was a C programmer, I would have. Unfortunately I know diddly squat about C. See it then as a feature request :-)
I've been experiencing very weird, intermittent and seemingly random timeouts from FR, and after increasing (doubling) the size of my thread pool, most of my issues seemed to have gone away. Surely, those kind of stats in terms of the thread pool will not only be very helpful, but it will be absolutely beneficial in terms of debugging performance issues?
Not really. Increasing the size of the thread pool is putting a band-aid on the problem. It won't fix anything.
It will fix allot if you KNOW your authentication process WILL be slow. Since I've put the 'band aid' on, I have not had one single issue with either auth or acct, again, specific to my application per say. Unfortunately, in real world cases it is not always possible to process authentication requests sub 2ms, or whatever the 'ideal' number would be. Sometimes you just have to work with what you have, and make the best of it. -- Chris.
participants (3)
-
Alan DeKok -
Arran Cudbard-Bell -
Chris Knipe