I use FreeRADIUS Version 3.0.20 as both proxy and (auth/acct) server. I have more than 200 realms. Every one of them is redirecting to a different site file with a separate sql module but with a shared sql_pool. I recently had a strange problem, every time the system hits 150 auth requests per second accounting packets drop from 1000 req/s to about 200 req/s and the system begins to delay accounting requests and accounting no response log messages arise in connected clients. The system operates normally under heavy load and no stress is visible to me in the sql server. I am using a monitor system and it shows no drop or increase in sql load. During accounting drop the ssd usage drops from 80% to 20%. My radiusd thread pool configuration for 36 cores and 256GB ram and enterprise ssd is: thread pool { start_servers = 60 max_servers = 144 min_spare_servers = 20 max_spare_servers = 60 max_requests_per_server = 0 auto_limit_acct = no } My shared sql_pool around all 200 sql modules is: pool { start = 36 min = 18 max = 72 spare = 36 uses = 0 retry_delay = 30 lifetime = 0 idle_timeout = 60 } I have already checked and tuned mysql server to best values and checked the indices for the tables and everything is pretty well configured I am 80% sure the problem is with freeradius configuration because what solve the problem temporarily is restarting freeradius service until it rehits 150 auth req/s and problem rearise. Do you agree with me that the problem is from freeradius or it could be something else and what can I do to solve it ? Thanks in advance Ibrahem
On May 3, 2025, at 7:51 AM, Ibrahemoo Ghost <ibrahemooghost@gmail.com> wrote:
I recently had a strange problem, every time the system hits 150 auth requests per second accounting packets drop from 1000 req/s to about 200 req/s and the system begins to delay accounting requests and accounting no response log messages arise in connected clients.
The server has an internal queue which prioritizes Access-Requests over Accounting-Requests. This means that if there are many Access-Requests, and they take a long time to process, the Accounting-Request packets will be delayed. The server will process Accounting-Requests only when there are no more Access-Requests to process. That could be the issue here. So is the server taking a long time to process Access-Requests? You should be able to look at the server stats to see how long each packet is taking. See sites-available/status, and sites-available/control. The "status" virtual server a return statistics in a RADIUS packet. The "control" virtual server lets you run a command-line "radmin" tool. That lets you look at elapsed times (i.e. time between request and reply). You can look at stats by home server, packet type, and listener. Unfortunately, the elapsed times are only available via radmin, and not via RADIUS packets. That makes it a bit more difficult to track the times automatically, but not impossible.
The system operates normally under heavy load and no stress is visible to me in the sql server. I am using a monitor system and it shows no drop or increase in sql load.
150 auth/s and 1000 acct/s should not be a heavy load. We've tested the server at 20K auth/s using OpenLDAP, and 2K+ acct/s to SQL. The overall design of the configuration has more impact on performance than CPU. i.e. Using a 36 core CPU won't help, if the server is spending all of its time waiting for the SQL server to return results. You can check this by creating a test configuration where the server does nothing at all, and immediately returns for all packets: authorize { accept } accounting { ok } This should be able to handle 20K+ packets/s.
I have already checked and tuned mysql server to best values and checked the indices for the tables and everything is pretty well configured I am 80% sure the problem is with freeradius configuration because what solve the problem temporarily is restarting freeradius service until it rehits 150 auth req/s and problem rearise.
That is weird.
Do you agree with me that the problem is from freeradius or it could be something else and what can I do to solve it ?
The problem is likely a combination of the server configuration and interaction with the databases. Some additional things to check: * don't send logs to systemd for syslog. We've seen it max out at a few hundred log messages per second, which blocks FreeRADIUS * remove everything from the configuration that you're not using. The "unix" module is very slow, for example. If you don't need it, remove it from the configuration * use radmin to get the full server stats, including elapsed time for requests, and how long packets are staying in the input queues. Probably the most important thing to do is the last one. Looking only at overall packets/s is just too "high level" to tell you much about what the server is doing. Once you get detailed stats abut everything the server does, it becomes much easier to tell exactly where the problem is. Alan DeKok.
Ok so now you pointed me in the right direction, in my configuration I have an external script written in C lang to authenticate users using the exec module which is what takes a long time to execute and causes the problem. But the module is important to my setup, my question is how can I solve the accounting packet drop internally, is modifying the pool values make any difference? On Sat, May 3, 2025 at 5:21 PM Alan DeKok <aland@deployingradius.com> wrote:
On May 3, 2025, at 7:51 AM, Ibrahemoo Ghost <ibrahemooghost@gmail.com> wrote:
I recently had a strange problem, every time the system hits 150 auth requests per second accounting packets drop from 1000 req/s to about 200 req/s and the system begins to delay accounting requests and accounting no response log messages arise in connected clients.
The server has an internal queue which prioritizes Access-Requests over Accounting-Requests.
This means that if there are many Access-Requests, and they take a long time to process, the Accounting-Request packets will be delayed. The server will process Accounting-Requests only when there are no more Access-Requests to process.
That could be the issue here. So is the server taking a long time to process Access-Requests? You should be able to look at the server stats to see how long each packet is taking. See sites-available/status, and sites-available/control.
The "status" virtual server a return statistics in a RADIUS packet.
The "control" virtual server lets you run a command-line "radmin" tool. That lets you look at elapsed times (i.e. time between request and reply). You can look at stats by home server, packet type, and listener. Unfortunately, the elapsed times are only available via radmin, and not via RADIUS packets. That makes it a bit more difficult to track the times automatically, but not impossible.
The system operates normally under heavy load and no stress is visible to me in the sql server. I am using a monitor system and it shows no drop or increase in sql load.
150 auth/s and 1000 acct/s should not be a heavy load. We've tested the server at 20K auth/s using OpenLDAP, and 2K+ acct/s to SQL.
The overall design of the configuration has more impact on performance than CPU. i.e. Using a 36 core CPU won't help, if the server is spending all of its time waiting for the SQL server to return results.
You can check this by creating a test configuration where the server does nothing at all, and immediately returns for all packets:
authorize { accept }
accounting { ok }
This should be able to handle 20K+ packets/s.
I have already checked and tuned mysql server to best values and checked the indices for the tables and everything is pretty well configured I am 80% sure the problem is with freeradius configuration because what solve the problem temporarily is restarting freeradius service until it rehits 150 auth req/s and problem rearise.
That is weird.
Do you agree with me that the problem is from freeradius or it could be something else and what can I do to solve it ?
The problem is likely a combination of the server configuration and interaction with the databases.
Some additional things to check:
* don't send logs to systemd for syslog. We've seen it max out at a few hundred log messages per second, which blocks FreeRADIUS
* remove everything from the configuration that you're not using. The "unix" module is very slow, for example. If you don't need it, remove it from the configuration
* use radmin to get the full server stats, including elapsed time for requests, and how long packets are staying in the input queues.
Probably the most important thing to do is the last one. Looking only at overall packets/s is just too "high level" to tell you much about what the server is doing. Once you get detailed stats abut everything the server does, it becomes much easier to tell exactly where the problem is.
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On May 3, 2025, at 6:09 PM, Ibrahemoo Ghost <ibrahemooghost@gmail.com> wrote:
Ok so now you pointed me in the right direction, in my configuration I have an external script written in C lang to authenticate users using the exec module which is what takes a long time to execute and causes the problem.
Yes, that would be it. Exec'ing a script is very, very, slow. The server also comes with Perl and Python plugins. Switching to those will be significantly faster than running a script. But the short answer is "don't do that". The server has enough functionality that external scripts are pretty much never needed. And after 25+ years, the only time I've seen Perl / Python being needed is when there's a third-party API which is only available via a Perl / Python library. I have confidence that most scripts can be re-written in "unlang" statements. Perhaps you could explain what the script does?
But the module is important to my setup, my question is how can I solve the accounting packet drop internally, is modifying the pool values make any difference?
No. The issue is that the server is spending all of its time waiting for the script to return. There are a few possible solutions here which could help: 1) convert the script to "unlang" policies. This is almost always possible 2) or, run two copies of FreeRADIUS. One for authentication, and one for accounting. That way if the authentication server is slow, it doesn't affect the accounting server. 3) convert your configuration to the git "master" branch which will become v4. :) If you're not using RADIUS/TLS, it should be reasonably straightforward. The entire threading model of v4 has been redesigned. In v3, the server needs to block while it's waiting for a script to return. In v4, the server doesn't need to block. This means that it can continue running, and process Accounting-Request packets, while it's waiting for an authentication script to return. But the best solution is "don't run a script for every packet". Alan DeKok.
participants (2)
-
Alan DeKok -
Ibrahemoo Ghost