Freeradius as a loadbalancer - logging dropped accounting requests
Hello, I am running several instances of Freeradius 3.0.13 on a single machine and one of them acts as a load balancer distributing just accounting request among four home servers listening on loopback on different ports: thread pool { start_servers = 64 max_servers = 1500 min_spare_servers = 50 max_spare_servers = 80 max_requests_per_server = 0 auto_limit_acct = no } realm NULL { acct_pool = DRACO_acct_pool nostrip } realm DEFAULT { acct_pool = DRACO_acct_pool nostrip } home_server_pool DRACO_acct_pool { type = keyed-balance home_server = rzt-gc-01 home_server = rzt-gc-02 home_server = rzt-gc-03 home_server = rzt-gc-04 } home_server rzt-gc-01 { type = acct ipaddr = 127.0.0.1 port = 2814 secret = testing123 response_window = 20 zombie_period = 40 revive_interval = 120 status_check = status-server check_interval = 10 check_timeout = 4 num_answers_to_alive = 3 coa { irt = 2 mrt = 16 mrc = 5 mrd = 30 } limit { max_connections = 16 max_requests = 0 lifetime = 0 idle_timeout = 0 } } ... and so on. The load is quite heavy - about 3.5k req/s. Balancing works fine, the key is Calling-Station-ID (MSISDN). What I can see is that there are some dropped packets in the stats: # echo "stats client acct" | radmin -d /etc/freeradius_lb/ radmin> stats client acct requests 39511449 responses 39508797 dup 0 invalid 0 malformed 0 bad_authenticator 0 dropped 1584 unknown_types 0 last_packet 1498477859 elapsed.1us 0 elapsed.10us 0 elapsed.100us 33765867 elapsed.1ms 5742921 elapsed.10ms 9 elapsed.100ms 0 elapsed.1s 0 elapsed.10s 0 radmin> The homeservers have no performance issues: # echo "stats client acct" | radmin -d /etc/freeradius_04/ radmin> stats client acct requests 159032334 responses 159030646 dup 21 invalid 0 malformed 0 bad_authenticator 0 dropped 21 unknown_types 0 last_packet 1498478023 elapsed.1us 0 elapsed.10us 0 elapsed.100us 158795858 elapsed.1ms 234786 elapsed.10ms 2 elapsed.100ms 0 elapsed.1s 0 elapsed.10s 0 radmin> Is there a way how to indetify the reason why are some req. dropped? I.e. logging such requestest? LB there is no lack of CPU or memory on the server, no drops in the UDP stack.. Thanks Ales
On Jun 26, 2017, at 8:01 AM, Aleš Rygl <ales@rygl.net> wrote:
# echo "stats client acct" | radmin -d /etc/freeradius_04/ radmin> stats client acct requests 159032334 responses 159030646 dup 21 ... dropped 21
Is there a way how to indetify the reason why are some req. dropped? I.e. logging such requestest? LB there is no lack of CPU or memory on the server, no drops in the UDP stack..
See the "dup" field. The NAS is sending duplicate Accounting-Request packets while the server is still processing the old one. So, the new one is dropped. This is usually because the RADIUS server is slow to respond to a particular request. i.e. transient load spikes, etc. Alan DeKok.
Hi, yes, nevertheless "dup" are just 21 packets over several days (and this is the same for remaining home servers) while the LB instance dropped 2k of requests within 2 hours... According to some other stats it dropping happens mainly every hour in peaks. I was monitoring the number of threads and it is constant all the time and there is really big headroom provide by config. Ales On pondělí 26. června 2017 8:25:33 CEST Alan DeKok wrote:
On Jun 26, 2017, at 8:01 AM, Aleš Rygl <ales@rygl.net> wrote:
# echo "stats client acct" | radmin -d /etc/freeradius_04/ radmin> stats client acct requests 159032334 responses 159030646 dup 21 ... dropped 21
Is there a way how to indetify the reason why are some req. dropped? I.e. logging such requestest? LB there is no lack of CPU or memory on the server, no drops in the UDP stack.. See the "dup" field.
The NAS is sending duplicate Accounting-Request packets while the server is still processing the old one. So, the new one is dropped.
This is usually because the RADIUS server is slow to respond to a particular request. i.e. transient load spikes, etc.
Yes, this makes sense, nevertheless the balancer instance has
Alan DeKok.
On Jun 26, 2017, at 8:38 AM, Aleš Rygl <ales@rygl.net> wrote:
Hi, yes, nevertheless "dup" are just 21 packets over several days (and this is the same for remaining home servers) while the LB instance dropped 2k of requests within 2 hours... According to some other stats it dropping happens mainly every hour in peaks.
Then it's a transient load issue. Either there are peaks higher than the system can handle, or something else on the system is stealing CPU time.
I was monitoring the number of threads and it is constant all the time and there is really big headroom provide by config.
You can configure "max_requests" as high as you want, but if the system is starved for CPU, it will still drop packets. I've tested FR in simple configs at 20K packets/s on low-end systems. If you're getting less than that (and you likely will), it's because the back-end database is slow, or the system is CPU starved, etc. If you're losing packets at 1000 packets/s, FreeRADIUS is not the cause. Alan DeKok.
Hi Alan,
Then it's a transient load issue. Either there are peaks higher than the system can handle, or something else on the system is stealing CPU time.
I was monitoring the number of threads and it is constant all the time and there is really big headroom provide by config.
You can configure "max_requests" as high as you want, but if the system is starved for CPU, it will still drop packets.
FR is using up to 140% CPU of 48 core system (256GiB RAM, SSD array) I have reniced it and assigned real-time ionice class, the overall CPU of the system is about 20%.
I've tested FR in simple configs at 20K packets/s on low-end systems. If you're getting less than that (and you likely will), it's because the back-end database is slow, or the system is CPU starved, etc.
There is no logging to file, no database in loadbalancer config. It does not even listen for authentication. I can disable all unnecessary modules that could slow it down. There is no attributes rewriting or so. I need just accounting balancing with a really minimalistic config.
If you're losing packets at 1000 packets/s, FreeRADIUS is not the cause.
It must be a misunderstanding. The overall drop rate of LB is about 4,6e-05 and drops are finally solved by a retransmission. I was just wonder if there is a way how to detect/log drops or avoid them by config optimisation. Thanks Ales
On Jun 26, 2017, at 10:23 AM, Aleš Rygl <ales@rygl.net> wrote:
FR is using up to 140% CPU of 48 core system (256GiB RAM, SSD array) I have reniced it and assigned real-time ionice class, the overall CPU of the system is about 20%.
That's a huge system...
There is no logging to file, no database in loadbalancer config. It does not even listen for authentication. I can disable all unnecessary modules that could slow it down. There is no attributes rewriting or so. I need just accounting balancing with a really minimalistic config.
OK.
If you're losing packets at 1000 packets/s, FreeRADIUS is not the cause.
It must be a misunderstanding. The overall drop rate of LB is about 4,6e-05 and drops are finally solved by a retransmission. I was just wonder if there is a way how to detect/log drops or avoid them by config optimisation.
There's no real way to do that. Alan DeKok.
Hi Alan. thanks for answers.
FR> > is using up to 140% CPU of 48 core system (256GiB RAM, SSD array) I have reniced it and assigned real-time ionice class, the overall CPU of the system is about 20%. That's a huge system...
Yes, IMHO there is no lack of any kind of resources. I am going to try to move FR balancer instance to a different physical server just to be sure.
If you're losing packets at 1000 packets/s, FreeRADIUS is not the cause.
It must be a misunderstanding. The overall drop rate of LB is about 4,6e-05 and drops are finally solved by a retransmission. I was just wonder if there is a way how to detect/log drops or avoid them by config optimisation. There's no real way to do that.
OK It is also strange that the drops (represented by retransmissions) are nearly independent on the average packet rate. I have the same amount of retransmissions at 2 a.m. (1k req/s) and at 4 p.m. (~4k req/s). Which is less than FR could just as a balancer IMHO do. The they appear mainly at the top of the hour where we have spikes of concurrent accounting req. It looks like FR can handle quite high rate of steady load but the bottleneck could be in the amount of concurrent requests in spikes. Regards Ales
On Jun 28, 2017, at 3:05 AM, Aleš Rygl <ales@rygl.net> wrote:
It is also strange that the drops (represented by retransmissions) are nearly independent on the average packet rate.
Because it's an average packet rate. Low packet rates aren't an issue.
I have the same amount of retransmissions at 2 a.m. (1k req/s) and at 4 p.m. (~4k req/s). Which is less than FR could just as a balancer IMHO do.
The they appear mainly at the top of the hour where we have spikes of concurrent accounting req. It looks like FR can handle quite high rate of steady load but the bottleneck could be in the amount of concurrent requests in spikes.
Yes. When the traffic spikes to more than FR can handle, bad things happen. And as I said, FR can do 20K packets/s without an issue. The limiting factor is almost always the database. Alan DeKok.
participants (2)
-
Alan DeKok -
Aleš Rygl