The "listen" sections in 3.0 have a new entry: max_pps = 300 If that socket receives MORE than that many packets/s, it throws away all new packets. The packet/s counter is tracked in a sliding window over the last second. That means "the number of packets received in the second immediately before receiving the current packet". It's not tracking the number of packets in the current wall-clock second. The feature is mainly intended for accounting sockets. It is *very* expensive to process accounting packets. They can usually be discarded without a problem, as the NAS will retransmit them. The idea is to set the "max_pps" to some fairly large value. Then if the server receives a sudden influx of traffic, it won't die. Instead, it will continue to make progress at a steady rate. The feature has another configuration entry in the "thread pool" section: auto_limit_acct = yes/no If set to "yes", it auto-limits accounting packets. The server maintains a queue of packets for child threads. With auto_limit_acct=yes, it tracks the rate at which packets leave the queue, and the rate at which they enter the queue. If the queue is more than half full, AND the rate of entry is larger than the exit rate, it throws the accounting packet away. All other packet types are unaffected. The idea here is for the server to automatically adapt to overload conditions. If it's getting traffic than it can handle, it starts to throw away that traffic. Suggestions for better heuristics are welcomed. Testing is also welcomed. :) Alan DeKok.
On Wed, Feb 22, 2012 at 05:30:18PM +0100, Alan DeKok wrote:
If the queue is more than half full, AND the rate of entry is larger than the exit rate, it throws the accounting packet away. All other packet types are unaffected.
The idea here is for the server to automatically adapt to overload conditions. If it's getting traffic than it can handle, it starts to throw away that traffic.
Suggestions for better heuristics are welcomed.
How about WRED? e.g. if queue is more than 50% full then randomly drop incoming packets with a probability of (1 - (cur_queue_size/max_queue_size)*2) However the only reason I can see to do this is if auth and acct are in the same queue, so the server gives priority to handling auth if the queue is getting full.
Brian Candler wrote:
e.g. if queue is more than 50% full then randomly drop incoming packets with a probability of (1 - (cur_queue_size/max_queue_size)*2)
That's a good heuristic. The only downside is that the queue is there precisely to handle spikes of traffic. So it might be OK for the queue to be half full. It means that the server will gradually empty it, and therefore be OK.
However the only reason I can see to do this is if auth and acct are in the same queue, so the server gives priority to handling auth if the queue is getting full.
They're all in the same queue, unfortunately. That's why the new configuration option is "auto_limit_acct", so it applies to accounting packets. The addition of the rate-checking means that the packets get dropped ONLY if the queue is growing. If it's shrinking, then there's no reason to drop packets, even if the queue is 75% full. That's pretty much what it does now. I'm not sure what benefits WRED would add. Right now the "drop packets" limit is a hard limit. But dropping packets means that the "add to queue" counter is NOT incremented. The result is a few packets will be dropped, the "input pps" will go below the "output pps", and it will allow more packets to be added to the queue. The result is that the queue will be filled no faster than it's emptied. With WRED, the queue will continue to grow until it's full. I'm not sure how that helps... Alan DeKok.
On 23/02/12 12:04, Alan DeKok wrote:
I'm not sure what benefits WRED would add. Right now the "drop packets" limit is a hard limit. But dropping packets means that the "add to queue" counter is NOT incremented. The result is a few packets will be dropped, the "input pps" will go below the "output pps", and it will allow more packets to be added to the queue.
The result is that the queue will be filled no faster than it's emptied. With WRED, the queue will continue to grow until it's full. I'm not sure how that helps...
Indeed. RED-like behaviour makes sense if the sending system can detect the loss and *slow down*. It's not clear to me that this is the case; the NAS can re-send, and then stop re-sending, but will that actually decrease offered load at the radius server? I'd need to work it out on paper - my brain is full of DWDM at the moment! But it seems to me that, if you wanted RED-like behaviour i.e. reduce offered load at the radius server then the heuristic should be: if queue_len > threshold: compute drop probability if (drop AND Acct-Status-Type==Interim) send immediate Accounting-Response code=ok i.e. shut the NAS up immediately, for interim requests only, and don't execute the server {} block. I think this would be a bad idea. OTOH, it *might* make sense to expose the queue length / percentage as a control: variable to the server{} block; that way, if you wanted to, local sites could do this: preacct { if (control:FreeRADIUS-Acct-QueueLoad > 50) { ok } else { .. } } [It's a shame there isn't an Accounting-Response code "Overloaded"; this could apply some kind of real-time delta to the Acct-Interim-Interval associated with the session(s) on the NAS, But that's rather pie in the sky]
Phil Mayers wrote:
RED-like behaviour makes sense if the sending system can detect the loss and *slow down*. It's not clear to me that this is the case;
RADIUS doesn't do that.
the NAS can re-send, and then stop re-sending, but will that actually decrease offered load at the radius server?
RFC 5080 defines how retransmissions are handled. The NAS *should* add jitter, and perform exponential backoff. Sadly, no one implements it. :(
OTOH, it *might* make sense to expose the queue length / percentage as a control: variable to the server{} block; that way, if you wanted to, local sites could do this:
preacct { if (control:FreeRADIUS-Acct-QueueLoad > 50) { ok } else { .. } }
Nice. <clickety-click> $ git pull And see the last commit log. No documentation or examples as yet. That comes later. :) You should be able to test it by setting "auto_limit_acct = yes" in the "thread pool" section. Then, use if (control:FreeRADIUS-Queue-Use-Percentage > 30) { ... } You won't ever see it greater than 50, due to the previous described behavior of "auto_limit_acct". Maybe the auto_limit_acct check should be set to 75% of the queue size. But my $0.02 is when that happens, the system is pretty much dead anyways.
[It's a shame there isn't an Accounting-Response code "Overloaded"; this could apply some kind of real-time delta to the Acct-Interim-Interval associated with the session(s) on the NAS, But that's rather pie in the sky]
Yeah. It's not going to happen. Alan DeKok.
On Thu, Feb 23, 2012 at 01:04:10PM +0100, Alan DeKok wrote:
I'm not sure what benefits WRED would add.
I think it is more likely to hit a dynamic equilibrium: the % of packets dropped increases proportionately as the queue gets fuller. Otherwise I suspect it will become bursty: queue goes over 50% and is increasing, we start dropping packets, the queue starts to shrink, we start accepting packets for a while, the queue starts to grow, we start dropping packets again... OTOH, we wouldn't get the full benefit of this unless the NASes implement exponential backoff. If they just resend packets every 3 seconds, say, then an overload situation will remain an overload situation anyway.
Dropping packets is enough to recovery as long as the server can cope with the average load over the whole day. Basically you push the bubble along until the load drops enough to recover. How this interacts with health-probes from load balancers and other monitoring platforms would be interesting to consider. Perhaps some source IPs should never be dropped ? On 23 Feb 2012, at 16:07, Brian Candler wrote:
On Thu, Feb 23, 2012 at 01:04:10PM +0100, Alan DeKok wrote:
I'm not sure what benefits WRED would add.
I think it is more likely to hit a dynamic equilibrium: the % of packets dropped increases proportionately as the queue gets fuller.
Otherwise I suspect it will become bursty: queue goes over 50% and is increasing, we start dropping packets, the queue starts to shrink, we start accepting packets for a while, the queue starts to grow, we start dropping packets again...
OTOH, we wouldn't get the full benefit of this unless the NASes implement exponential backoff. If they just resend packets every 3 seconds, say, then an overload situation will remain an overload situation anyway. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/devel.html
Brian Candler wrote:
I think it is more likely to hit a dynamic equilibrium: the % of packets dropped increases proportionately as the queue gets fuller.
That will likely happen, yes.
Otherwise I suspect it will become bursty: queue goes over 50% and is increasing, we start dropping packets, the queue starts to shrink, we start accepting packets for a while, the queue starts to grow, we start dropping packets again...
Why is that a problem? It's also a dynamic equilibrium.
OTOH, we wouldn't get the full benefit of this unless the NASes implement exponential backoff. If they just resend packets every 3 seconds, say, then an overload situation will remain an overload situation anyway.
Exactly. RFC 5080 specifies a backoff method, but no one uses it. Alan DeKok.
participants (4)
-
Alan DeKok -
Alister Winfield -
Brian Candler -
Phil Mayers