We moved to per-request memory pools in 3.0.8 or so. The benefit is that the server should spend less time doing allocations. The down side seems to be that the server is using more memory. A lot more memory. In some cases, double what it used before. So for 3.0.10 we've decreased the size of the pools, which means that the memory goes back to something reasonable. A related problem is multi-packet EAP sessions. A typical PEAP session can take 10-15 round trips. Due to the possibility of packets being lost, the server caches request and responses for 5 seconds. That makes sense for PAP. It doesn't make sense for PEAP. Why? When PEAP request packet N+1 is received from the client, we *know* that the client has received PEAP reply packet N. So we can clean that reply packet up immediately, instead of sitting on it for another 5 seconds. The result is that the server should only be sitting on *one* live packet, instead of 1 live one, and 10 old ones. The attached patch implements that checking. The new behaviour is automatic, and can't be turned off. I'd appreciate it if people could test the patch, and see if it (a) doesn't break the server, and (b) helps with memory pressure, and (c) doesn't impact performance too much. The patch is against v3.0.x from today (5333d5d179da). I'm wary of putting the patch into 3.0, but it may be a good candidate for 3.1. Alan DeKok.
The other benefit of this code is that it should now be possible (with some more code changes) to detect EAP packets which traverse a different proxy path. i.e. packets go: supplicant -> AP -> RADIUS client -> proxy 1 -> home server 3-4 packets later, proxy 1 fails. We now have packets going: supplicant -> AP -> RADIUS client -> proxy 2 -> home server in the *middle* of an EAP conversation. Right now, the server just says "unknown EAP state", and the auth fails. With a bit more work, we could find the State from the original packet, and echo back the original reply to proxy2, instead of to proxy1. That should help a bit with proxy fail-over. Comments? Stones? On Sep 18, 2015, at 11:10 AM, Alan DeKok <aland@deployingradius.com> wrote:
We moved to per-request memory pools in 3.0.8 or so. The benefit is that the server should spend less time doing allocations. The down side seems to be that the server is using more memory. A lot more memory. In some cases, double what it used before. So for 3.0.10 we've decreased the size of the pools, which means that the memory goes back to something reasonable.
A related problem is multi-packet EAP sessions. A typical PEAP session can take 10-15 round trips. Due to the possibility of packets being lost, the server caches request and responses for 5 seconds. That makes sense for PAP. It doesn't make sense for PEAP.
Why? When PEAP request packet N+1 is received from the client, we *know* that the client has received PEAP reply packet N. So we can clean that reply packet up immediately, instead of sitting on it for another 5 seconds. The result is that the server should only be sitting on *one* live packet, instead of 1 live one, and 10 old ones.
The attached patch implements that checking. The new behaviour is automatic, and can't be turned off. I'd appreciate it if people could test the patch, and see if it (a) doesn't break the server, and (b) helps with memory pressure, and (c) doesn't impact performance too much.
The patch is against v3.0.x from today (5333d5d179da). I'm wary of putting the patch into 3.0, but it may be a good candidate for 3.1.
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/devel.html
Hi,
We moved to per-request memory pools in 3.0.8 or so. The benefit is that the server should spend less time doing allocations. The down side seems to be that the server is using more memory. A lot more memory. In some cases, double what it used before. So for 3.0.10 we've decreased the size of the pools, which means that the memory goes back to something reasonable.
well, define 'a lot' - if this means it cannot run on eg RaspberryPi or OpenWRT systems any more than thats an issue... perhaps the server can check its environment and limit the pool size dynamically if the value of memory is < X ?
A related problem is multi-packet EAP sessions. A typical PEAP session can take 10-15 round trips. Due to the possibility of packets being lost, the server caches request and responses for 5 seconds. That makes sense for PAP. It doesn't make sense for PEAP.
yes.....I've always thought that its simpler to have the whole thing cleaned up but, as you say, if you've had the next part of the conversation then why keep the previous bit...you cannt go back to it
The attached patch implements that checking. The new behaviour is automatic, and can't be turned off. I'd appreciate it if people could test the patch, and see if it (a) doesn't break the server, and (b) helps with memory pressure, and (c) doesn't impact performance too much.
(c) would really mean a lot of heavy benchmarking with EAP testing tools rather than the wishy washy 'it feels slower' from a real life deployment
The patch is against v3.0.x from today (5333d5d179da). I'm wary of putting the patch into 3.0, but it may be a good candidate for 3.1.
but 3.1 is development/test never to be used in production..... (isnt that 3.2 when 3.1 is 'stable/sorted' ?) in which case it will be a long time before things can be really seen in the live world...a bit like a couple of other proposed recent changes. perhaps in 3.0.x with a configure flag? --memory-adjust or such? alan
On Sep 18, 2015, at 12:17 PM, A.L.M.Buxey@lboro.ac.uk wrote:
We moved to per-request memory pools in 3.0.8 or so. The benefit is that the server should spend less time doing allocations. The down side seems to be that the server is using more memory. A lot more memory. In some cases, double what it used before. So for 3.0.10 we've decreased the size of the pools, which means that the memory goes back to something reasonable.
well, define 'a lot' - if this means it cannot run on eg RaspberryPi or OpenWRT systems any more than thats an issue... perhaps the server can check its environment and limit the pool size dynamically if the value of memory is < X ?
Checking the environment is hard to do in a portable manner. The memory change mainly affects people doing high load EAP. And the changes upcoming in 3.0.10 take memory usage back to where it was... little to nothing.
(c) would really mean a lot of heavy benchmarking with EAP testing tools rather than the wishy washy 'it feels slower' from a real life deployment
I have some scripts. I'll take a look.
The patch is against v3.0.x from today (5333d5d179da). I'm wary of putting the patch into 3.0, but it may be a good candidate for 3.1.
but 3.1 is development/test never to be used in production..... (isnt that 3.2 when 3.1 is 'stable/sorted' ?) in which case it will be a long time before things can be really seen in the live world...a bit like a couple of other proposed recent changes. perhaps in 3.0.x with a configure flag? --memory-adjust or such?
Possibly. We'll probably release 3.1 / 3.2 in a year or so. I'd like to stop new development on 3.0, and keep changes to bug fixes. Then, the next release becomes the "experiment with new features" release. Alan DeKok.
Hi,
We'll probably release 3.1 / 3.2 in a year or so. I'd like to stop new development on 3.0, and keep changes to bug fixes. Then, the next release becomes the "experiment with new features" release.
..which would be 3.2.0 ideally - 3.1.x will be at some unknonw number of x and those who have tried/tested things will be ready for 3.2.x production. Then all new stuff goes into 3.3.x - or 4.0.x ? it would be good to keep the odd numbered sub-release (3.1, 3.3, 3.5 as the test/dev track with the 3.0, 3.2 etc being the stable/release..... with some cut-off time for the previous one. how long would support be for 3.0.x after 3.2.x came out for example? alan
On Sep 18, 2015, at 12:34 PM, A.L.M.Buxey@lboro.ac.uk wrote:
We'll probably release 3.1 / 3.2 in a year or so. I'd like to stop new development on 3.0, and keep changes to bug fixes. Then, the next release becomes the "experiment with new features" release.
..which would be 3.2.0 ideally - 3.1.x will be at some unknonw number of x and those who have tried/tested things will be ready for 3.2.x production. Then all new stuff goes into 3.3.x - or 4.0.x ?
Yes.
it would be good to keep the odd numbered sub-release (3.1, 3.3, 3.5 as the test/dev track with the 3.0, 3.2 etc being the stable/release..... with some cut-off time for the previous one. how long would support be for 3.0.x after 3.2.x came out for example?
Probably 3 years, given various OS vendors who shall remain nameless. Alan DeKok.
On Sep 18, 2015, at 12:56 PM, Alan DeKok <aland@deployingradius.com> wrote:
On Sep 18, 2015, at 12:34 PM, A.L.M.Buxey@lboro.ac.uk wrote:
We'll probably release 3.1 / 3.2 in a year or so. I'd like to stop new development on 3.0, and keep changes to bug fixes. Then, the next release becomes the "experiment with new features" release.
..which would be 3.2.0 ideally - 3.1.x will be at some unknonw number of x and those who have tried/tested things will be ready for 3.2.x production. Then all new stuff goes into 3.3.x - or 4.0.x ?
Yes.
it would be good to keep the odd numbered sub-release (3.1, 3.3, 3.5 as the test/dev track with the 3.0, 3.2 etc being the stable/release..... with some cut-off time for the previous one. how long would support be for 3.0.x after 3.2.x came out for example?
Probably 3 years, given various OS vendors who shall remain nameless.
Limited sympathy for those who won't upgrade between point versions. This isn't Windows XP, we're not Microsoft. The changes between v3.0.x and v3.1.x will be minor. https://github.com/FreeRADIUS/freeradius-server/blob/v3.1.x/raddb/README.rst -Arran
participants (3)
-
A.L.M.Buxey＠lboro.ac.uk -
Alan DeKok -
Arran Cudbard-Bell