One more change for v3 I think
I put a change into master which should probably go into v3, too: https://github.com/FreeRADIUS/freeradius-server/blob/master/src/main/connect... The connection pool re-uses connections based on “most recently used”. But the exact definition of MRU is important. It turns out that “last returned to the pool” is a bad metric. Because the connection could be blocked / slow. And re-using that connection would mean re-using a blocked / slow connection. The solution is to define MRU as “most recently removed from the pool, and returned to the pool”. This change prioritizes connections which are fast. The code is a bit complicated, because it has to keep track of more information. But it should result in systems which are more stable. Alan DeKok.
On Tue, Feb 24, 2015 at 11:58:55AM -0500, Alan DeKok wrote:
I put a change into master which should probably go into v3, too:
https://github.com/FreeRADIUS/freeradius-server/blob/master/src/main/connect...
The connection pool re-uses connections based on “most recently used”. But the exact definition of MRU is important. It turns out that “last returned to the pool” is a bad metric. Because the connection could be blocked / slow. And re-using that connection would mean re-using a blocked / slow connection.
That makes sense - and the FB link is an interesting read. Not something I'd normally think about with link aggregation! So essentially when things start to get busy, a connection may fail quicky, meaning it goes back into the pool and is then picked up again rather than other entries that have been hanging around idle for a while.
The solution is to define MRU as “most recently removed from the pool, and returned to the pool”. This change prioritizes connections which are fast.
The code is a bit complicated, because it has to keep track of more information. But it should result in systems which are more stable.
Quick skim through and it looks OK to me, as long as the code in src/lib/heap.c is well behaved (which I've not stared that hard at!). Cheers, Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
On Feb 24, 2015, at 12:54 PM, Matthew Newton <mcn4@leicester.ac.uk> wrote:
That makes sense - and the FB link is an interesting read. Not something I'd normally think about with link aggregation!
Exactly.
So essentially when things start to get busy, a connection may fail quicky, meaning it goes back into the pool and is then picked up again rather than other entries that have been hanging around idle for a while.
If a connection fails, it’s closed. The issue is connections which *almost* work. They take a long time to do things… which slows everything else down. The connection pool should prioritize *responsive* connections. Alan DeKok.
I assume responsive includes not producing errors or null-responses (bad memory of testing a web service for performance and not spotting it had started to produce 50x replies at an wonderful rate). Okay I understand how hard this is to include in the re-use metric but always worth thinking about if its possible. Alister On 24 Feb 2015, at 18:21, Alan DeKok <aland@deployingradius.com> wrote:
On Feb 24, 2015, at 12:54 PM, Matthew Newton <mcn4@leicester.ac.uk> wrote:
That makes sense - and the FB link is an interesting read. Not something I'd normally think about with link aggregation!
Exactly.
So essentially when things start to get busy, a connection may fail quicky, meaning it goes back into the pool and is then picked up again rather than other entries that have been hanging around idle for a while.
If a connection fails, it’s closed. The issue is connections which *almost* work. They take a long time to do things… which slows everything else down.
The connection pool should prioritize *responsive* connections.
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/devel.html
On Feb 25, 2015, at 5:35 AM, Alister Winfield <alister@ticklers.org> wrote:
I assume responsive includes not producing errors or null-responses (bad memory of testing a web service for performance and not spotting it had started to produce 50x replies at an wonderful rate). Okay I understand how hard this is to include in the re-use metric but always worth thinking about if its possible.
*Query* failures are different from *connection* failures. If the query fails, that’s OK. The data might not exist in SQL. Both the connection and the database are OK. If the connection fails, then the connection might be bad. It’s best to close the bad connection, and re-open it. When a connection is “not responsive”, it falls into neither category. :( The SQL query returns something (eventually), the connection is still active… but not necessarily OK. There may be network effects which cause the connection to be unresponsive. It’s best to keep that connection open, because it *is* working. But it’s also best to avoid using it if at all possible. If it’s left unused for a while, it will be closed due to the idle timeout, and a new connection will open. The hope is that new connection doesn’t suffer from the same problems as the unresponsive one. Alan DeKok.
On 25 Feb 2015, at 09:39, Alan DeKok <aland@deployingradius.com> wrote:
On Feb 25, 2015, at 5:35 AM, Alister Winfield <alister@ticklers.org> wrote:
I assume responsive includes not producing errors or null-responses (bad memory of testing a web service for performance and not spotting it had started to produce 50x replies at an wonderful rate).
I'm assuming this is through a load balancer of some kind, with sticky connections? That's quite interesting. 500 is a genuine server is fubar error. Not all protocols have that kind of signalling. You don't want to re-establish the connection, but de-prioritising it might be an idea. If the 500 error is occurring across all connections they all get de-prioritised equally. Might be worth adding a function to the API to allow the module to say that it thinks the connection or the thing at the end of it is screwed. -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
Alan DeKok <aland@deployingradius.com> wrote: > The solution is to define MRU as “most recently removed from the > pool, and returned to the pool”. This change prioritizes connections > which are fast. This might also have better cache properties.
participants (5)
-
Alan DeKok -
Alister Winfield -
Arran Cudbard-Bell -
Matthew Newton -
Michael Richardson