Hi, I'm trying to setup a robust RADIUS authentication proxy. All this radius will do is proxy all auth requests to a set of four backend RADIUS handlers. I have a 2.1.6 server that I've configured with four home_server entries and one home_server_pool that load-balances across the four. It works when all four backends are up, but if any 1 of the backend goes down, then requests that get directed to that backend result in an Access-Reject packet being returned to the NAS. Is there a way to configure freeradius so that instead of returning an Access- Reject packet, the server will instead switch to the next configured server and retry the request there? It may mean that it takes a little longer for the request to be handled, but that's better than it being rejected. Thanks for any advice or assistance. Philip
I'm trying to setup a robust RADIUS authentication proxy. All this radius will do is proxy all auth requests to a set of four backend RADIUS handlers. I have a 2.1.6 server that I've configured with four home_server entries and one home_server_pool that load-balances across the four. It works when all four backends are up, but if any 1 of the backend goes down, then requests that get directed to that backend result in an Access-Reject packet being returned to the NAS. Is there a way to configure freeradius so that instead of returning an Access- Reject packet, the server will instead switch to the next configured server and retry the request there? It may mean that it takes a little longer for the request to be handled, but that's better than it being rejected.
No, but you can enable do_not_respond policy (see policy.conf). Server then won't respond to the NAS. That should result in repeated request which should (chances are) end up with different home server. This would be in effect during zombie period. Once the home server is marked dead no requests will go to it. Ivan Kalik Kalik Informatika ISP
On Jul 10, 2009, at 4:05 AM, Ivan Kalik wrote:
I'm trying to setup a robust RADIUS authentication proxy. All this radius will do is proxy all auth requests to a set of four backend RADIUS handlers. I have a 2.1.6 server that I've configured with four home_server entries and one home_server_pool that load-balances across the four. It works when all four backends are up, but if any 1 of the backend goes down, then requests that get directed to that backend result in an Access-Reject packet being returned to the NAS. Is there a way to configure freeradius so that instead of returning an Access- Reject packet, the server will instead switch to the next configured server and retry the request there? It may mean that it takes a little longer for the request to be handled, but that's better than it being rejected.
No, but you can enable do_not_respond policy (see policy.conf). Server then won't respond to the NAS. That should result in repeated request which should (chances are) end up with different home server. This would be in effect during zombie period. Once the home server is marked dead no requests will go to it.
Yeah,that's what I'm doing. The problem is that the retries are not being sent to a different home server (or any home server). They are being dropped as retransmits because internally, freeradius is tracking that no reply was sent to them earlier. I have tried treaking cleanup_delay to 0 or 1 to flush these out sooner, but it does not work -- they do not appear to be tracked the same way as normal responses. Here are the debug messages from radiusd -X: rad_recv: Access-Request packet from host 127.0.0.1 port 47163, id=155, length=59 Ignoring retransmit from client SERVERS port 47163 - ID: 155, no reply was configured Is there any way to prevent an ignored response from being tracked this way so that retransmits will be treated as new requests? Or am I just not sending enough retransmits? Philip
Yeah,that's what I'm doing. The problem is that the retries are not being sent to a different home server (or any home server). They are being dropped as retransmits because internally, freeradius is tracking that no reply was sent to them earlier. I have tried treaking cleanup_delay to 0 or 1 to flush these out sooner, but it does not work -- they do not appear to be tracked the same way as normal responses. Here are the debug messages from radiusd -X:
rad_recv: Access-Request packet from host 127.0.0.1 port 47163, id=155, length=59 Ignoring retransmit from client SERVERS port 47163 - ID: 155, no reply was configured
Yes, length of that is controlled by response_window. Server will ignore retransmits while waiting for response. If you shorten response_window home server will be marked as zombie faster. Ivan Kalik Kalik Informatika ISP
Ivan Kalik wrote:
Yeah,that's what I'm doing. The problem is that the retries are not being sent to a different home server (or any home server). They are being dropped as retransmits because internally, freeradius is tracking that no reply was sent to them earlier. I have tried treaking cleanup_delay to 0 or 1 to flush these out sooner, but it does not work -- they do not appear to be tracked the same way as normal responses. Here are the debug messages from radiusd -X:
rad_recv: Access-Request packet from host 127.0.0.1 port 47163, id=155, length=59 Ignoring retransmit from client SERVERS port 47163 - ID: 155, no reply was configured
Yes, length of that is controlled by response_window. Server will ignore retransmits while waiting for response. If you shorten response_window home server will be marked as zombie faster.
I must be missing something, because even after the home_server has been marked as a zombie, the proxy is still ignoring the retransmits. Furthermore, it's taking much longer than the response_window for the home_server to be marked as a zombie. I have a response_window of 1, trying to force the home_server to be marked zombie as fast as possible. Here are the log messages (I've stripped out test packet contents) for the three client attempts using radtest, which sends 3 packets for a total processing time of 15 seconds: rad_recv: Access-Request packet from host 127.0.0.1 port 39091, id=56, length=59 +- entering group authorize {...} ++[control] returns notfound +- entering group pre-proxy {...} [attr_filter.pre-proxy] expand: %{Realm} -> DEFAULT attr_filter: Matched entry DEFAULT at line 50 ++[attr_filter.pre-proxy] returns updated Sending Access-Request of id 175 to xxx.xxx.xxx.12 port 1812 Proxying request 0 to home server xxx.xxx.xxx.12 port 1812 Sending Access-Request of id 175 to xxx.xxx.xxx.12 port 1812 Going to the next request Waking up in 0.9 seconds. Waking up in 3.9 seconds. rad_recv: Access-Request packet from host 127.0.0.1 port 39091, id=56, length=59 Sending duplicate proxied request to home server xxx.xxx.xxx.12 port 1812 - ID: 175 Sending Access-Request of id 175 to xxx.xxx.xxx.12 port 1812 Rejecting request 0 due to lack of any response from home server xxx.xxx.xxx.12 port 1812 Found Post-Proxy-Type +- entering group Fail {...} ++[control] returns noop ++- entering policy do_not_respond {...} +++[control] returns noop +++[handled] returns handled ++- policy do_not_respond returns handled Going to the next request PROXY: Marking home server xxx.xxx.xxx.12 port 1812 as zombie (it looks like it is dead). Sending Status-Server of id 81 to xxx.xxx.xxx.12 port 1812 Message-Authenticator := 0x00000000000000000000000000000000 NAS-Identifier := "Status Check. Are you alive?" Waking up in 3.9 seconds. Waking up in 3.9 seconds. rad_recv: Access-Request packet from host 127.0.0.1 port 39091, id=56, length=59 Ignoring retransmit from client SERVERS port 39091 - ID: 56, no reply was configured Waking up in 2.9 seconds. Sending Status-Server of id 37 to xxx.xxx.xxx.12 port 1812 Message-Authenticator := 0x00000000000000000000000000000000 NAS-Identifier := "Status Check. Are you alive?" Waking up in 3.9 seconds. Waking up in 1.6 seconds.
Philip Molter wrote:
I must be missing something, because even after the home_server has been marked as a zombie, the proxy is still ignoring the retransmits.
Yes... see the debug log. You have configured the "do_not_respond" policy. As a result, the server doesn't respond to retransmits.
Furthermore, it's taking much longer than the response_window for the home_server to be marked as a zombie.
Yes. The server doesn't set up timers for each individual packet that mark the home server as unresponsive. That's just too hard. Instead, it waits for the client to retransmit the packet, and THEN the server applies the policies.
I have a response_window of 1, trying to force the home_server to be marked zombie as fast as possible.
See the documentation in raddb/proxy.conf. It clearly says that the minimum useful value is 5. Even that is likely to be small. And the server will NOT fail over during the zombie period. It can't. It has no idea if the home server is really down or not.
Here are the log messages (I've stripped out test packet contents) for the three client attempts using radtest, which sends 3 packets for a total processing time of 15 seconds: ... rad_recv: Access-Request packet from host 127.0.0.1 port 39091, id=56, length=59 Sending duplicate proxied request to home server xxx.xxx.xxx.12 port 1812 - ID: 175
As expected...
Sending Access-Request of id 175 to xxx.xxx.xxx.12 port 1812 Rejecting request 0 due to lack of any response from home server xxx.xxx.xxx.12 port 1812
But there's no response
Found Post-Proxy-Type +- entering group Fail {...} ++[control] returns noop ++- entering policy do_not_respond {...}
So you say DO NOT RESPOND TO THIS PACKET.
+++[control] returns noop +++[handled] returns handled ++- policy do_not_respond returns handled Going to the next request PROXY: Marking home server xxx.xxx.xxx.12 port 1812 as zombie (it looks like it is dead).
And the home server is marked dead, as you want.
Sending Status-Server of id 81 to xxx.xxx.xxx.12 port 1812 Message-Authenticator := 0x00000000000000000000000000000000 NAS-Identifier := "Status Check. Are you alive?" Waking up in 3.9 seconds. Waking up in 3.9 seconds. rad_recv: Access-Request packet from host 127.0.0.1 port 39091, id=56, length=59 Ignoring retransmit from client SERVERS port 39091 - ID: 56, no reply was configured
And the server doesn't respond, because that's what you told it to do. I think the main issue here is that you're expecting RADIUS to be a reliable and robust transport protocol. It's not. It just doesn't work that way. There will ALWAYS be packets in "limbo" when a home server goes down. It takes time to determine that the server is down, and during that time, all of the packets sent to it are in "limbo". When the server decides that the home server is down, it WILL retransmit the packets to a backup server. This is documented in proxy.conf. However, it MAY timeout some packets, because it took too long for the home server to respond. There is really very little you can do to work around this problem. FreeRADIUS *will* fail over to a backup home server. That's what the "fail-over" configuration is for in the home server pools. But it takes time to figure this out. How do you propose that the proxy determine that a home server is down, without taking any time to do so, and without making any false positive errors? Alan DeKok.
I must be missing something, because even after the home_server has been marked as a zombie, the proxy is still ignoring the retransmits. Furthermore, it's taking much longer than the response_window for the home_server to be marked as a zombie.
rad_recv: Access-Request packet from host 127.0.0.1 port 39091, id=56, length=59 +- entering group authorize {...} ++[control] returns notfound +- entering group pre-proxy {...} [attr_filter.pre-proxy] expand: %{Realm} -> DEFAULT attr_filter: Matched entry DEFAULT at line 50 ++[attr_filter.pre-proxy] returns updated Sending Access-Request of id 175 to xxx.xxx.xxx.12 port 1812 Proxying request 0 to home server xxx.xxx.xxx.12 port 1812 Sending Access-Request of id 175 to xxx.xxx.xxx.12 port 1812 Going to the next request Waking up in 0.9 seconds. Waking up in 3.9 seconds. rad_recv: Access-Request packet from host 127.0.0.1 port 39091, id=56, length=59 Sending duplicate proxied request to home server xxx.xxx.xxx.12 port 1812 - ID: 175 Sending Access-Request of id 175 to xxx.xxx.xxx.12 port 1812 Rejecting request 0 due to lack of any response from home server xxx.xxx.xxx.12 port 1812 Found Post-Proxy-Type +- entering group Fail {...} ++[control] returns noop ++- entering policy do_not_respond {...}
That's why!
+++[control] returns noop +++[handled] returns handled ++- policy do_not_respond returns handled Going to the next request PROXY: Marking home server xxx.xxx.xxx.12 port 1812 as zombie (it looks like it is dead). Sending Status-Server of id 81 to xxx.xxx.xxx.12 port 1812 Message-Authenticator := 0x00000000000000000000000000000000 NAS-Identifier := "Status Check. Are you alive?" Waking up in 3.9 seconds. Waking up in 3.9 seconds. rad_recv: Access-Request packet from host 127.0.0.1 port 39091, id=56, length=59 Ignoring retransmit from client SERVERS port 39091 - ID: 56, no reply was configured
I can see your point. You would like to argue that the request should be taken of the list even if no response was configured - since server didn't respond because of the do_not_respond policy. I am not sure that can be made to work. Ivan Kalik Kalik Informatika ISP
Alan and Ivan, Thanks for your patience with this. I'm migrating from an old RADIUS platform that supports this behavior to freeradius, and I'm just trying to make sure I get everything working. Ivan Kalik wrote:
rad_recv: Access-Request packet from host 127.0.0.1 port 39091, id=56, length=59 Ignoring retransmit from client SERVERS port 39091 - ID: 56, no reply was configured
I can see your point. You would like to argue that the request should be taken of the list even if no response was configured - since server didn't respond because of the do_not_respond policy. I am not sure that can be made to work.
What I really want is just, instead of the request being marked as failed when one of the home servers doesn't respond, for the proxy subsystem to just try sending the request to another configured home server. If the proxy has tried sending a request to every non-zombie home server in the list and still hasn't gotten anything, then it can mark the request as failed. The way I originally thought it was going to work is similar to how modules are load-balanced. If I have five SQL servers loaded through 5 named SQL module configs, it will try the first, then the second, then the third until one of them returns success. It would be great if the proxy load-balancing could work the same way. Philip
Philip Molter wrote:
Thanks for your patience with this. I'm migrating from an old RADIUS platform that supports this behavior to freeradius, and I'm just trying to make sure I get everything working.
What behavior? Failover from one home server to another? FreeRADIUS does this already. I think what you want is to have the re-transmits switch from one home server to another *before* the first one has been marked dead. This is difficult to do automatically. Something like "send retransmits to a backup server" is possible, but can have cause other problems. But you can use "radmin" to do this manually.
What I really want is just, instead of the request being marked as failed when one of the home servers doesn't respond, for the proxy subsystem to just try sending the request to another configured home server.
But it already does that. Run the server, and watch how it behaves. As I said before, the difficulty is determining *when* to do this failover.
If the proxy has tried sending a request to every non-zombie home server in the list and still hasn't gotten anything, then it can mark the request as failed.
Sorry, but it takes time to determine that a home server has failed. By the time this decision has been made for 2-3 home servers, 30 seconds have usually passed, and the NAS has given up on the request.
The way I originally thought it was going to work is similar to how modules are load-balanced. If I have five SQL servers loaded through 5 named SQL module configs, it will try the first, then the second, then the third until one of them returns success. It would be great if the proxy load-balancing could work the same way.
Unless I'm really missing something, it already does this. Just configure "type = load-balance" in the home server pool. Have you done this? What do you expect the proxy to do with requests sent to a home server that *might* be down? How should the proxy decide that the home server is down? Be specific. Draw flow diagrams... If you can come up with a better algorithm, then by all means we'll implement it. But coming up with an algorithm that works *well* from limited information is hard. The issue with your configuration is that you are trying valiantly to game the system. You're setting the timers *way* too low, and the marking the requests as failed too early. When the NAS retransmits, you claim you want the proxy to fail over to another server... AFTER you've already told it to give up on the request. Your configuration is contradicting your stated needs. Fix one or the other so that there is no contradiction. Alan DeKok.
Alan DeKok wrote:
Philip Molter wrote:
Thanks for your patience with this. I'm migrating from an old RADIUS platform that supports this behavior to freeradius, and I'm just trying to make sure I get everything working.
What behavior? Failover from one home server to another? FreeRADIUS does this already.
I think what you want is to have the re-transmits switch from one home server to another *before* the first one has been marked dead. This is difficult to do automatically. Something like "send retransmits to a backup server" is possible, but can have cause other problems.
But you can use "radmin" to do this manually.
What I really want is just, instead of the request being marked as failed when one of the home servers doesn't respond, for the proxy subsystem to just try sending the request to another configured home server.
But it already does that. Run the server, and watch how it behaves. As I said before, the difficulty is determining *when* to do this failover.
If the proxy has tried sending a request to every non-zombie home server in the list and still hasn't gotten anything, then it can mark the request as failed.
Sorry, but it takes time to determine that a home server has failed. By the time this decision has been made for 2-3 home servers, 30 seconds have usually passed, and the NAS has given up on the request.
The way I originally thought it was going to work is similar to how modules are load-balanced. If I have five SQL servers loaded through 5 named SQL module configs, it will try the first, then the second, then the third until one of them returns success. It would be great if the proxy load-balancing could work the same way.
Unless I'm really missing something, it already does this. Just configure "type = load-balance" in the home server pool.
Have you done this?
Yes, this is the configuration I'm currently running, and it's not working for me. I have a radclient sending a request, retrying 10 times on a 5-second timer, and after 10 retries, it still hasn't gotten a response. After the second retry, the proxy has marked the server as at least a zombie and started status-checks, but every retransmit after that is getting a cached result of no response.
What do you expect the proxy to do with requests sent to a home server that *might* be down? How should the proxy decide that the home server is down? Be specific. Draw flow diagrams...
This is what I want to happen client req -> proxy proxy req -> home server #1 client ret -> proxy proxy ret -> home server #1 [proxy fails home server #1 for lack of response] client ret -> proxy proxy req -> home server #2 proxy <- resp home server #2 client <- resp proxy This is what is happening with my post-proxy config: client req -> proxy proxy req -> home server #1 client ret -> proxy proxy ret -> home server #1 [proxy fails home server #1 for lack of response] client ret -> proxy [proxy detects retransmit, does nothing] client ret -> proxy [proxy detects retransmit, does nothing] client ret -> proxy [proxy detects retransmit, does nothing] ... This is what happens without a post-proxy config: client req -> proxy proxy req -> home server #1 client ret -> proxy proxy ret -> home server #1 [proxy fails home server #1 for lack of response] client <- rej proxy
If you can come up with a better algorithm, then by all means we'll implement it. But coming up with an algorithm that works *well* from limited information is hard.
The issue with your configuration is that you are trying valiantly to game the system. You're setting the timers *way* too low, and the marking the requests as failed too early. When the NAS retransmits, you claim you want the proxy to fail over to another server... AFTER you've already told it to give up on the request.
My config is not marking any request as failed. If I do not configure anything for Post-Proxy-Type, I get back an Access-Reject right when the first home server fails. There is no failover. The comments in proxy.conf make that clear: # If the home server doesn't respond to the request within # this time, this server will consider the request dead, and # respond to the NAS with an Access-Reject. In other words, if the server the load-balance solution happens to choose doesn't respond to my request, tough luck. I might have 19 other servers configured that are up, the request I just sent is getting an Access-Reject. The Post-Proxy-Type is just a hack to at least not send back an Access-Reject which breaks the whole process.
Your configuration is contradicting your stated needs. Fix one or the other so that there is no contradiction.
Okay, so I obviously do not understand how I can tweak response_window and zombie_period to make sure that requests that can be serviced by many possible RADIUS home servers do not return an Access-Reject when one of those home servers does not respond. Here are my stated needs. The client sends a request to the proxy. If a home server does not respond within a short period of time to the request, a second home server is chosen. If the second home server does not respond to the same request, then a third is chosen. This continues until all possible home servers are exhausted. At that point, an Access-Reject packet is sent back to the client. Otherwise, the response from the home server is sent back to the client. How do I configure that? It doesn't seem to matter what I set response_window or zombie_period to, once the first home server fails to respond, an Access-Reject (or nothing if I configure a post-proxy handler) is returned to the client. My client's not going to retry the request if he gets an Access-Reject, so I need the proxy to retry it. Is that possible? Philip
Yes, this is the configuration I'm currently running, and it's not working for me. I have a radclient sending a request, retrying 10 times on a 5-second timer, and after 10 retries, it still hasn't gotten a response. After the second retry, the proxy has marked the server as at least a zombie and started status-checks, but every retransmit after that is getting a cached result of no response.
What do you expect the proxy to do with requests sent to a home server that *might* be down? How should the proxy decide that the home server is down? Be specific. Draw flow diagrams...
This is what I want to happen
client req -> proxy proxy req -> home server #1 client ret -> proxy proxy ret -> home server #1 [proxy fails home server #1 for lack of response] client ret -> proxy proxy req -> home server #2 proxy <- resp home server #2 client <- resp proxy
You were told what to do: patch the source code so the request is removed from the list if do_not_respond policy is enabled. At present server can either respond or not respond to the request. It can't pretend it never recieved it as things stand now.
How do I configure that?
It can't be configured. Source needs to be patched.
My client's not going to retry the request if he gets an Access-Reject, so I need the proxy to retry it.
What makes you think that? Users tend to retry dozen or so times with wrong username/password/exired acount before they opt to try another ISP (in my case they have another 150 to choose from) or call the helpline. Small minority will actually check their account settings themselves and almost nobody will try only once and then give up - not unless they have tried and failed a few times previously. Ivan Kalik Kalik Informatika ISP
On Jul 10, 2009, at 5:23 PM, Ivan Kalik wrote:
Yes, this is the configuration I'm currently running, and it's not working for me. I have a radclient sending a request, retrying 10 times on a 5-second timer, and after 10 retries, it still hasn't gotten a response. After the second retry, the proxy has marked the server as at least a zombie and started status-checks, but every retransmit after that is getting a cached result of no response.
What do you expect the proxy to do with requests sent to a home server that *might* be down? How should the proxy decide that the home server is down? Be specific. Draw flow diagrams...
This is what I want to happen
client req -> proxy proxy req -> home server #1 client ret -> proxy proxy ret -> home server #1 [proxy fails home server #1 for lack of response] client ret -> proxy proxy req -> home server #2 proxy <- resp home server #2 client <- resp proxy
You were told what to do: patch the source code so the request is removed from the list if do_not_respond policy is enabled. At present server can either respond or not respond to the request. It can't pretend it never recieved it as things stand now.
I did not see your message until after Alan's.
How do I configure that?
It can't be configured. Source needs to be patched.
My client's not going to retry the request if he gets an Access-Reject, so I need the proxy to retry it.
What makes you think that? Users tend to retry dozen or so times with wrong username/password/exired acount before they opt to try another ISP (in my case they have another 150 to choose from) or call the helpline. Small minority will actually check their account settings themselves and almost nobody will try only once and then give up - not unless they have tried and failed a few times previously.
I'm not using RADIUS as a backend for ISP gear. I am using a RADIUS proxy to serve requests for service software, and when false failures come back, customers get error boxes in their software and contact our support angry that our authentications are returning transient errors. Furthermore, I consider it bad public face to return errors to customers when they should not get them. Yes, customers can always retry, but we can also retry for them when know the reason is not due to invalid information. Thanks for your help guys. I do not want to sit there and beat a dead horse. I will either patch the software or go with another piece of software.
I'm not using RADIUS as a backend for ISP gear. I am using a RADIUS proxy to serve requests for service software, and when false failures come back, customers get error boxes in their software and contact our support angry that our authentications are returning transient errors. Furthermore, I consider it bad public face to return errors to customers when they should not get them. Yes, customers can always retry, but we can also retry for them when know the reason is not due to invalid information.
I think that you are going about it the wrong way. You wont proxy to pretend that home server has not gone down. How about this - instead of a group of stand-alone load-balanced home servers create a (true) high availability cluster. If your home server is always available this issue doesn't come up. And your customer always gets a response. Ivan Kalik Kalik Informatika ISP
On Jul 11, 2009, at 12:14 PM, Ivan Kalik wrote:
I'm not using RADIUS as a backend for ISP gear. I am using a RADIUS proxy to serve requests for service software, and when false failures come back, customers get error boxes in their software and contact our support angry that our authentications are returning transient errors. Furthermore, I consider it bad public face to return errors to customers when they should not get them. Yes, customers can always retry, but we can also retry for them when know the reason is not due to invalid information.
I think that you are going about it the wrong way. You wont proxy to pretend that home server has not gone down. How about this - instead of a group of stand-alone load-balanced home servers create a (true) high availability cluster. If your home server is always available this issue doesn't come up. And your customer always gets a response.
Well, if I get the proxy handling to function the way I am envisioning, I effectively create a high-availability cluster with the proxy as my availability manager. :) But why not setup a high-availability cluster as a home server? First, I already have an existing pool of dumb home servers that I would like to continue using. Second, those home servers are incredibly cheap and easily replaceable. A high-availability cluster probably would not be. Third, if my home servers start having issues with the load, the easiest thing to do to just add more dumb home servers and update the proxies to spread that load out across the new ones in addition to the old ones. Easy scaling. And why use a proxy in the first place? I can use that proxy to work around a bunch of different NASes not having the ability to use a pool of home servers. I do not want the proxy to pretend that the home server has not gone down (in fact, it very much needs to accept that any individual home server may be down). I want it to hide the fact that a single home server is not responding and not have that result in the entire pool appearing to have gone down (if only for a single request). NASes already handle the unreliability in the network by retransmitting packets. I can have the proxy use that to its advantage by not giving the NAS any clue that a single home server in the pool did not respond to a request in a timely fashion (do this by sending retransmits to another server). A proxy that does not respond to the initial request because the initial request never got responded to by the home server, but then responds to subsequent retransmits of that request because the proxy transmitted them to a different home server that was up and responding just appears as a slow RADIUS server to the NAS. My customers do not really complain about login taking a long time (30s, etc.), but they really complain when their client tells them their login is not valid when they know it is. Philip
I think that you are going about it the wrong way. You wont proxy to pretend that home server has not gone down. How about this - instead of a group of stand-alone load-balanced home servers create a (true) high availability cluster. If your home server is always available this issue doesn't come up. And your customer always gets a response.
Well, if I get the proxy handling to function the way I am envisioning, I effectively create a high-availability cluster with the proxy as my availability manager. :)
As you have seen it's not straightforward.
But why not setup a high-availability cluster as a home server? First, I already have an existing pool of dumb home servers that I would like to continue using.
Not an issue. They would just work in a cluster.
Second, those home servers are incredibly cheap and easily replaceable. A high-availability cluster probably would not be.
Ahem. It can be just as cheap, since you probably already have it. http://www.linux-ha.org/
Third, if my home servers start having issues with the load, the easiest thing to do to just add more dumb home servers and update the proxies to spread that load out across the new ones in addition to the old ones. Easy scaling.
Again, not an issue. You just join additional server(s) to the cluster. It's even easier since you don't have to make any changes on the proxywhen adding new servers to the cluster.
I do not want the proxy to pretend that the home server has not gone down (in fact, it very much needs to accept that any individual home server may be down). I want it to hide the fact that a single home server is not responding and not have that result in the entire pool appearing to have gone down (if only for a single request).
That is exactly what high availability solution is about. No matter how many component servers fail it will work as long as there is one still responding. Ivan Kalik Kalik Informatika ISP
On Jul 11, 2009, at 3:55 PM, Ivan Kalik wrote:
I think that you are going about it the wrong way. You wont proxy to pretend that home server has not gone down. How about this - instead of a group of stand-alone load-balanced home servers create a (true) high availability cluster. If your home server is always available this issue doesn't come up. And your customer always gets a response.
Well, if I get the proxy handling to function the way I am envisioning, I effectively create a high-availability cluster with the proxy as my availability manager. :)
As you have seen it's not straightforward.
Second, those home servers are incredibly cheap and easily replaceable. A high-availability cluster probably would not be.
Ahem. It can be just as cheap, since you probably already have it.
Linux HA is a great project. If I was only dealing with a local situation, I would consider it. My solution is going to be working with my home servers, but also home servers beyond my control. I did not mean to imply I am trying to implement an HA RADIUS cluster through this discussion. I am not. It is just a side-effect of the process. Given what Alan's talked above, I believe I have a very small patch that allows for more flexible handling of no-response situations. Hopefully it will a) work and b) be palatable to the freeradius team. Philip
Philip Molter wrote:
Yes, this is the configuration I'm currently running, and it's not working for me. I have a radclient sending a request, retrying 10 times on a 5-second timer, and after 10 retries, it still hasn't gotten a response. After the second retry, the proxy has marked the server as at least a zombie and started status-checks, but every retransmit after that is getting a cached result of no response.
Could you possibly try READING my messages? The default configuration does NOT include the "do_not_respond" policy. *YOU* are the one who configured that, as I have said multiple times. If you don't want it to get the cached "do not respond", policy, then DON'T CONFIGURE IT It's that easy.
This is what I want to happen
client req -> proxy proxy req -> home server #1 client ret -> proxy proxy ret -> home server #1 [proxy fails home server #1 for lack of response] client ret -> proxy proxy req -> home server #2 proxy <- resp home server #2 client <- resp proxy
It does that (mostly). But only if you don't break the server.
This is what happens without a post-proxy config:
client req -> proxy proxy req -> home server #1 client ret -> proxy proxy ret -> home server #1 [proxy fails home server #1 for lack of response] client <- rej proxy
That happens for the most part because you played with the configuration to make the proxy timeouts super-short. As I said, don't do that. And as I also said, it takes time to determine that a home server is dead. During this time, the request MAY time out. When a request times out, the NAS has likely given up on it, so failing over to another home server is useless.
My config is not marking any request as failed. If I do not configure anything for Post-Proxy-Type, I get back an Access-Reject right when the first home server fails. There is no failover. The comments in proxy.conf make that clear:
Yes... that's for ONE request. Most proxies handle more than one request during any 30-60 second period. The OTHER requests will fail over to other home servers, so long as they are still within their individual lifetime.
In other words, if the server the load-balance solution happens to choose doesn't respond to my request, tough luck.
And I explained why, and how this happens. Did you read those explanations?
I might have 19 other servers configured that are up, the request I just sent is getting an Access-Reject.
Again... because it takes TIME to determine that the proxy is down... and during that time, the request times out.
The Post-Proxy-Type is just a hack to at least not send back an Access-Reject which breaks the whole process.
Yes... because the "do not respond" policy tells it DO NOT RESPOND. If you want the server to respond, don't configure the DO NOT RESPOND. That much should be pretty obvious.
Okay, so I obviously do not understand how I can tweak response_window and zombie_period to make sure that requests that can be serviced by many possible RADIUS home servers do not return an Access-Reject when one of those home servers does not respond.
i.e. you want NO request to fail processing when a home server fails. This is extremely difficult to do. Any naive approach that has quick failover can have other negative side-effects. (Additional network traffic, system load, duplicate processing of requests, etc.)
The client sends a request to the proxy. If a home server does not respond within a short period of time to the request, a second home server is chosen. If the second home server does not respond to the same request, then a third is chosen. This continues until all possible home servers are exhausted. At that point, an Access-Reject packet is sent back to the client. Otherwise, the response from the home server is sent back to the client.
Doing that requires source code mods, because that quick fail-over can have negative side-effects. i.e. The server does NOT support configurations that can negatively affect it's performance. On top of that, the "try all possible home servers" is impossible. There is ALSO a 30 second lifetime for the request. After 30 seconds, the NAS has given up, so failing over to another home server is useless. On top of that, the NAS will only retry 3-6 times. So if you have 19 home servers, at *best* it would fail over to 3-6 of them, before the request is marked "timeout". I sincerely hope you see now that the situation is rather more complicated than the simple "try all home server" statement.
How do I configure that? It doesn't seem to matter what I set response_window or zombie_period to, once the first home server fails to respond, an Access-Reject (or nothing if I configure a post-proxy handler) is returned to the client. My client's not going to retry the request if he gets an Access-Reject, so I need the proxy to retry it.
That last sentence is nonsense. Once the client gets an Access-Reject for *any* reason, it is impossible for the proxy to "retry" that request. If you want the proxy to fail over, send it more than ONE request at a time (like a normal proxying system), and do NOT configure the "do not respond" policy. The proxy WILL fail over, but due to the imperfect nature of the universe, some requests MAY time out and get rejected. With a better detection algorithm, the number of failures might get smaller than it is today, but it is IMPOSSIBLE to get the number down to zero.
Is that possible?
No. RADIUS doesn't work like that. No amount of magic on the proxy will cause the NAS to retry forever (which is the only way to have the proxy cycle through all home servers for one request). If you configure the NAS to retry forever, then all you will do is push network failures off to some other part of the network. This is how IP connectivity works: Networks are imperfect. There is absolutely nothing you can do about that. Alan DeKok.
On Jul 11, 2009, at 2:14 AM, Alan DeKok wrote:
Philip Molter wrote:
Yes, this is the configuration I'm currently running, and it's not working for me. I have a radclient sending a request, retrying 10 times on a 5-second timer, and after 10 retries, it still hasn't gotten a response. After the second retry, the proxy has marked the server as at least a zombie and started status-checks, but every retransmit after that is getting a cached result of no response.
Could you possibly try READING my messages?
The default configuration does NOT include the "do_not_respond" policy. *YOU* are the one who configured that, as I have said multiple times.
If you don't want it to get the cached "do not respond", policy, then
DON'T CONFIGURE IT
It's that easy.
I do not want to get ANY cached response. I do not want to get any Access-Reject. If I do not configure a 'do not respond' response, I get an Access-Reject, which is even worse because my end-client gets an error when he should not. What I want is for a no-response from a home server to be treated as a no-response to the NAS, and the subsequent retransmit from the NAS to be processed as a retransmit to a different home server.
This is what I want to happen
client req -> proxy proxy req -> home server #1 client ret -> proxy proxy ret -> home server #1 [proxy fails home server #1 for lack of response] client ret -> proxy proxy req -> home server #2 proxy <- resp home server #2 client <- resp proxy
It does that (mostly). But only if you don't break the server.
No, it does not do that all. I have yet to see a retransmit from a client actually get tried on a different server than the one used for the original request. Once the proxy fails to receive a response from the originally chosen home-server, it handles the packet as a failure. If it sends back an Access-Reject packet, the request is rejected by the NAS to the client and that NAS stops retrying THAT REQUEST (ie. the end-client gets an error). If I add a configuration to not send back anything, then the NAS will retransmit, but like you have made abundantly clear, the proxy remembers that you sent back no response to the original request and skips all further processing of the retransmit. I have set response_windows and zombie_periods to minimums. I have set response_windows and zombie_periods to maximums. For a given single request, only one home server is tried, and if that home server is down, the request and any other retransmits of that request will not succeed. Yes, if the NAS sends another separate request with a different ID, it will be proxied to a different home server, but that does not help the poor guy who had the hard luck of his request hitting the bad home server. He will get an error message. He will have to retry or call support or whatever.
This is what happens without a post-proxy config:
client req -> proxy proxy req -> home server #1 client ret -> proxy proxy ret -> home server #1 [proxy fails home server #1 for lack of response] client <- rej proxy
That happens for the most part because you played with the configuration to make the proxy timeouts super-short. As I said, don't do that.
It does not matter whether the timeouts are short or long. This always happens. See my note above. In fact, no matter what I set the timeouts to, it always seems to fail the server and reject the request after the first retransmit to the proxy (2 packets, about 10 seconds, regardless of the response_window or zombie_period settings). Yes, a subsequent, different request will go to a different home server, but, again, I want to use the proxy to provide smarter resiliency across a pool of servers. If you know of settings for response_window and zombie_period that I can use that will provide the behavior in my "this is what I want to happen" example, could you provide them please? Because all of the settings I use seem to result in the same behavior.
Okay, so I obviously do not understand how I can tweak response_window and zombie_period to make sure that requests that can be serviced by many possible RADIUS home servers do not return an Access-Reject when one of those home servers does not respond.
i.e. you want NO request to fail processing when a home server fails.
This is extremely difficult to do. Any naive approach that has quick failover can have other negative side-effects. (Additional network traffic, system load, duplicate processing of requests, etc.)
I guess I do not see those as negatives. That is exactly what I want to happen. RADIUS network traffic is tiny. The system load created by sending multiple requests to a home server or a bunch of home servers is minimal. I am not seeing how you are adding any more load when instead, the proxy sends back an Access-Reject, which, in the best case scenario, will result in the end-client re-authenticating, generating yet another request. In the worst case scenario, the client accepts the reject as validation that their account cannot be authorized and presents the wrong result to the end-user (whether that be a guy sitting on the end of a dial-up line or a piece of system software trying to determine whether an account is valid). All you are doing is pushing the logic for retrying from a machine that knows the there are multiple possible home servers that can respond to a machine that does not via a response that says, effectively, "Do not retry. Your request is invalid." Your argument that the RADIUS server cannot handle a retry does not hold water to me, but regardless, I can envision configurations where you would want to minimize all processing by the RADIUS proxy itself (most machines now have way more processing power than a simple RADIUS proxy can consume, so that is not a common need anymore). I wish the option was available. There seem to be knobs for a lot of other things.
The client sends a request to the proxy. If a home server does not respond within a short period of time to the request, a second home server is chosen. If the second home server does not respond to the same request, then a third is chosen. This continues until all possible home servers are exhausted. At that point, an Access-Reject packet is sent back to the client. Otherwise, the response from the home server is sent back to the client.
Doing that requires source code mods, because that quick fail-over can have negative side-effects. i.e. The server does NOT support configurations that can negatively affect it's performance.
See my note above for why the work to be done by the server is no more or no less than just returning a reject once the timeout is hit. You are either going to be processing more retries to the home server or more retries from the NAS. Either way, you are going to increase your load.
On top of that, the "try all possible home servers" is impossible. There is ALSO a 30 second lifetime for the request. After 30 seconds, the NAS has given up, so failing over to another home server is useless.
On top of that, the NAS will only retry 3-6 times. So if you have 19 home servers, at *best* it would fail over to 3-6 of them, before the request is marked "timeout".
Okay, AT BEST you get 3-6 different home servers in a 30-second period. Right now, AT BEST I get 1. Which method is more resilient? Which method results in no false rejections being returned to the NAS? The worst that can happen is that the NAS gets no response, which is exactly what would happen if the NAS queried that one home server directly. The proxy can even be smart about it and only retry to a different home server when the NAS retransmits (which I believe it already does), so if the NAS stops retransmitting because it has given up, so does the proxy, but please, let the NAS give up first. The proxy does not know how many times that the NAS will retry. I have my NASes configured to retry for up to 60 seconds, once every 2 seconds. They will retry 30 times. It is more important to me that authentication requests succeed, even if they succeed slowly. It sounds to me like freeradius is making assumptions about how NASes should work, and as a result, reducing the flexibility it provides.
I sincerely hope you see now that the situation is rather more complicated than the simple "try all home server" statement.
How do I configure that? It doesn't seem to matter what I set response_window or zombie_period to, once the first home server fails to respond, an Access-Reject (or nothing if I configure a post-proxy handler) is returned to the client. My client's not going to retry the request if he gets an Access-Reject, so I need the proxy to retry it.
That last sentence is nonsense. Once the client gets an Access- Reject for *any* reason, it is impossible for the proxy to "retry" that request.
*sigh* Exactly. Once the client gets an Access-Reject, the NAS has told the client that the request is invalid. An end-user querying the NAS gets an error message. A piece of system software querying the NAS gets notified that the account is not valid. The implication is that a retry is futile, even though the account is not actually invalid. The account is perfectly valid. The proxy just gave up too soon (and by too soon, I mean "before it tried more than one of its home servers"). I want the proxy to retry the request to a different home server precisely to prevent the NAS (and thus the client) from getting an Access-Reject when it does not have to. This is typically how load-balancers with failover capability work. They try their best to make sure individual requests succeed when they can.
If you want the proxy to fail over, send it more than ONE request at a time (like a normal proxying system), and do NOT configure the "do not respond" policy.
So my NAS now has to send two separate requests for the same authentication, and pick the one that does not come back with an Access-Reject? Which NAS does that? Or are you saying that my end- client has to not accept the fact that he was rejected and keep retrying until he either a) gets an accept or b) gets rejected so many times he accepts it as gospel? Either way, it makes no sense. Either way, the proxy is creating a retry loop. Again, I am not arguing that the proxy will not fail over. It will for subsequent requests. What a fail-over solution will typically do, though, is fail over even for a given single request, so that all requests are handled as resiliently as possible. In other words, a NAS does not need to see a single failed request from the proxy for the proxy to trigger a failover.
The proxy WILL fail over, but due to the imperfect nature of the universe, some requests MAY time out and get rejected. With a better detection algorithm, the number of failures might get smaller than it is today, but it is IMPOSSIBLE to get the number down to zero.
To a NAS, there is a big difference between a timeout and a reject. If it does not get a response, a NAS will typically handle the client differently than if it gets an explicit rejection. Right now, a timeout event from the home server results in an explicit rejection (unless I configure it not to send that reject). It IS possible to get the number down to zero, because I have used RADIUS software that does it. The only time it should ever be non-zero is if all home servers that can possibly be tried in a given window (which might not be all of them, but is most likely going to be more than one of them) fail to respond. Like I said, I am trying to migrate to freeradius for some other features. I have used two other proprietary RADIUS server software packages that implement this behavior.
No. RADIUS doesn't work like that. No amount of magic on the proxy will cause the NAS to retry forever (which is the only way to have the proxy cycle through all home servers for one request). If you configure the NAS to retry forever, then all you will do is push network failures off to some other part of the network.
Right. Precisely. I want to push the network failure handling to the proxy, which has the knowledge that there are multiple points of failure. The NAS does not know that there are 20 possible servers to respond to it. All it knows is that there is 1 RADIUS server it can talk to (the proxy) and if the proxy says the request was rejected, the request is considered rejected. The end-client certainly does not know what can fail. The proxy knows that there are 20 servers. When it decides to fail one server out, it KNOWS a) that the proxied request was not rejected, it just was not responded to by the home server and b) that it can try that request to another home server before it tells the NAS that the request is rejected (the request has not been rejected, of course, since no home server has responded one way or the other yet and until the proxy responds to the NAS, the NAS will not know one way or the other). I also understand that Accept-Challenge can complicate the proxying, but that is solvable as well with standard state tracking.
This is how IP connectivity works: Networks are imperfect. There is absolutely nothing you can do about that.
I know that networks are imperfect. The answer to that imperfection is to retry, not to give up. When you tell a NAS that the request has been rejected when, in fact, it has not, you are not effectively retrying. You are saying, "Do not retry. You actually got this failed result." But look, I have gone through the code. Ivan's right, that there is no way to get the behavior I want in freeradius without either a module (not sure if this is even possible to accomplish via a module because proxying is not handled via a module ) or by hacking the code to change how proxy no-responses are handled. It just frustrates me that you challenge the value of this. For people like me who use freeradius not to serve dial gear but to serve as robust authentication platforms for on-network services, where sending a false rejection to a client is an SLA issue, having a proxy that can robustly and transparently handle transient network failures is very valuable. With that, we do not have to reprogram or replace NAS software (some of which we cannot control) to handle those kinds of transient network failures for us. Philip
I think there's a fundamental disconnect here. I'm trying to explain that RADIUS is an imperfect protocol. You're trying to find ways of configuring FreeRADIUS to be work around those imperfections. My suggestions are: 1) realize that RADIUS is imperfect. If a home server fails, there will *always* be a request that is lost, rejected, timed out, etc. The client WILL fail authentication and disconnect the user when this happens. This is how RADIUS works. 2) proxy fail-over DOES work in the server. Maybe not exactly the way you want... but I recall asking you for specific suggestions as to how to make it better, and getting... not much. 3) If you want to try source code mods, go to src/main/event.c. Look in the function no_response_to_proxied_request(). Find the line: post_proxy_fail_handler(request); Delete it. Re-compile && re-install radiusd. Then try the fail over tests again. It SHOULD cause fail-over to backup home servers for one request. Do NOT configure the "do_not_respond" policy. Try setting "response_window = 5" and "zombie_period = 5". 4) Try setting the home server pool type to "load-balance" (again, as I have suggested). It WILL still fail over from one server to another. But the "load-balance" portion will spread the load MUCH more evenly across all home servers, and there will be FEWER failed requests when a home server dies. I hope those suggestions will be followed. They are your best hope for resolving this issue. And now for a detailed response to the rest of your comments.
Yes, if the NAS sends another separate request with a different ID, it will be proxied to a different home server, but that does not help the poor guy who had the hard luck of his request hitting the bad home server. He will get an error message. He will have to retry or call support or whatever.
The suggestions above will help. The code changes MAY help. If you want the users to always get a response, ensure that there are never any failures in your system. No amount of poking FreeRADIUS will fix the problem that home servers go down.
Yes, a subsequent, different request will go to a different home server, but, again, I want to use the proxy to provide smarter resiliency across a pool of servers.
So... use "load-balance", as I suggested. That can provide a slightly better response in some situations.
I guess I do not see those as negatives. That is exactly what I want to happen. RADIUS network traffic is tiny. The system load created by sending multiple requests to a home server or a bunch of home servers is minimal. I am not seeing how you are adding any more load when instead, the proxy sends back an Access-Reject, which, in the best case scenario, will result in the end-client re-authenticating, generating yet another request.
You will be sending packets to TWO home servers, rather than one. This might be fine in your situation. It is definitely not fine in other situations. FreeRADIUS is designed to work in a wide variety of environments. This means that it might NOT work exactly the way you demand. The solution is simple: you have source code. Fix it. If we add a fix that will make *your* situation work, it is likely to break *other* peoples networks. We can't take that risk.
Your argument that the RADIUS server cannot handle a retry does not hold water to me,
That isn't what I said. That's part of the disconnect in communication.
Okay, AT BEST you get 3-6 different home servers in a 30-second period. Right now, AT BEST I get 1. Which method is more resilient? Which method results in no false rejections being returned to the NAS?
Which method is working in 100,000 deployments? You should note that I asked you for *specific* suggestions for a better algorithm. Your response was "I want it to fail over sooner". That is unhelpful. If you are so set on demanding something better, then offer *concrete* suggestions for how to fix it. Look at the code. It's available, commented, and reasonably clean. Come up with a *better* method, and we'll implement it. The current repetition of "it's bad and I want it to work differently" isn't useful.
I have my NASes configured to retry for up to 60 seconds, once every 2 seconds. They will retry 30 times. It is more important to me that authentication requests succeed, even if they succeed slowly. It sounds to me like freeradius is making assumptions about how NASes should work, and as a result, reducing the flexibility it provides.
That's what "max_request_time" is for. If your NAS is retransmitting for 60 seconds, set "max_request_time" to 60 or 62 seconds.
If you want the proxy to fail over, send it more than ONE request at a time (like a normal proxying system), and do NOT configure the "do not respond" policy.
So my NAS now has to send two separate requests for the same authentication, and pick the one that does not come back with an Access-Reject?
That is not what I meant. You seemed to be claiming that it NEVER failed over. I pointed out that it does fail over, and gave an example of when and how it fails over.
Again, I am not arguing that the proxy will not fail over.
Well... that wasn't at all clear from your messages.
To a NAS, there is a big difference between a timeout and a reject. If it does not get a response, a NAS will typically handle the client differently than if it gets an explicit rejection.
Huh? How? Will it accept the user? Will it let them in? Will it give them some "minimal" service, even if they weren't authenticated? That violates all RADIUS specifications, best practices, and network security guidelines I'm aware of.
Right now, a timeout event from the home server results in an explicit rejection (unless I configure it not to send that reject). It IS possible to get the number down to zero, because I have used RADIUS software that does it. The only time it should ever be non-zero is if all home servers that can possibly be tried in a given window (which might not be all of them, but is most likely going to be more than one of them) fail to respond. Like I said, I am trying to migrate to freeradius for some other features. I have used two other proprietary RADIUS server software packages that implement this behavior.
Well... offer *specific* suggestions for changes to FreeRADIUS that will help implement this. Try the suggested patches, and see if they help.
I also understand that Accept-Challenge can complicate the proxying, but that is solvable as well with standard state tracking.
That I disagree with. EAP makes re-routing proxied Access-Challenges pretty much impossible. (Except in certain rare situation)
I know that networks are imperfect. The answer to that imperfection is to retry, not to give up. When you tell a NAS that the request has been rejected when, in fact, it has not, you are not effectively retrying. You are saying, "Do not retry. You actually got this failed result."
No. That's *your* NAS behavior. Most NASes authenticate end users, who will hit the "connect network" button again when something fails. This is another cause of the miscommunication. It seems that your NASes behave *very* differently from standard RADIUS NASes. They treat timeouts as "re-try authentication". But.. if they behave that way, why did they time out in the first place? Why not just set the timeouts to infinity? That way authentication will *never* fail.
But look, I have gone through the code. Ivan's right, that there is no way to get the behavior I want in freeradius without either a module (not sure if this is even possible to accomplish via a module because proxying is not handled via a module ) or by hacking the code to change how proxy no-responses are handled. It just frustrates me that you challenge the value of this.
Nonsense. I asked *specifically* for suggestions as to a better algorithm. I'm refusing to implement a vague and poorly defined suggestion. That shouldn't be a surprise. Come up with a well-defined algorithm that's better than the current one, and we'll implement it.
For people like me who use freeradius not to serve dial gear but to serve as robust authentication platforms for on-network services, where sending a false rejection to a client is an SLA issue, having a proxy that can robustly and transparently handle transient network failures is very valuable. With that, we do not have to reprogram or replace NAS software (some of which we cannot control) to handle those kinds of transient network failures for us.
I understand that. Please also understand that it doesn't help to say "make it better... I don't know HOW, but you guys need to make it better". We can't implement magic. We CAN implement concrete suggestions. Alan DeKok.
On Jul 11, 2009, at 10:15 AM, Alan DeKok wrote:
I think there's a fundamental disconnect here. I'm trying to explain that RADIUS is an imperfect protocol. You're trying to find ways of configuring FreeRADIUS to be work around those imperfections.
My suggestions are:
1) realize that RADIUS is imperfect. If a home server fails, there will *always* be a request that is lost, rejected, timed out, etc. The client WILL fail authentication and disconnect the user when this happens. This is how RADIUS works.
You are talking to me like I do not understand how RADIUS works. I understand RADIUS is imperfect. My goal is to handle as many of the imperfections at the proxy level rather than at the end-user level (see below for why I want to do that).
2) proxy fail-over DOES work in the server. Maybe not exactly the way you want... but I recall asking you for specific suggestions as to how to make it better, and getting... not much.
I am not sure how I can be more clear than "when a response isn't received, retry the request to a different server rather than return treat it as a failure." But you can read below for an algorithm for achieving this. The algorithm is simple (I think) and can be enabled via an option or control VP, so it will not break any existing work.
3) If you want to try source code mods, go to src/main/event.c. Look in the function no_response_to_proxied_request(). Find the line:
post_proxy_fail_handler(request);
Delete it. Re-compile && re-install radiusd. Then try the fail over tests again. It SHOULD cause fail-over to backup home servers for one request. Do NOT configure the "do_not_respond" policy. Try setting "response_window = 5" and "zombie_period = 5".
I did try that. It did not do what I was attempting to do. I am trying to patch with the algorithm I describe further on down. Freeradius can offer a robust transparent internal failover OR the existing failure handling. It is not an either/or scenario.
4) Try setting the home server pool type to "load-balance" (again, as I have suggested). It WILL still fail over from one server to another. But the "load-balance" portion will spread the load MUCH more evenly across all home servers, and there will be FEWER failed requests when a home server dies.
I am not sure how you ever got the impression that the home server pool type has been set to anything but "load-balance". It has been set to "load-balance" since the very beginning.
I guess I do not see those as negatives. That is exactly what I want to happen. RADIUS network traffic is tiny. The system load created by sending multiple requests to a home server or a bunch of home servers is minimal. I am not seeing how you are adding any more load when instead, the proxy sends back an Access-Reject, which, in the best case scenario, will result in the end-client re-authenticating, generating yet another request.
You will be sending packets to TWO home servers, rather than one. This might be fine in your situation. It is definitely not fine in other situations.
In the case that one of the home servers fails to respond, yes, that is what that means. I agree that is not ideal for all situations. Such behavior can be controlled via options. Again, this is not an either/or scenario. I am not asking you to break the existing implementation. I am simply looking for a solution to my needs within the current framework.
FreeRADIUS is designed to work in a wide variety of environments. This means that it might NOT work exactly the way you demand. The solution is simple: you have source code. Fix it. If we add a fix that will make *your* situation work, it is likely to break *other* peoples networks.
We can't take that risk.
Again, that is what configuration options are for, so people can control their setups for their configurations. You make it sound like I want to ditch your current methods of doing things. I do not. I am simply trying to find more flexibility in how things are handled. I guarantee you that a solution for this will not break anyone else's networks unless they configure it.
Which method is working in 100,000 deployments?
You should note that I asked you for *specific* suggestions for a better algorithm. Your response was "I want it to fail over sooner". That is unhelpful.
That is not what I requested. I requested that it not send back a rejection to the NAS and instead internally retry the request with a different home server. That is not "fail over sooner." That is "fail over transparently." One way of accomplishing this (an algorithm as you have requested) is to realize that when the NAS does not get back a response, it will (if configured) retransmit the original request. When the proxy does not get back a response from the home server it chose (after whatever length of time), it can fail out the home server, not send any response back to the NAS, forget that it ever saw the original request from the NAS, and when the NAS retransmits its request, the proxy will see it as a new original request, choose a new home server (the previous one has been failed out), and then hopefully the proxy will get a response from this new home server that it will pass back to the NAS. If the NAS is configured with a relatively long authentication timeout and the proxy is configured with a relatively short response window for the home server, then that should give enough time for the proxy to try multiple home servers, each try being triggered by a retransmit from the NAS, before the NAS treats the proxy as having timed out and then moves on to timeout handling (see below for a scenario where timeout handling is different than rejection handling). This requires relatively quick fail out of home servers, something which can already be configured. It requires that the tracking hash of the original request packet and proxy be cleared. Both of these can be enabled via an option, probably at the pool level, so that it does not change the behavior for existing configs, but adds this capability for people who want this kind of transparent failover. However, I think the cleanest way to implement is to add a new Response-Packet-Type of (suggestion) 'Proxy-Clear-Response' that is treated like the above for proxied requests and is treated as 'Do-Not- Respond' if not used within the a post-proxy failure handler.
If you are so set on demanding something better, then offer *concrete* suggestions for how to fix it. Look at the code. It's available, commented, and reasonably clean.
Come up with a *better* method, and we'll implement it. The current repetition of "it's bad and I want it to work differently" isn't useful.
Does that seem like a method that can work. Again, not to replace anything but to supplement it?
If you want the proxy to fail over, send it more than ONE request at a time (like a normal proxying system), and do NOT configure the "do not respond" policy.
So my NAS now has to send two separate requests for the same authentication, and pick the one that does not come back with an Access-Reject?
That is not what I meant. You seemed to be claiming that it NEVER failed over. I pointed out that it does fail over, and gave an example of when and how it fails over.
Again .... transparent failover ... that NAS should not have to get an error to effect a failover on the proxy.
To a NAS, there is a big difference between a timeout and a reject. If it does not get a response, a NAS will typically handle the client differently than if it gets an explicit rejection.
Huh? How? Will it accept the user? Will it let them in? Will it give them some "minimal" service, even if they weren't authenticated?
That violates all RADIUS specifications, best practices, and network security guidelines I'm aware of.
Many NASes can use an internal user cache as a backup to a non- responding or slowly-responding RADIUS server. If the proxy returns a an actual Access-Reject, the NAS accepts that and says the request is invalid. If the proxy returns nothing, the NAS can say, "Well, my RADIUS server is down, but I have this record for the same user/pass in my cache and previously, it received an Access-Accept. Let me accept this request." That does not break any RADIUS specifications I know of, but it does provide a better experience for the end-client, ie. a down RADIUS server does not completely kill your authentication abilities. Certainly, if the home server returns a reject, that rejection should get passed back to the end-client by the NAS, regardless of what it may have cached, but if the home server passes back nothing, the NAS should not receive a rejection. The home RADIUS server never rejected anything. I would like to let the NAS decide if a timeout is treated as a rejection or not, not the proxy. Right now, I can do that with do_not_respond, BUT it would be even better if the proxy would try harder to contact more home servers if my NAS is waiting longer for a RADIUS server to respond.
Right now, a timeout event from the home server results in an explicit rejection (unless I configure it not to send that reject). It IS possible to get the number down to zero, because I have used RADIUS software that does it. The only time it should ever be non-zero is if all home servers that can possibly be tried in a given window (which might not be all of them, but is most likely going to be more than one of them) fail to respond. Like I said, I am trying to migrate to freeradius for some other features. I have used two other proprietary RADIUS server software packages that implement this behavior.
Well... offer *specific* suggestions for changes to FreeRADIUS that will help implement this. Try the suggested patches, and see if they help.
I will try specific patches.
I also understand that Accept-Challenge can complicate the proxying, but that is solvable as well with standard state tracking.
That I disagree with. EAP makes re-routing proxied Access-Challenges pretty much impossible. (Except in certain rare situation)
Right. You cannot reroute them. You just have to make sure they get destined for the home server that can handle them, which is usually the home server that handled the initial Access-Request. Like I said, you just make sure that happens by tracking where the original Access- Challenge response came from, or you ignore it and you say that if a failover happens in between the Access-Request and the Access- Challenge, the end-user will receive an error.
I know that networks are imperfect. The answer to that imperfection is to retry, not to give up. When you tell a NAS that the request has been rejected when, in fact, it has not, you are not effectively retrying. You are saying, "Do not retry. You actually got this failed result."
No. That's *your* NAS behavior. Most NASes authenticate end users, who will hit the "connect network" button again when something fails.
Right, and my users complain when that happens. I'm looking for a RADIUS solution that provides a better experience for my customers. I am very understanding of the imperfections of networks in general and RADIUS specifically, but my end-users much less so. It is easier to use software to reduce complaints than it is to educate every user on the intricacies of an authentication protocol.
This is another cause of the miscommunication. It seems that your NASes behave *very* differently from standard RADIUS NASes. They treat timeouts as "re-try authentication".
They do not. See above.
But.. if they behave that way, why did they time out in the first place? Why not just set the timeouts to infinity? That way authentication will *never* fail.
Timeouts from the NASes standpoint are not treated as re-try authentication. The timeouts on my NASes are long enough for the proxy the NAS communicates with to try multiple home servers. The timeout on the proxy for each home server is shorter, so that it fails over to another home server more quickly, within the window of the authentication request on the NAS. I still want my NASes to timeout.
But look, I have gone through the code. Ivan's right, that there is no way to get the behavior I want in freeradius without either a module (not sure if this is even possible to accomplish via a module because proxying is not handled via a module ) or by hacking the code to change how proxy no-responses are handled. It just frustrates me that you challenge the value of this.
Nonsense. I asked *specifically* for suggestions as to a better algorithm. I'm refusing to implement a vague and poorly defined suggestion. That shouldn't be a surprise.
I hope my description above of the proxy forgetting the original request when it fails out a home server is an example of a different algorithm. I will not call it a better algorithm, because I am not looking for that algorithm to replace anything, just to supplement it as an additional option. It is definitely a better algorithm for me.
Come up with a well-defined algorithm that's better than the current one, and we'll implement it.
For people like me who use freeradius not to serve dial gear but to serve as robust authentication platforms for on-network services, where sending a false rejection to a client is an SLA issue, having a proxy that can robustly and transparently handle transient network failures is very valuable. With that, we do not have to reprogram or replace NAS software (some of which we cannot control) to handle those kinds of transient network failures for us.
I understand that. Please also understand that it doesn't help to say "make it better... I don't know HOW, but you guys need to make it better".
Please tell me if the example I gave above is a good HOW. I am trying to come up with a code-based solution today, but I am not nearly as familiar with the code as you probably are. Philip
Philip Molter wrote:
I did try that. It did not do what I was attempting to do.
Hmm... "it didn't work".
I am trying to patch with the algorithm I describe further on down. Freeradius can offer a robust transparent internal failover OR the existing failure handling. It is not an either/or scenario.
As I said, I'm open to suggestions.
I am not sure how you ever got the impression that the home server pool type has been set to anything but "load-balance". It has been set to "load-balance" since the very beginning.
I've been reading your messages. It has *not* been clear that the pool type was set to "load-balance".
One way of accomplishing this (an algorithm as you have requested) is to realize that when the NAS does not get back a response, it will (if configured) retransmit the original request. When the proxy does not get back a response from the home server it chose (after whatever length of time), it can fail out the home server, not send any response back to the NAS, forget that it ever saw the original request from the NAS, and when the NAS retransmits its request, the proxy will see it as a new original request, choose a new home server (the previous one has been failed out), and then hopefully the proxy will get a response from this new home server that it will pass back to the NAS.
That's a good idea, but not really possible as-is.
This requires relatively quick fail out of home servers, something which can already be configured. It requires that the tracking hash of the original request packet and proxy be cleared.
I'm not so sure. For one, deleting the original tracking has is not an option. It's also not necessary. See the existing code for how requests can fail-over from one home server to another *without* a problem. The code already supports this.
Both of these can be enabled via an option, probably at the pool level, so that it does not change the behavior for existing configs, but adds this capability for people who want this kind of transparent failover. However, I think the cleanest way to implement is to add a new Response-Packet-Type of (suggestion) 'Proxy-Clear-Response' that is treated like the above for proxied requests and is treated as 'Do-Not-Respond' if not used within the a post-proxy failure handler.
I'm not sure that's necessary.
Does that seem like a method that can work. Again, not to replace anything but to supplement it?
It's an *additional* method, rather than a *better* method. It requires additional code, additional state machine checks, and as such... I'm biased against it. It's just too much like a site-specific hack for it to be integrated into the main distribution. Adding a *better* method is the preferred approach. It's OK to change existing behavior, so long as it's for the better.
Many NASes can use an internal user cache as a backup to a non-responding or slowly-responding RADIUS server. If the proxy returns a an actual Access-Reject, the NAS accepts that and says the request is invalid. If the proxy returns nothing, the NAS can say, "Well, my RADIUS server is down, but I have this record for the same user/pass in my cache and previously, it received an Access-Accept. Let me accept this request." That does not break any RADIUS specifications I know of,
Nonsense. NASes that cache authentication credentials are *completely* outside of the RADIUS specification. It's like adding a jet engine to a car. There's no law *preventing* it, so it must be legal, right?
I hope my description above of the proxy forgetting the original request when it fails out a home server is an example of a different algorithm.
It's simple, and it can't work. Re-processing duplicate packets from scratch is simply not an option. Please explain *why* the suggested patch I had doesn't work. What does it do? How does it behave differently than what you expect?
Please tell me if the example I gave above is a good HOW. I am trying to come up with a code-based solution today, but I am not nearly as familiar with the code as you probably are.
The code can be safely ignored. The algorithm depends on only a few pieces of information: proxy home server various timeouts NAS retransmits If you can describe an algorithm based on that information, it can be implemented. Alan DeKok.
On Jul 11, 2009, at 2:23 PM, Alan DeKok wrote:
Philip Molter wrote:
I did try that. It did not do what I was attempting to do.
Hmm... "it didn't work".
I apologize I was not more specific. The retransmits kept getting sent to the same failed home server rather than the failed home server being marked dead and the retransmits going to a different home server. I have figured out why. The minimum zombie_period is 20, hard-coded in realms.c. The zombie_period of 5 you recommended which I tried was not taking effect, which lead to my 20 second test timeout kicking in before the proxy had waited long enough to actually mark the server as dead (The 5th retransmit would have triggered the failover, but the proxy only got 3 retransmits). And that does exactly what I want for this case. I can provide a patch that does the following things: a) allows lower values than 5 for response_window and 20 for zombie_period (I will not change recommendations) b) makes the post_proxy_fail_handler optional on a pool-by-pool basis Does that seem acceptable? You seem hesitant to accept a solution that you do not think could be used for more than a few people. This solution is going to be minimally invasive to the code. Also, is there a config with which the retransmit proxy failover code could actually be triggered without the patch? I cannot see it. Failover only happens after the response_window is exceeded, and if the response_window is exceeded, the original request is replied to with an Access-Reject message, which means any retransmits will be never reach the REQUEST_PROXIED state in received_retransmits() after the response_window is exceeded. Am I reading that correctly?
Does that seem like a method that can work. Again, not to replace anything but to supplement it?
It's an *additional* method, rather than a *better* method. It requires additional code, additional state machine checks, and as such... I'm biased against it. It's just too much like a site- specific hack for it to be integrated into the main distribution.
Adding a *better* method is the preferred approach. It's OK to change existing behavior, so long as it's for the better.
You had the retransmit failover code already written. It seems not much needs to be done to allow a pool configuration to continue on after the response_window has been exceeded. Let me submit a patch and you see what you think.
Many NASes can use an internal user cache as a backup to a non-responding or slowly-responding RADIUS server. If the proxy returns a an actual Access-Reject, the NAS accepts that and says the request is invalid. If the proxy returns nothing, the NAS can say, "Well, my RADIUS server is down, but I have this record for the same user/ pass in my cache and previously, it received an Access-Accept. Let me accept this request." That does not break any RADIUS specifications I know of,
Nonsense. NASes that cache authentication credentials are *completely* outside of the RADIUS specification. It's like adding a jet engine to a car. There's no law *preventing* it, so it must be legal, right?
Well, there's nothing in the RADIUS specification that describes or even recommends how a lack of response must be handled by the NAS. You make it sound as if the NAS is doing something illegal by using a previous cached accept. It's not. The NAS can implement whatever logic it wants, and that particular feature is one that leads to a better user experience. Just because you think a failure-to-contact is the same as a denial does not mean that other vendors have not come up with solutions that can work around it. RFC 2607 is clear that the proxy should not respond to the client unless it receives a reply from the home server. At the very least, returning a rejection is not an accurate portrayal of the state of the authentication. It would be a better representation to just let it timeout, but I understand returning the rejection so that the NAS can short-circuit more quickly the transaction.
Philip Molter wrote:
I apologize I was not more specific. The retransmits kept getting sent to the same failed home server rather than the failed home server being marked dead and the retransmits going to a different home server. I have figured out why. The minimum zombie_period is 20, hard-coded in realms.c. The zombie_period of 5 you recommended which I tried was not taking effect, which lead to my 20 second test timeout kicking in before the proxy had waited long enough to actually mark the server as dead (The 5th retransmit would have triggered the failover, but the proxy only got 3 retransmits).
OK. So making the zombie period shorter would have made it fail over sooner. That's fine.
And that does exactly what I want for this case. I can provide a patch that does the following things:
a) allows lower values than 5 for response_window and 20 for zombie_period (I will not change recommendations)
That's fine. People are free to destroy their own systems if they don't follow the recommendations.
b) makes the post_proxy_fail_handler optional on a pool-by-pool basis
If the early "reject" is wrong, it might be best to just delete it. Sites with a small number of home servers will still run the post_proxy_fail_handler, just a little bit later than they do now.
Does that seem acceptable? You seem hesitant to accept a solution that you do not think could be used for more than a few people. This solution is going to be minimally invasive to the code.
It seems fine.
Also, is there a config with which the retransmit proxy failover code could actually be triggered without the patch? I cannot see it.
For the original request that started this? No. For the other requests, see other calls to home_server_ldb(), around line 2580 of event.c.
Failover only happens after the response_window is exceeded, and if the response_window is exceeded, the original request is replied to with an Access-Reject message, which means any retransmits will be never reach the REQUEST_PROXIED state in received_retransmits() after the response_window is exceeded. Am I reading that correctly?
Yes... but the suggested patch *deletes* the code that makes the request fail over "response_window" is exceeded. So... that request should fail over to another home server, if zombie_period is set low enough.
You had the retransmit failover code already written. It seems not much needs to be done to allow a pool configuration to continue on after the response_window has been exceeded. Let me submit a patch and you see what you think.
OK.
Well, there's nothing in the RADIUS specification that describes or even recommends how a lack of response must be handled by the NAS.
The RADIUS specifications are missing a whole lot of things. Like "interim updates should be sent from the same source IP as the start/stop packets". Obvious, right? Well... there are products that violate this.
You make it sound as if the NAS is doing something illegal by using a previous cached accept. It's not.
I said it's "outside of RADIUS". It does not follow the RADIUS operational model, which is that the RADIUS server authenticates users. If the NAS caches credentials, it's authenticating users via a non-RADIUS method.
The NAS can implement whatever logic it wants, and that particular feature is one that leads to a better user experience. Just because you think a failure-to-contact is the same as a denial does not mean that other vendors have not come up with solutions that can work around it.
It's a "fail-safe" security practice. The alternative is to take the RADIUS server down, and then what? Does the NAS let everyone on the network? What happens if their credentials have been cached, but they haven't paid their bills? The work-around you're talking about is *very* site-specific. ISP's and telcos would go crazy if NAS vendors implemented it. They could lose a lot of money...
RFC 2607 is clear that the proxy should not respond to the client unless it receives a reply from the home server. At the very least, returning a rejection is not an accurate portrayal of the state of the authentication. It would be a better representation to just let it timeout, but I understand returning the rejection so that the NAS can short-circuit more quickly the transaction.
Not returning a response also makes the NAS think that the proxy is down, when it's not. See the status-server draft for a discussion of this issue, and a solution: http://tools.ietf.org/internet-drafts/draft-ietf-radext-status-server Yes, I'm not just a random opinionated guy on the net. I have 4-5 RADIUS specifications either published, or on track to be published. Alan DeKok.
On Jul 12, 2009, at 1:58 AM, Alan DeKok wrote:
Philip Molter wrote:
b) makes the post_proxy_fail_handler optional on a pool-by-pool basis
If the early "reject" is wrong, it might be best to just delete it. Sites with a small number of home servers will still run the post_proxy_fail_handler, just a little bit later than they do now.
I have left it in as a configurable option. I would rather someone not upgrade their freeradius codebase with this patch and find that the behavior they have come to rely on has changed. Maybe you change it in the future (heck, maybe you change it now ... it is your code), but I did not want this patch to break anything for anyone.
Does that seem acceptable? You seem hesitant to accept a solution that you do not think could be used for more than a few people. This solution is going to be minimally invasive to the code.
It seems fine.
Patch is attached to this e-mail. Please let me know if you would like it sent somewhere else or in some other format.
You make it sound as if the NAS is doing something illegal by using a previous cached accept. It's not.
I said it's "outside of RADIUS". It does not follow the RADIUS operational model, which is that the RADIUS server authenticates users. If the NAS caches credentials, it's authenticating users via a non-RADIUS method.
Many NASes use a variety of different backends to authenticate users. Freeradius should not always assume that it is the only source for authentication and make decisions for the NAS. Note I said "always." Freeradius *can* assume if told so (or inversely, it can be told not to assume, depending on which way you want the default to work).
The NAS can implement whatever logic it wants, and that particular feature is one that leads to a better user experience. Just because you think a failure-to-contact is the same as a denial does not mean that other vendors have not come up with solutions that can work around it.
It's a "fail-safe" security practice. The alternative is to take the RADIUS server down, and then what? Does the NAS let everyone on the network? What happens if their credentials have been cached, but they haven't paid their bills?
What happens indeed? That is not for the RADIUS server to decide. That is for the NAS to decide. The NAS ultimately decides what to do with the information it receives from the RADIUS server. If the RADIUS server (or proxy in this case) sends back inaccurate information, the NAS cannot correctly make those decisions. I actually have a use-case where if the authentication server is down, customers are allowed temporary access to our service with limits. If I cannot keep my service up, I take on the liability of that, rather than going dark.
The work-around you're talking about is *very* site-specific. ISP's and telcos would go crazy if NAS vendors implemented it. They could lose a lot of money...
ISPs and telcos are not the only organizations that use RADIUS servers! There are many other service providers out there that authenticate customers that do not fit into an ISP/telco model. Of course the workaround I am talking about is site-specific, because every site has different business logic. And again, I am not saying that the NAS should NOT deny access to individuals. I am not saying they SHOULD either. All I am saying is that the RADIUS proxy should not assume that because it did not receive a response from its home server, that the NAS will reject the client. The RADIUS server does not know what its reply is going to be used for, so it should send back as accurate a response as possible (in this case, none).
RFC 2607 is clear that the proxy should not respond to the client unless it receives a reply from the home server. At the very least, returning a rejection is not an accurate portrayal of the state of the authentication. It would be a better representation to just let it timeout, but I understand returning the rejection so that the NAS can short-circuit more quickly the transaction.
Not returning a response also makes the NAS think that the proxy is down, when it's not. See the status-server draft for a discussion of this issue, and a solution:
http://tools.ietf.org/internet-drafts/draft-ietf-radext-status-server
Yes, I have read it. I wish more RADIUS servers and NASes implemented it. Still, have the ability to let the NAS figure out if the RADIUS server is down through its own logic (whether that be through more Access-Accept packets or Status-Server packets or pings or whatever). I would like the proxy to not return an Access-Reject because the proxy thinks the NAS is too dumb to figure out on its own whether or not the proxy is actually down, just like Freeradius itself is not so dumb as to assume that one no-response is tantamount to a home server being down (that is why you have a zombie period after all). However, I also know that some people have dumb NASes and they want Freeradius to make that determination for them. That is why I proposed this configuration method for the behavior.
Yes, I'm not just a random opinionated guy on the net. I have 4-5 RADIUS specifications either published, or on track to be published.
And I am not just some guy who started using RADIUS yesterday. I have run a proxy farm for years that backends to thousands of home servers maintained by hundreds of different organizations around the world. I do not fit into the telco/ISP model of RADIUS usage, but I am not doing anything fancy with RADIUS. I just have a different resiliency model than an ISP or telco. I am another use-case for your code. I would very much like to switch the entire farm over to Freeradius because I think it is a very well implemented product. I just need to it support -- in addition to its current features, not in replacement of any features -- some more flexible handling of various proxy- related events. When you process thousands of authentications a second over links that range from sub-ms to over 300ms, against servers that go up and down for maintenance or network outage or just plain packet loss, you require a bit more flexibility than is required by an ISP/telco model. Patch attached. Let me know what you think. Philip
Philip Molter wrote:
I have left it in as a configurable option. I would rather someone not upgrade their freeradius codebase with this patch and find that the behavior they have come to rely on has changed.
While I am concerned about changing behavior, that reject on "response_window" happened ONLY for the first request to hit "response_window". All others continued to be proxied to the home server, even when it was zombie. So the original behavior was arguably inconsistent, and wrong.
Patch is attached to this e-mail. Please let me know if you would like it sent somewhere else or in some other format.
That's OK. I've committed changes that: a) allow response_window && zombie_period to be smaller b) proxying prefers live home servers to zombie servers zombie servers are still used if there's no other choice. This improves the stability of proxying c) response_window being hit now does nothing other than start the zombie period. The first request that causes this is NOT rejected. The old behavior can still be configured if desired. d) documentation for all of the above. Alan DeKok.
Alan DeKok wrote:
c) response_window being hit now does nothing other than start the zombie period. The first request that causes this is NOT rejected. The old behavior can still be configured if desired.
How does this change affect accounting handling? Does the request now fail over to the next proxy server? If the post_proxy_fail_handler isn't called, does that break any of the methods implemented in the robust accounting proxy example server?
Philip Molter wrote:
How does this change affect accounting handling?
Very little. Accounting requests aren't normally retransmitted. Instead, new ones are generated.
Does the request now fail over to the next proxy server?
Yes. The proxy && fail-over code is independent of packet type.
If the post_proxy_fail_handler isn't called, does that break any of the methods implemented in the robust accounting proxy example server?
It may still be called, just less often. Alan DeKok.
What I really want is just, instead of the request being marked as failed when one of the home servers doesn't respond, for the proxy subsystem to just try sending the request to another configured home server. If the proxy has tried sending a request to every non-zombie home server in the list and still hasn't gotten anything, then it can mark the request as failed.
That's not really on, as zombie server is still considered alive. You can try this: instead of do_not_respond policy create a do_not_respond module (patch). It would do the same as the policy plus remove the request from the list once it fails. Then next NAS retransmit has a good chance of going to a non-zombie home server. Ivan Kalik Kalik Informatika ISP
participants (3)
-
Alan DeKok -
Ivan Kalik -
Philip Molter