Proxy realms and home_server_pool fallback not working
Hi I'm looking to add more robustness into my proxy architecture and noticed in the home_server_pool there is the option for "fallback = virtualrealm" so if all home servers fail then a last resort home_server is used with some config locally to always accept / reject customers based on the realm they are coming from. I'm not using the status_check as some of the downstream clients don't support status-server, but I will look into that to see if it makes a difference. However for this situation I would expect if you are using or not using Status server checks shouldn't have any impact on how the fallback server works. In the proxy.conf I have configured: home_server ProxyDest { type = auth+acct ipaddr = 192.168.1.113 port = 1812 secret = password response_window = 1 require_message_authenticator = no zombie_period = 5 revive_interval = 10 status_check = none #status_check = status-server # username = "test_user_please_reject_me" # password = "this is really secret" check_interval = 10 num_answers_to_alive = 3 max_outstanding = 65536 } home_server cacheuser { virtual_server = cacheuser } # Main server pool # home_server_pool ProxyDestPool { type = fail-over home_server = ProxyDest # home_server = cacheuser fallback = cacheuser } Then in my virtual server I have configured: server cacheuser { authorize { accept } So when the Destination server is up life is good. (0) Proxying request to home server 192.168.1.113 port 1812 timeout 1.000000 (0) Sent Access-Request Id 26 from 0.0.0.0:58512 to 192.168.1.113:1812 length 337 ... Waking up in 0.3 seconds. (0) Marking home server 192.168.1.113 port 1812 alive (0) Clearing existing &reply: attributes (0) Received Access-Accept Id 26 from 192.168.1.113:1812 to 192.168.1.116:58512 length 55 But if the server is down the first request I get a reject as expected due to the home server being down. (2) Proxying request to home server 192.168.1.113 port 1812 timeout 1.000000 (2) Sent Access-Request Id 17 from 0.0.0.0:47755 to 192.168.1.113:1812 length 337 ... Waking up in 0.3 seconds. (2) Expecting proxy response no later than 0.669753 seconds from now Waking up in 0.4 seconds. (2) No proxy response, giving up on request and marking it done Marking home server 192.168.1.113 port 1812 as zombie (it has not responded in 1.000000 seconds). (2) ERROR: Failing proxied request for user "peter", due to lack of any response from home server 192.168.1.113 port 1812 (2) Clearing existing &reply: attributes But the second and subsequent request I would expect to get proxied to the local fallback virtual server as the home_server has been marked as zombie. But that never seems to happen. It keeps on rejecting the requests and fallback never seems to be used. If I configure a second home server in the pool. home_server_pool ProxyDestPool { type = fail-over home_server = ProxyDest home_server = cacheuser fallback = cacheuser } Then the second server is failed over to when the first fails. Which is all good if I wanted to use the type fail-over, but if I wanted to use load-balance then I can't have my fallback server as a home server otherwise a percentage of requests will always be local which isn't ideal. The other interesting thing with the failover is I set the check_interval to 10 seconds, or 30 seconds. But it only seems that the first client is re-checked after 60 seconds and assumed to be back up. Waking up in 0.2 seconds. Marking home server 192.168.1.113 port 1812 alive again... we have no idea if it really is alive or not. Waking up in 1.0 seconds. I would have thought that zombie_period = 5 revive_interval = 10 check_interval = 10 Would mean that the client would be re-checked in 10 seconds. Am I mis-understanding how fallback is supposed to work? Cheers Peter
On Mar 6, 2016, at 6:54 PM, Peter Lambrechtsen <peter@crypt.co.nz> wrote:
I'm looking to add more robustness into my proxy architecture and noticed in the home_server_pool there is the option for "fallback = virtualrealm" so if all home servers fail then a last resort home_server is used with some config locally to always accept / reject customers based on the realm they are coming from. I'm not using the status_check
Then you can do "status_check = request". An Access-Accept or Access-Reject response will be accepted as an indication that the home server i alive.
as some of the downstream clients don't support status-server, but I will look into that to see if it makes a difference.
It should.
However for this situation I would expect if you are using or not using Status server checks shouldn't have any impact on how the fallback server works.
It does. A lot. The problem is that without Status-Server, FreeRADIUS has to *guess* when the home server is alive. And the guess is usually wrong. Because most guesses are wrong.
In the proxy.conf I have configured:
home_server ProxyDest { type = auth+acct ipaddr = 192.168.1.113 port = 1812 secret = password response_window = 1 require_message_authenticator = no zombie_period = 5 revive_interval = 10
That's really low. After 10s, just mark the home server alive? It should be 60s at the minimum. Maybe 5min.
But if the server is down the first request I get a reject as expected due to the home server being down.
That's good.
But the second and subsequent request I would expect to get proxied to the local fallback virtual server as the home_server has been marked as zombie. But that never seems to happen. It keeps on rejecting the requests and fallback never seems to be used.
Hmm... I'll take a look.
If I configure a second home server in the pool. ... Then the second server is failed over to when the first fails. Which is all good if I wanted to use the type fail-over, but if I wanted to use load-balance then I can't have my fallback server as a home server otherwise a percentage of requests will always be local which isn't ideal.
Yes. You can't do load-balance and fallback. You *can* put something into Post-Proxy-Type Fail. Which is probably what we should do. And remove the fallback virtual server. This allows the same behaviour for all packets, and simplifies the proxy code.
The other interesting thing with the failover is I set the check_interval to 10 seconds, or 30 seconds. But it only seems that the first client is re-checked after 60 seconds and assumed to be back up.
Because you have revive_interval set.
Waking up in 0.2 seconds. Marking home server 192.168.1.113 port 1812 alive again... we have no idea if it really is alive or not.
And that message is printed only when you have revive_interval set. The solution is to *not* set revive_interval. And use Status-Server exclusively.
Waking up in 1.0 seconds.
I would have thought that
zombie_period = 5 revive_interval = 10 check_interval = 10
Would mean that the client would be re-checked in 10 seconds.
check_interval and revive_interval should be mutually exclusive. It just doesn't make sense to both check that the home server is alive every 10s, and then *always* mark it as alive after 10s.
Am I mis-understanding how fallback is supposed to work?
A bit. But the fallback virtual server should work. Tho I'm inclined to remove it in 3.1, as it makes everything more complicated. Alan DeKok.
On Mon, Mar 7, 2016 at 2:55 PM, Alan DeKok <aland@deployingradius.com> wrote:
On Mar 6, 2016, at 6:54 PM, Peter Lambrechtsen <peter@crypt.co.nz> wrote:
I'm looking to add more robustness into my proxy architecture and noticed in the home_server_pool there is the option for "fallback = virtualrealm" so if all home servers fail then a last resort home_server is used with some config locally to always accept / reject customers based on the realm they are coming from. I'm not using the status_check
Then you can do "status_check = request". An Access-Accept or Access-Reject response will be accepted as an indication that the home server i alive.
as some of the downstream clients don't support status-server, but I will look into that to see if it makes a difference.
It should.
However for this situation I would expect if you are using or not using Status server checks shouldn't have any impact on how the fallback server works.
It does. A lot.
The problem is that without Status-Server, FreeRADIUS has to *guess* when the home server is alive. And the guess is usually wrong. Because most guesses are wrong.
Yes, I have figured that out. I'm now pinging all our downstream radius clients to see which respond to something sane when sent a Status, and then turning on Status server for them.
In the proxy.conf I have configured:
home_server ProxyDest { type = auth+acct ipaddr = 192.168.1.113 port = 1812 secret = password response_window = 1 require_message_authenticator = no zombie_period = 5 revive_interval = 10
That's really low. After 10s, just mark the home server alive?
It should be 60s at the minimum. Maybe 5min.
It was purely for testing as waiting around for 10 seconds is much better than waiting around for 2 mins. Now with check_interval with status turned on things are making more sense.
But the second and subsequent request I would expect to get proxied to the local fallback virtual server as the home_server has been marked as zombie. But that never seems to happen. It keeps on rejecting the requests and fallback never seems to be used.
Hmm... I'll take a look.
If I configure a second home server in the pool. ... Then the second server is failed over to when the first fails. Which is all good if I wanted to use the type fail-over, but if I wanted to use load-balance then I can't have my fallback server as a home server otherwise a percentage of requests will always be local which isn't ideal.
Yes. You can't do load-balance and fallback.
You *can* put something into Post-Proxy-Type Fail. Which is probably what we should do. And remove the fallback virtual server.
What could I do in Post-Proxy-Type? As I can't call the virtual server, and Proxy-To-Realm doesn't proxy to a new destination nor does setting the control to accept. There doesn't seem to be a way to turn a Reject from a failed proxy request back into an Accept. (0) ERROR: Failing proxied request for user "peter", due to lack of any response from home server 192.168.1.113 port 1812 (0) Clearing existing &reply: attributes (0) Found Post-Proxy-Type Fail-Authentication (0) # Executing group from file ./sites-enabled/default (0) Post-Proxy-Type Fail-Authentication { (0) policy accept { (0) update control { (0) &Response-Packet-Type = Access-Accept (0) } # update control = noop (0) [handled] = handled (0) } # policy accept = handled (0) } # Post-Proxy-Type Fail-Authentication = handled (0) There was no response configured: rejecting request (0) Using Post-Auth-Type Reject
This allows the same behaviour for all packets, and simplifies the proxy code.
The other interesting thing with the failover is I set the check_interval to 10 seconds, or 30 seconds. But it only seems that the first client is re-checked after 60 seconds and assumed to be back up.
Because you have revive_interval set.
Waking up in 0.2 seconds. Marking home server 192.168.1.113 port 1812 alive again... we have no idea if it really is alive or not.
And that message is printed only when you have revive_interval set.
The solution is to *not* set revive_interval. And use Status-Server exclusively.
Waking up in 1.0 seconds.
I would have thought that
zombie_period = 5 revive_interval = 10 check_interval = 10
Would mean that the client would be re-checked in 10 seconds.
check_interval and revive_interval should be mutually exclusive. It just doesn't make sense to both check that the home server is alive every 10s, and then *always* mark it as alive after 10s.
Am I mis-understanding how fallback is supposed to work?
A bit.
But the fallback virtual server should work. Tho I'm inclined to remove it in 3.1, as it makes everything more complicated.
Thanks for all your help on this, the fail-over with the second server being the virtual seems to work well, just means I am restricted to a single server and can't use load-balance. But having this config would be my ideal: home_server_pool ProxyDestPool { type = load-balance home_server = ProxyDest1 home_server = ProxyDest2 home_server = ProxyDest3 fallback = cacheuser } Where if all the home servers go awol I use the local virtual server cacheuser. Many thanks Peter
On Mar 7, 2016, at 3:22 AM, Peter Lambrechtsen <peter@crypt.co.nz> wrote:
Yes, I have figured that out. I'm now pinging all our downstream radius clients to see which respond to something sane when sent a Status, and then turning on Status server for them.
Or just send Access-Request with a fake username "thisismejusttesting". They'll respond with an Access-Reject, which is good enough to determine that they're alive.
What could I do in Post-Proxy-Type?
Anything you can do anywhere else. The fallback virtual server is just there for ease of use. But... it complicates the proxy handling, as you've seen. A simpler approach is to put all of the "unlang" handling into... unlang. And not into the proxy code.
As I can't call the virtual server,
We'll fix that for 3.2.
and Proxy-To-Realm doesn't proxy to a new destination nor does setting the control to accept.
The home server pools should take care of fail-over to another home server. But yes, once the whole pool has failed... you can't send the packet to a different destination. That's what home server pools are for...
There doesn't seem to be a way to turn a Reject from a failed proxy request back into an Accept.
A failed proxy request is not really a reject... it's just a failed request. And you can force it to be an Access-Accept via Post-Proxy-Type Fail: post-proxy { ... Post-Proxy-Type Fail-Authentication { update control { Response-Packet-Type := Access-Accept } } ... } We'll work on simplifying that for 3.2, also.
Thanks for all your help on this, the fail-over with the second server being the virtual seems to work well, just means I am restricted to a single server and can't use load-balance. But having this config would be my ideal:
home_server_pool ProxyDestPool { type = load-balance home_server = ProxyDest1 home_server = ProxyDest2 home_server = ProxyDest3 fallback = cacheuser }
That works for me. When all home servers in a "load-balance" pool are down, it uses the fallback virtual server: (0) } # authorize = updated Home server pool example.net failing over to fallback example.net Proxying to virtual server example.net Alan DeKok.
On Tue, Mar 8, 2016 at 3:04 AM, Alan DeKok <aland@deployingradius.com> wrote:
On Mar 7, 2016, at 3:22 AM, Peter Lambrechtsen <peter@crypt.co.nz> wrote:
Yes, I have figured that out. I'm now pinging all our downstream radius clients to see which respond to something sane when sent a Status, and then turning on Status server for them.
Or just send Access-Request with a fake username "thisismejusttesting". They'll respond with an Access-Reject, which is good enough to determine that they're alive.
What could I do in Post-Proxy-Type?
Anything you can do anywhere else.
The fallback virtual server is just there for ease of use. But... it complicates the proxy handling, as you've seen. A simpler approach is to put all of the "unlang" handling into... unlang. And not into the proxy code.
As I can't call the virtual server,
We'll fix that for 3.2.
and Proxy-To-Realm doesn't proxy to a new destination nor does setting the control to accept.
The home server pools should take care of fail-over to another home server. But yes, once the whole pool has failed... you can't send the packet to a different destination. That's what home server pools are for...
There doesn't seem to be a way to turn a Reject from a failed proxy request back into an Accept.
A failed proxy request is not really a reject... it's just a failed request. And you can force it to be an Access-Accept via Post-Proxy-Type Fail:
post-proxy { ...
Post-Proxy-Type Fail-Authentication { update control { Response-Packet-Type := Access-Accept }
} ... }
This doesn't seem to work in 3.0.x head, I will test it on 3.1.x tomorrow. (0) ERROR: Failing proxied request for user "peter", due to lack of any response from home server 192.168.1.113 port 1812 (0) Clearing existing &reply: attributes (0) Found Post-Proxy-Type Fail-Authentication (0) # Executing group from file ./sites-enabled/default (0) Post-Proxy-Type Fail-Authentication { (0) update control { (0) Response-Packet-Type := Access-Accept (0) } # update control = noop (0) policy accept { (0) update control { (0) &Response-Packet-Type = Access-Accept (0) } # update control = noop (0) [handled] = handled (0) } # policy accept = handled (0) } # Post-Proxy-Type Fail-Authentication = handled (0) There was no response configured: rejecting request <- How do I configure a response? (0) Using Post-Auth-Type Reject (0) # Executing group from file ./sites-enabled/default (0) Post-Auth-Type REJECT { We'll work on simplifying that for 3.2, also.
Thanks for all your help on this, the fail-over with the second server being the virtual seems to work well, just means I am restricted to a single server and can't use load-balance. But having this config would be my ideal:
home_server_pool ProxyDestPool { type = load-balance home_server = ProxyDest1 home_server = ProxyDest2 home_server = ProxyDest3 fallback = cacheuser }
That works for me. When all home servers in a "load-balance" pool are down, it uses the fallback virtual server:
(0) } # authorize = updated Home server pool example.net failing over to fallback example.net Proxying to virtual server example.net
I think this must work in 3.1 as it doesn't work for me in 3.0.x head from last week, as I just tried this and fallback didn't seem to get applied at all. I'll test 3.1 head tomorrow morning.
On Mar 8, 2016, at 4:24 AM, Peter Lambrechtsen <peter@crypt.co.nz> wrote:
This doesn't seem to work in 3.0.x head, I will test it on 3.1.x tomorrow.
I've pushed a fix.
I think this must work in 3.1 as it doesn't work for me in 3.0.x head from last week, as I just tried this and fallback didn't seem to get applied at all.
v3.0.x head worked for me yesterday what I tried that. The "fallback" code for home_server_pools is independent of the type of the home_server_pool. You may be running into a timer issue... i.e. if the timers are short, the home_server is marked alive, and the fallback is never used. I used "radmin" to forcibly set the home_server state to "dead". That avoids the timer issues, and the fallback works correctly. Alan DeKok.
On Wed, Mar 9, 2016 at 5:19 AM, Alan DeKok <aland@deployingradius.com> wrote:
On Mar 8, 2016, at 4:24 AM, Peter Lambrechtsen <peter@crypt.co.nz> wrote:
This doesn't seem to work in 3.0.x head, I will test it on 3.1.x tomorrow.
I've pushed a fix.
That's fixed it... Brilliant :)
I think this must work in 3.1 as it doesn't work for me in 3.0.x head from last week, as I just tried this and fallback didn't seem to get applied at all.
v3.0.x head worked for me yesterday what I tried that.
The "fallback" code for home_server_pools is independent of the type of the home_server_pool.
You may be running into a timer issue... i.e. if the timers are short, the home_server is marked alive, and the fallback is never used.
I used "radmin" to forcibly set the home_server state to "dead". That avoids the timer issues, and the fallback works correctly.
I think that was my issue, as I was using a second VM on the network as the proxy destination I was shutting down the destination server and not waiting for the zombie period to expire. (9) } # authorize = updated Home server pool ProxyDestPool failing over to fallback cacheuser (9) # Executing section pre-proxy from file ./sites-enabled/default That seems to be my issue, I've just re-tested that with 3.0.x head and had the zombie_timeout set too high. After I wound that number down to the same as check_interval and once the server went to zombie then the fallback occurred. zombie_period = 10 check_interval = 10 num_answers_to_alive = 2 This way once the server has been offline for 10 seconds it's zombied and fallback occurs. Then the check interval 10 x 2 means after the server has been back up and responded to 2x alives then it goes back to the remote proxy server. Granted I won't have the values set this low in production, but since this will be a high volume server with some critical services on it. I suspect I will stick with 30 seconds or 1 min for the check interval but keep the zombie value at 20 seconds. So if a radius server dies or becomes unresponsive we don't wait around until we mark it zombie before we start authing everyone locally. Then have a reasonable backoff before we attempt to start authing again. Many thanks again. Peter
On Mar 8, 2016, at 4:54 PM, Peter Lambrechtsen <peter@crypt.co.nz> wrote:
I think that was my issue, as I was using a second VM on the network as the proxy destination I was shutting down the destination server and not waiting for the zombie period to expire.
Yeah. It's documented, but it's not immediately obvious.
That seems to be my issue, I've just re-tested that with 3.0.x head and had the zombie_timeout set too high. After I wound that number down to the same as check_interval and once the server went to zombie then the fallback occurred.
Good.
Granted I won't have the values set this low in production, but since this will be a high volume server with some critical services on it. I suspect I will stick with 30 seconds or 1 min for the check interval but keep the zombie value at 20 seconds.
The check interval can be set lower without any problem. It's only one RADIUS packet.
So if a radius server dies or becomes unresponsive we don't wait around until we mark it zombie before we start authing everyone locally. Then have a reasonable backoff before we attempt to start authing again.
Yes. It's OK to set zombie_period to a low value. In any normal deployment, you'll be proxying many packets a second to a home server. If it doesn't respond to *any* packets for 10 seconds, you're pretty sure it's dead. 1 to 2 seconds is probably too low, as there may be transient network issues which are that long.
Many thanks again.
You're welcome. Alan DeKok.
participants (2)
-
Alan DeKok -
Peter Lambrechtsen