Handling unreliable proxy partners
We proxy a lot of authentication to radius servers that we do not manage/control. In a number of failure scenarios in order to maintain a good customer experience it is better for us to give access to the network even though we can't authenticate the users so customers continue to receive service even if that means giving some non-customer access. We might restrict the length of these sessions or turn on QoS restrictions. Using fallback virtual servers and (in non fallback virtual servers) Post-Proxy-Type Fail-Authentication allows us to handle the cases where no radius servers are responding for a partner or proxying of a single request fails, by returning an access-accept ourselves and automatically kicks in when one of those situations presents itself. Any common radius attributes that need to be set to put restrictions on those sessions can be put in a policy and called from each section rather than duplicating code. This works well. We'd also like a manual mechanism that our support team can trigger to cover other failure scenarios, eg the remote radius server is incorrectly returning access-reject for all valid users, and those scenarios that we haven't been able to think of but will occur, inevitably at the most inconvenient of times. My first attempt at this was that the support team could use radmin to set the home servers to dead which would mean packets were routed via the falback virtual server. I initially thought this worked as a solution, but if FreeRadius is doing status checks against the remote servers then it will automatically bring them back into service as long as the status check requests are responding, which if say the remote partner is responding with access-rejects to even valid users is not what you want. I also considered using ip tables to drop the packets, which would trigger the use of the fallback virtual server, but if you'd decided to go into this state of accepting everything because of some intermittent network problem then dropping all packets with ip tables would make debugging that problem using radclient/radtest very difficult. One idea I haven't explored is having two copies of each virtual server, in different files, one for the normal situation and one for the failure situation and switching which one to use using symlinks and radmin to reload the configuration. So far what I have come up with so far is within a virtual server pre-proxy section to use the exec module to call a simple shell script that check for the presence of flag files indicating which if any partners are in a bad state. The support team are responsible for creating these files. If any flag files are present the the script adds a radius attribute for each, the value indicating which partner. In the pre-proxy section I can then check for this attribute and value if it indicates that the partner the virtual server is handling is in a failure state then call accept from the always module which will cancel the proxying attempt and send an access-accept. We can also call any policy that would also get called in the fallback virtual server or Post-Proxy-Type Fail-Authentication if we want common radius attributes to be returned in the response to apply some sort of QoS restriction. The rlm_exec documentation states using exec is very slow and something like the perl module would be more appropriate for a live environment. Before I carry on down the path of performance testing this and trying perl/python/rest/custom C module does anyone have any thoughts/observations or alternative suggestions? Thanks, Paul
I wrote code to cache every user of the downstream systems into a local ldap database during post-auth with each realm as a sub OU in the directory as we were only using username and password rather than EAP-TLS. That way I could cache and encrypt the password as well as any additional VSAs the client was sending as some had a per user static IP or a select range of VSAs they could add. Then I had a very low retry and timeout values for them and used a fail virtual server to handle looking up ldap based on the realm them check the password and return any VSAs as needed. Worked well for some very critical systems and all of the complaints about general flakiness for some clients just went away. I think I had something like 500 downstream realms but the code just created realms in ldap as needed using some Perl. Also added a attribute on the OU to disable caching if the customer wanted the requests to fail if their server wasn’t responding. Hardest part was having notifications on up/down of the virtual server as it just handled it in the background. On Thu, 20 May 2021 at 05:20, Paul Moser via Freeradius-Users < freeradius-users@lists.freeradius.org> wrote:
We proxy a lot of authentication to radius servers that we do not manage/control.
In a number of failure scenarios in order to maintain a good customer experience it is better for us to give access to the network even though we can't authenticate the users so customers continue to receive service even if that means giving some non-customer access. We might restrict the length of these sessions or turn on QoS restrictions.
Using fallback virtual servers and (in non fallback virtual servers) Post-Proxy-Type Fail-Authentication allows us to handle the cases where no radius servers are responding for a partner or proxying of a single request fails, by returning an access-accept ourselves and automatically kicks in when one of those situations presents itself. Any common radius attributes that need to be set to put restrictions on those sessions can be put in a policy and called from each section rather than duplicating code.
This works well.
We'd also like a manual mechanism that our support team can trigger to cover other failure scenarios, eg the remote radius server is incorrectly returning access-reject for all valid users, and those scenarios that we haven't been able to think of but will occur, inevitably at the most inconvenient of times.
My first attempt at this was that the support team could use radmin to set the home servers to dead which would mean packets were routed via the falback virtual server. I initially thought this worked as a solution, but if FreeRadius is doing status checks against the remote servers then it will automatically bring them back into service as long as the status check requests are responding, which if say the remote partner is responding with access-rejects to even valid users is not what you want.
I also considered using ip tables to drop the packets, which would trigger the use of the fallback virtual server, but if you'd decided to go into this state of accepting everything because of some intermittent network problem then dropping all packets with ip tables would make debugging that problem using radclient/radtest very difficult.
One idea I haven't explored is having two copies of each virtual server, in different files, one for the normal situation and one for the failure situation and switching which one to use using symlinks and radmin to reload the configuration.
So far what I have come up with so far is within a virtual server pre-proxy section to use the exec module to call a simple shell script that check for the presence of flag files indicating which if any partners are in a bad state. The support team are responsible for creating these files. If any flag files are present the the script adds a radius attribute for each, the value indicating which partner. In the pre-proxy section I can then check for this attribute and value if it indicates that the partner the virtual server is handling is in a failure state then call accept from the always module which will cancel the proxying attempt and send an access-accept. We can also call any policy that would also get called in the fallback virtual server or Post-Proxy-Type Fail-Authentication if we want common radius attributes to be returned in the response to apply some sort of QoS restriction.
The rlm_exec documentation states using exec is very slow and something like the perl module would be more appropriate for a live environment. Before I carry on down the path of performance testing this and trying perl/python/rest/custom C module does anyone have any thoughts/observations or alternative suggestions?
Thanks,
Paul - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On May 19, 2021, at 1:20 PM, Paul Moser via Freeradius-Users <freeradius-users@lists.freeradius.org> wrote:
We'd also like a manual mechanism that our support team can trigger to cover other failure scenarios, eg the remote radius server is incorrectly returning access-reject for all valid users, and those scenarios that we haven't been able to think of but will occur, inevitably at the most inconvenient of times.
My first attempt at this was that the support team could use radmin to set the home servers to dead which would mean packets were routed via the falback virtual server. I initially thought this worked as a solution, but if FreeRadius is doing status checks against the remote servers then it will automatically bring them back into service as long as the status check requests are responding, which if say the remote partner is responding with access-rejects to even valid users is not what you want.
Yes. The server tries very hard to proxy packets.
One idea I haven't explored is having two copies of each virtual server, in different files, one for the normal situation and one for the failure situation and switching which one to use using symlinks and radmin to reload the configuration.
That's probably too complex. I wouldn't recommend it.
So far what I have come up with so far is within a virtual server pre-proxy section to use the exec module to call a simple shell script that check for the presence of flag files indicating which if any partners are in a bad state. The support team are responsible for creating these files. If any flag files are present the the script adds a radius attribute for each, the value indicating which partner. In the pre-proxy section I can then check for this attribute and value if it indicates that the partner the virtual server is handling is in a failure state then call accept from the always module which will cancel the proxying attempt and send an access-accept. We can also call any policy that would also get called in the fallback virtual server or Post-Proxy-Type Fail-Authentication if we want common radius attributes to be returned in the response to apply some sort of QoS restriction.
The rlm_exec documentation states using exec is very slow and something like the perl module would be more appropriate for a live environment. Before I carry on down the path of performance testing this and trying perl/python/rest/custom C module does anyone have any thoughts/observations or alternative suggestions?
Use "rlm_always". From the Changelog for 3.0.22: * New xlat for setting status of rlm_always instances and new resource-check example virtual server for manipulating control flow in unlang policies based on status of some external resource. Patches from Terry Burton. https://github.com/FreeRADIUS/freeradius-server/blob/v3.0.x/raddb/mods-avail... You can also also use "radmin" to poke the "always" configuration live: radmin> set module config always rcode fail The idea would be to create an instance of the "always" module, for each home server / pool you're proxying to. You can then do something like: always server_x_status { rcode = ok } and then server_x_status if (ok) { proxy to server x } And then you can set that to ok / fail, depending on whatever you want. Alan DeKok.
On Thu, 20 May 2021 at 05:37, Alan DeKok <aland@deployingradius.com> wrote:
On May 19, 2021, at 1:20 PM, Paul Moser via Freeradius-Users < freeradius-users@lists.freeradius.org> wrote:
We'd also like a manual mechanism that our support team can trigger to cover other failure scenarios, eg the remote radius server is incorrectly returning access-reject for all valid users, and those scenarios that we haven't been able to think of but will occur, inevitably at the most inconvenient of times.
My first attempt at this was that the support team could use radmin to set the home servers to dead which would mean packets were routed via the falback virtual server. I initially thought this worked as a solution, but if FreeRadius is doing status checks against the remote servers then it will automatically bring them back into service as long as the status check requests are responding, which if say the remote partner is responding with access-rejects to even valid users is not what you want.
Yes. The server tries very hard to proxy packets.
One idea I haven't explored is having two copies of each virtual server, in different files, one for the normal situation and one for the failure situation and switching which one to use using symlinks and radmin to reload the configuration.
That's probably too complex. I wouldn't recommend it.
So far what I have come up with so far is within a virtual server pre-proxy section to use the exec module to call a simple shell script that check for the presence of flag files indicating which if any partners are in a bad state. The support team are responsible for creating these files. If any flag files are present the the script adds a radius attribute for each, the value indicating which partner. In the pre-proxy section I can then check for this attribute and value if it indicates that the partner the virtual server is handling is in a failure state then call accept from the always module which will cancel the proxying attempt and send an access-accept. We can also call any policy that would also get called in the fallback virtual server or Post-Proxy-Type Fail-Authentication if we want common radius attributes to be returned in the response to apply some sort of QoS restriction.
The rlm_exec documentation states using exec is very slow and something like the perl module would be more appropriate for a live environment. Before I carry on down the path of performance testing this and trying perl/python/rest/custom C module does anyone have any thoughts/observations or alternative suggestions?
Use "rlm_always". From the Changelog for 3.0.22:
* New xlat for setting status of rlm_always instances and new resource-check example virtual server for manipulating control flow in unlang policies based on status of some external resource. Patches from Terry Burton.
https://github.com/FreeRADIUS/freeradius-server/blob/v3.0.x/raddb/mods-avail...
You can also also use "radmin" to poke the "always" configuration live:
radmin> set module config always rcode fail
The idea would be to create an instance of the "always" module, for each home server / pool you're proxying to. You can then do something like:
always server_x_status { rcode = ok }
and then
server_x_status if (ok) { proxy to server x }
And then you can set that to ok / fail, depending on whatever you want.
Alan DeKok.
I found Replicate-To-Realm very useful here. I could proxy the request to a secondary server to process the request and create the database entries and check if the password was correct and decided if I needed to update the database or not out of band from the main instance as the database couldn’t write as quickly as FR could process requests. I can dig up the code and sanitise it and post it publicly if that would help. It was using FR3 and an LDAP backend and dynamic clients. I remember asking if dynamic realms would be possible but it turns out it’s just easier having a static file and doing rolling updates of the proxies when a new realm came onboard.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Use "rlm_always". From the Changelog for 3.0.22:
* New xlat for setting status of rlm_always instances and new resource-check example virtual server for manipulating control flow in unlang policies based on status of some external resource. Patches from Terry Burton. Ah - looks promising I'd better go and re-read the 3.0.22 release notes and check what other new toys I've missed. Thanks.
I've just pushed changes to v3 which lets you use radmin: radmin> set home_server state <ipaddr> <port> [udp|tcp] down which marks the home server as "down", and does not try to revive it. it can only be revived with: radmin> set home_server state <ipaddr> <port> [udp|tcp] alive That should make some things simpler. Alan DeKok.
Hi Alan would the use of other keywords br easier/less confusing like: disable | enable instead of: down | alive Cheers Jonathan On Fri, 21 May 2021, 14:51 Alan DeKok, <aland@deployingradius.com> wrote:
I've just pushed changes to v3 which lets you use radmin:
radmin> set home_server state <ipaddr> <port> [udp|tcp] down
which marks the home server as "down", and does not try to revive it. it can only be revived with:
radmin> set home_server state <ipaddr> <port> [udp|tcp] alive
That should make some things simpler.
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On May 22, 2021, at 8:31 PM, Jonathan <huffelduffel@gmail.com> wrote:
would the use of other keywords br easier/less confusing like: disable | enable
instead of: down | alive
Sure. You can open a PR on GitHub. The one note is that v3 is the stable release, and we will not be deleting or changing existing functionality, unless it is required for security. Alan DeKok.
I've just pushed changes to v3 which lets you use radmin: radmin> set home_server state <ipaddr> <port> [udp|tcp] down which marks the home server as "down", and does not try to revive it. it can only be revived with: radmin> set home_server state <ipaddr> <port> [udp|tcp] alive That should make some things simpler.
Yes, looks like an even simpler solution than the rlm_always. I'll give both a try. Thanks, Paul
I've tried this now and it looks like it's going to work really well for us. Thanks. One minor issue - when a home server has been marked as down radmin show home_server state <ipaddr> <port> shows the state as invalid. It looks like command.c needs a case adding for HOME_STATE_ADMIN_DOWN in command_show_home_server_state(). radmin show home_server list works fine. Paul ________________________________ From: Freeradius-Users <freeradius-users-bounces+paul.moser=bt.com@lists.freeradius.org> on behalf of Alan DeKok <aland@deployingradius.com> Sent: 21 May 2021 13:50 To: FreeRadius users mailing list <freeradius-users@lists.freeradius.org> Subject: Re: Handling unreliable proxy partners I've just pushed changes to v3 which lets you use radmin: radmin> set home_server state <ipaddr> <port> [udp|tcp] down which marks the home server as "down", and does not try to revive it. it can only be revived with: radmin> set home_server state <ipaddr> <port> [udp|tcp] alive That should make some things simpler. Alan DeKok. - List info/subscribe/unsubscribe? See https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.freeradius.org%2Flist%2Fusers.html&data=04%7C01%7Cpaul.moser%40bt.com%7C6d88ca317f9b4e6b1b4708d91c573072%7Ca7f356889c004d5eba4129f146377ab0%7C0%7C0%7C637571983062776261%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=jjNvp3qBq%2FpUPl33RSMalTr3nQsA1nHf12RVFTeP5rA%3D&reserved=0
On May 27, 2021, at 10:46 AM, Paul Moser via Freeradius-Users <freeradius-users@lists.freeradius.org> wrote:
I've tried this now and it looks like it's going to work really well for us. Thanks.
Good to hear.
One minor issue - when a home server has been marked as down radmin show home_server state <ipaddr> <port> shows the state as invalid. It looks like command.c needs a case adding for HOME_STATE_ADMIN_DOWN in command_show_home_server_state().
I've pushed a fix. Alan DeKok.
participants (4)
-
Alan DeKok -
Jonathan -
paul.moser＠bt.com -
Peter Lambrechtsen