Script execute if all Proxies are dead
We’re using freeradius-2.2.6-6.el6_7 to proxy inbound radius requests to home servers. proxy.conf home_server_pool primary_radius_auth { type = fail-over home_server = server1.fqdn.com home_server = server2.fqdn.com home_server = server3.fqdn.com home_server = server4.fqdn.com } Works great, until last night when we found all home servers were erroring in the log file. Jan 19 21:48:55 proxy-server radiusd[17073]: No response to status check 16506 for home server 00.00.00.01 port 1812 Jan 19 21:48:57 proxy-server radiusd[17073]: No response to status check 16509 for home server 00.00.00.02 port 1812 Jan 19 21:49:00 proxy-server radiusd[17073]: No response to status check 16510 for home server 00.00.00.03 port 1812 Jan 19 21:49:00 proxy-server radiusd[17073]: No response to status check 16510 for home server 00.00.00.04 port 1812 After this was noticed, we restarted the radiusd service on, and it fixed the issue. My question is, any suggestions on how we could kick a shell script if the status check determines all home servers are down at a given point? I’d like to have it restart the service, in an attempt to self-heal. Thanks Chris
On Fri, Jan 20, 2017 at 09:25:31PM +0000, Chris Taylor (chtaylo2) wrote:
Works great, until last night when we found all home servers were erroring in the log file.
Jan 19 21:48:55 proxy-server radiusd[17073]: No response to status check 16506 for home server 00.00.00.01 port 1812 Jan 19 21:48:57 proxy-server radiusd[17073]: No response to status check 16509 for home server 00.00.00.02 port 1812 Jan 19 21:49:00 proxy-server radiusd[17073]: No response to status check 16510 for home server 00.00.00.03 port 1812 Jan 19 21:49:00 proxy-server radiusd[17073]: No response to status check 16510 for home server 00.00.00.04 port 1812
After this was noticed, we restarted the radiusd service on, and it fixed the issue. My question is, any suggestions on how we could kick a shell script if the status check determines all home servers are down at a given point? I’d like to have it restart the service, in an attempt to self-heal.
Sounds like if the proxy servers are up but FR hasn't realised then it's probably a bug - but on version 2 that's not going to get looked at. You should add upgrading to v3 onto your to-do list really. But maybe look at it in a different way - I assume when this happened then auth broke? In which case what do you care about... that the proxy servers are all down, or that auth isn't working? I sometimes hit similar issues with winbind and/or AD not responding (usually when a domain controller goes down and winbind doesn't move to another one). It doesn't happen too often any more thankfully. But my script doesn't care about which DC might be down; it does an auth and sees if that works. If not, then it tries a few things to get everything working again. Maybe a better way of looking at it? If it's useful, a version of the script I wrote to do this is at https://gist.github.com/mcnewton/8c6c54ffc04acf031a08. It lacks comments I'm afraid. Supposed to be run from cron every minute. Matthew -- Matthew Newton, Ph.D. <mcn4@leicester.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
On Fri, Jan 20, 2017 at 10:48:12PM +0000, Matthew Newton wrote:
If it's useful, a version of the script I wrote to do this is at https://gist.github.com/mcnewton/8c6c54ffc04acf031a08. It lacks comments I'm afraid. Supposed to be run from cron every minute.
I should add - to use this the ad_auth_check_ok would need to be replaced with a function to do e.g. radtest or eapol_test and check for the Access-Accept, or whatever else you need to look for. It also checks disc space - in my case if I'm over 95% usage I keep FR stopped, which means clients move over to other RADIUS servers. Otherwise things can go bad. But... you get the idea. May be better to do this sort of thing than to check your proxy servers. Matthew -- Matthew Newton, Ph.D. <mcn4@leicester.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
On Jan 20, 2017, at 4:25 PM, Chris Taylor (chtaylo2) <chtaylo2@cisco.com> wrote:
We’re using freeradius-2.2.6-6.el6_7 to proxy inbound radius requests to home servers.
Please upgrade. 2.2.6 is 3 years old, and there have been fixes to the v2 branch. Maybe there's a fix in v2.2.9, in which case you can upgrade to that without changing anything. Otherwise, upgrade to v3. It's the currently supported version.
Works great, until last night when we found all home servers were erroring in the log file.
Jan 19 21:48:55 proxy-server radiusd[17073]: No response to status check 16506 for home server 00.00.00.01 port 1812
If the home servers are unresponsive, there isn't much you can do. Tho v2.2.9 has fixes to proxying which may help.
After this was noticed, we restarted the radiusd service on, and it fixed the issue. My question is, any suggestions on how we could kick a shell script if the status check determines all home servers are down at a given point?
Why? FreeRADIUS should Do The Right Thing. i.e.
I’d like to have it restart the service, in an attempt to self-heal.
That's a bad work-around to a potential bug in the server. Why not fix the underlying problem? Alan DeKok.
Thank you! I should have made clearer, the home servers were not unresponsive, only the radiusd server thought they were. We have other servers in rotation which were working fine at the time. Appreciate the feedback. I’ll get upgrade to v3 on the books. Have a good weekend Chris On 1/20/17, 6:17 PM, "Freeradius-Users on behalf of Alan DeKok" <freeradius-users-bounces+chtaylo2=cisco.com@lists.freeradius.org on behalf of aland@deployingradius.com> wrote: On Jan 20, 2017, at 4:25 PM, Chris Taylor (chtaylo2) <chtaylo2@cisco.com> wrote: > > We’re using freeradius-2.2.6-6.el6_7 to proxy inbound radius requests to home servers. Please upgrade. 2.2.6 is 3 years old, and there have been fixes to the v2 branch. Maybe there's a fix in v2.2.9, in which case you can upgrade to that without changing anything. Otherwise, upgrade to v3. It's the currently supported version. > Works great, until last night when we found all home servers were erroring in the log file. > > > Jan 19 21:48:55 proxy-server radiusd[17073]: No response to status check 16506 for home server 00.00.00.01 port 1812 If the home servers are unresponsive, there isn't much you can do. Tho v2.2.9 has fixes to proxying which may help. > After this was noticed, we restarted the radiusd service on, and it fixed the issue. My question is, any suggestions on how we could kick a shell script if the status check determines all home servers are down at a given point? Why? FreeRADIUS should Do The Right Thing. i.e. > I’d like to have it restart the service, in an attempt to self-heal. That's a bad work-around to a potential bug in the server. Why not fix the underlying problem? Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
participants (3)
-
Alan DeKok -
Chris Taylor (chtaylo2) -
Matthew Newton