Script execute if all Proxies are dead

[Matthew Newton]

On Fri, Jan 20, 2017 at 09:25:31PM +0000, Chris Taylor (chtaylo2) wrote:
> Works great, until last night when we found all home servers were erroring in the log file. 
> 
> Jan 19 21:48:55 proxy-server radiusd[17073]: No response to status check 16506 for home server 00.00.00.01 port 1812
> Jan 19 21:48:57 proxy-server radiusd[17073]: No response to status check 16509 for home server 00.00.00.02 port 1812
> Jan 19 21:49:00 proxy-server radiusd[17073]: No response to status check 16510 for home server 00.00.00.03 port 1812
> Jan 19 21:49:00 proxy-server radiusd[17073]: No response to status check 16510 for home server 00.00.00.04 port 1812
> 
> After this was noticed, we restarted the radiusd service on, and
> it fixed the issue.   My question is, any suggestions on how we
> could kick a shell script if the status check determines all
> home servers are down at a given point? I’d like to have it
> restart the service, in an attempt to self-heal. 

Sounds like if the proxy servers are up but FR hasn't realised
then it's probably a bug - but on version 2 that's not going to
get looked at. You should add upgrading to v3 onto your to-do list
really.

But maybe look at it in a different way - I assume when this
happened then auth broke? In which case what do you care about...
that the proxy servers are all down, or that auth isn't working?

I sometimes hit similar issues with winbind and/or AD not
responding (usually when a domain controller goes down and winbind
doesn't move to another one). It doesn't happen too often any more
thankfully. But my script doesn't care about which DC might be
down; it does an auth and sees if that works. If not, then it
tries a few things to get everything working again.

Maybe a better way of looking at it?

If it's useful, a version of the script I wrote to do this is at
https://gist.github.com/mcnewton/8c6c54ffc04acf031a08. It lacks
comments I'm afraid. Supposed to be run from cron every minute.

Matthew


-- 
Matthew Newton, Ph.D. <mcn4 at leicester.ac.uk>

Systems Specialist, Infrastructure Services,
I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom

For IT help contact helpdesk extn. 2253, <ithelp at le.ac.uk>