Zombie Infestation of Log file
Good day, I'm trying to figure out why my servers continue to be marked zombie, even though they continue to handle traffic. There appears to be no impact, just seemingly erroneous - or at least unexplained - log entries. I have three 2.1.8 servers that feeds accounting to a 4th server (via copy_acct_to_homeserver), running 1.1.7. The primary servers also sends some auth (via proxy) and lots of acct (some via proxy, but also via copy_acct_to_homeserver) to a pair of Cisco ACS servers. The radius.log file for the primary servers show they are marking the 4th and Cisco (upstream) servers as zombie quite regularly (but not simultaneously); they thankfully never get marked dead. All of these servers are attached to the same Ethernet Switch with a slight detour through a Router that does VLAN routing between them; The Cisco servers also proxy to various other servers outside my network. I have a debug output from one of the servers that I have studied at length; I recorded the debug for 6 minutes or so before one of the servers was marked zombie. This is from a production machine with a fair amount of traffic, so the debug file is 9MBs. I'm not sure if it would be appropriate to post it to the mailing list. I'd be happy to post it. Do you want specific excerpts, or the whole thing? I've set the response_window to as high as 60 seconds in the clients.conf file and I keep the zombie_period at 20 seconds. I've also turned off the status_check feature as 1.1.7 and Cisco ACS do not appear to support it. The clients.conf file says that after the response_window is up (so 60 seconds) and "a" response is not received, that the server is marked Zombie. Based on what I see, I'm interpreting this as meaning that if one response is not seen in 60 seconds, even if hundreds of other responses were successfully sent and received during those 60 seconds, then the server is marked Zombie. At this point the Zombie_Period kicks in, and the moment "any" successful response is received the server is marked as completely alive. In my case the Zombie_Period is canceled immediately (though sadly the log does not seem to show when Zombie ends.) Odd to me is that occasionally a primary server will mark the same upstream server as Zombie multiple times over a handful of seconds, but the other two primary servers rarely mark upstream servers dead near the same time. I cannot find in the debug, or in packet captures, where a response went missing for a full minute, so I'm trying to find out what is happening. My upstream servers do not appear taxed or unresponsive. Perhaps there is some sort of malformed response I should be looking for? I also get radutmp errors about a wrong NAS ID, though on brief analysis it doesn't appear related. Any suggestions to help track this down and eliminate the error messages is greatly appreciated. -Benjamin
On Tue, Apr 20, 2010 at 10:59:04PM -0800, Benjamin Marvin wrote:
The radius.log file for the primary servers show they are marking the 4th and Cisco (upstream) servers as zombie quite regularly (but not simultaneously);
I've set the response_window to as high as 60 seconds in the clients.conf file and I keep the zombie_period at 20 seconds.
What is your max_request_time? The radiusd.conf default is 30, which means that you it doesn't really make sense to have a proxy response_window close or larger than that (like 60) - your requests will get stuck on that particular first one and then get automatically cleaned up by the FR max_request_time handler after 30s.
I've also turned off the status_check feature as 1.1.7 and Cisco ACS do not appear to support it.
Without status_check, you rely on the timeouts - revive_interval and zombie_period. But, if you're talking to FR 1.1.7, that should be able to make it respond negatively to a single fake user/domain, and then you can use that for status_check = request on its clients. *Any* status_check is better on FR 2.x than none... speaking from horrible experience... -- 2. That which causes joy or happiness.
Josip Rodin wrote:
On Tue, Apr 20, 2010 at 10:59:04PM -0800, Benjamin Marvin wrote:
I've also turned off the status_check feature as 1.1.7 and Cisco ACS do not appear to support it.
You can configure a fake username && password for status checks. This *is* documented in raddb/proxy.conf.
Without status_check, you rely on the timeouts - revive_interval and zombie_period.
Which is much worse than status checks.
But, if you're talking to FR 1.1.7, that should be able to make it respond negatively to a single fake user/domain, and then you can use that for status_check = request on its clients.
*Any* status_check is better on FR 2.x than none... speaking from horrible experience...
Yup. It's not that 2.x is bad without status checks, it's that there is *no way* for anyone to do "the right thing" without status checks. Alan DeKok.
Hi,
Yup. It's not that 2.x is bad without status checks, it's that there is *no way* for anyone to do "the right thing" without status checks.
agreed - I'm behind status-checks all the way - either native sattus-check or a user who gets rejected. both work fine for testing upstream alive/readiness alan
On Wed, Apr 21, 2010 at 05:47:43PM +0200, Alan DeKok wrote:
Without status_check, you rely on the timeouts - revive_interval and zombie_period.
Which is much worse than status checks.
But, if you're talking to FR 1.1.7, that should be able to make it respond negatively to a single fake user/domain, and then you can use that for status_check = request on its clients.
*Any* status_check is better on FR 2.x than none... speaking from horrible experience...
Yup. It's not that 2.x is bad without status checks, it's that there is *no way* for anyone to do "the right thing" without status checks.
One thing that we talked I believe in private mail is good to point out on the mailing list as well - the current request cleaning up logic isn't really being kind to proxy settings and how the admins might interpret them - meaning there is nothing in the proxying code that will avoid having individual requests silently dropped on the floor if the timeouts expire. Indeed the only way to get the FR 2.x proxy code to retry and move on to a second home server in a pool is for its original *client* to retry too, within max_request_time, which can then trigger a fail-over in the proxying logic. -- 2. That which causes joy or happiness.
Josip Rodin wrote:
One thing that we talked I believe in private mail is good to point out on the mailing list as well - the current request cleaning up logic isn't really being kind to proxy settings and how the admins might interpret them - meaning there is nothing in the proxying code that will avoid having individual requests silently dropped on the floor if the timeouts expire.
The intent is to always call "post-proxy fail", but there are cases where that doesn't happen.
Indeed the only way to get the FR 2.x proxy code to retry and move on to a second home server in a pool is for its original *client* to retry too, within max_request_time, which can then trigger a fail-over in the proxying logic.
Yes. The server does *not* originate packets itself. It proxies packets only when the NAS sends it packets. This should be explained more clearly in proxy.conf. Alan DeKok.
participants (4)
-
Alan Buxey -
Alan DeKok -
Benjamin Marvin -
Josip Rodin