Zombie Clarification
Just to make sure I know what's going on... According to the docs, a server that does not respond to a request within the response_window (default: 20 sec) is considered a "zombie", and becomes eligible for the status checker. If no responses are received within the zombie_period (default: 40 sec), it is marked "dead". Once it's dead, it needs to respond to num_answers_to_alive requests before being marked as "alive". Assumign the zombie_period and check_interval are the defaults (40 sec, 30 sec), then the status checker would presumably run at least once during the zombie_period and bring a server back to life before it is marked dead. So here's the question... If the FR server receives no requests, then nothing gets forwarded to the home server. Does this scenario fall under "does not respond to a request within the response_window"? That is, should a quiet server constantly toggle in and out of zombie state? If so, then "zombie" is not really indicative of a problem and I can safely ignore the scenario. Hopefully this makes sense, and is useful to anyone else burrowing through the guts of the proxy status mechanism. Thanks! Norman Elton
Norman Elton wrote:
So here's the question... If the FR server receives no requests, then nothing gets forwarded to the home server. Does this scenario fall under "does not respond to a request within the response_window"?
Your question answers itself. Did the home server fail to respond to a request? No, because there was no request sent.
That is, should a quiet server constantly toggle in and out of zombie state?
The server doesn't do anything that stupid.
If so, then "zombie" is not really indicative of a problem and I can safely ignore the scenario.
The "zombie" state is there for a reason. Ignore zombies at your peril. Alan DeKok.
The "zombie" state is there for a reason. Ignore zombies at your peril.
Well, I understand how the alive/zombie/dead process SHOULD work, but I'm having trouble lining it up with what we're seeing. We're proxying to a windows NPS box. Here's the proxy config: home_server ias-1 { type = auth+acct ipaddr = 192.168.10.11 port = 1812 secret = "..." require_message_authenticator = yes response_window = 5 zombie_period = 20 revive_interval = 120 status_check = request username = "ping_user" password = "bad_password" check_interval = 10 num_answers_to_alive = 3 coa { irt = 2 mrt = 16 mrc = 5 mrd = 30 } limit { max_connections = 16 max_requests = 0 lifetime = 0 idle_timeout = 0 } } Now, for whatever reason, the Windows box decides to discard some requests. Unfortunately, the error reporting is pretty weak ("discarding invalid request"). Our Windows guys are digging into this. It seems to be client specific, we suspect something with our recently changed certificate. FreeRadius is dropping into zombie state, which is expected given that the home server is dropping requests. But our logs and packet captures indicate that the home server is never dropping the "ping_user" status checks that FR is using to determine the home server state. But, our FreeRadius logs indicate that the home_server is being flagged 'dead' immediately upon becoming zombie: Sun Mar 11 20:32:26 2012 : Proxy: Marking home server 192.168.10.11 port 1812 as zombie (it looks like it is dead). Sun Mar 11 20:32:26 2012 : Proxy: Received response to status check 133400 (1 in current sequence) Sun Mar 11 20:32:27 2012 : Proxy: Marking home server 192.168.10.11 port 1812 as dead. Sun Mar 11 20:32:27 2012 : Error: Discarding duplicate request from client aerohive-aps port 49034 - ID: 113 due to unfinished request 133390 Sun Mar 11 20:32:39 2012 : Proxy: Received response to status check 133441 (1 in current sequence) Sun Mar 11 20:32:39 2012 : Error: Discarding duplicate request from client aerohive-aps port 49034 - ID: 113 due to unfinished request 133390 Sun Mar 11 20:32:46 2012 : Error: Discarding duplicate request from client aerohive-aps port 46715 - ID: 29 due to unfinished request 133448 Sun Mar 11 20:32:47 2012 : Proxy: Received response to status check 133468 (2 in current sequence) Sun Mar 11 20:32:58 2012 : Error: Discarding duplicate request from client aerohive-aps port 46715 - ID: 29 due to unfinished request 133448 Sun Mar 11 20:32:59 2012 : Proxy: Received response to status check 133489 (3 in current sequence) Sun Mar 11 20:32:59 2012 : Proxy: Marking home server 192.168.10.11 port 1812 alive Sun Mar 11 20:33:23 2012 : Proxy: Marking home server 192.168.10.11 port 1812 as zombie (it looks like it is dead). Sun Mar 11 20:33:23 2012 : Proxy: Received response to status check 133580 (1 in current sequence) Sun Mar 11 20:33:27 2012 : Proxy: Marking home server 192.168.10.11 port 1812 as dead. Sun Mar 11 20:33:35 2012 : Proxy: Received response to status check 133621 (1 in current sequence) Sun Mar 11 20:33:51 2012 : Proxy: Received response to status check 133668 (2 in current sequence) Sun Mar 11 20:33:56 2012 : Proxy: Marking home server 192.168.10.12 port 1812 as zombie (it looks like it is dead). Sun Mar 11 20:33:56 2012 : Proxy: Received response to status check 133686 (1 in current sequence) Why is the server going into zombie state at 20:32:26 and immediately becoming dead at 20:32:27? Shouldn't it wait for the entire zombie_period before dropping dead? Thanks, Norman Elton
Norman Elton wrote:
Well, I understand how the alive/zombie/dead process SHOULD work, but I'm having trouble lining it up with what we're seeing. We're proxying to a windows NPS box. Here's the proxy config:
Part of the issue is that the timers on the proxy are independent of the timers on the client. The timers on the proxy fire ONLY when it receives packets from the client. So if the client doesn't retransmit, or if it retransmits on an unusual pattern, proxying becomes more difficult.
Now, for whatever reason, the Windows box decides to discard some requests. Unfortunately, the error reporting is pretty weak ("discarding invalid request"). Our Windows guys are digging into this. It seems to be client specific, we suspect something with our recently changed certificate.
I don't see how. Normal RADIUS doesn't use certificates. And if your home server *randomly* discards requests, then your priority should be to fix that. No amount of poking FreeRADIUS will make the home server magically work. No amount of poking FreeRADIUS will work around the fact that the home server is broken.
FreeRadius is dropping into zombie state, which is expected given that the home server is dropping requests. But our logs and packet captures indicate that the home server is never dropping the "ping_user" status checks that FR is using to determine the home server state. But, our FreeRadius logs indicate that the home_server is being flagged 'dead' immediately upon becoming zombie:
Check which version of the server you're running. Old versions sometimes had issues with zombie timers. See doc/ChangeLog for details.
Why is the server going into zombie state at 20:32:26 and immediately becoming dead at 20:32:27? Shouldn't it wait for the entire zombie_period before dropping dead?
Yes. Alan DeKok.
Hi, there was never any more on this thread, so just to add some final info
Now, for whatever reason, the Windows box decides to discard some requests. Unfortunately, the error reporting is pretty weak ("discarding invalid request"). Our Windows guys are digging into this. It seems to be client specific, we suspect something with our recently changed certificate.
I don't see how. Normal RADIUS doesn't use certificates.
And if your home server *randomly* discards requests, then your priority should be to fix that. No amount of poking FreeRADIUS will make the home server magically work. No amount of poking FreeRADIUS will work around the fact that the home server is broken.
Microsoft decided, in their wisdom, to just discard packets that arent right. this affects IAS and NPS. if your policy says, for example, NAS-Port-Type = Wireless-802.11 an the packet doesnt have that attribute...or its not Wireless-802.11..then the packet is just silently dropped. the RADIUS proxies throughout the proxy chain then think the server is dead.... status-server kicks in.... oh, guess what. they dont support that, so it stays marked dead. the remote proxies might be lucky...as their status-server will be answered by the proxy above them...which, if its FreeRADIUS or RADIATOR *will* respond in some way to show they are alive. IAS and NPS are a mess with proxied RADIUS - especially when there are policies involved. alan
Alan Buxey wrote:
Microsoft decided, in their wisdom, to just discard packets that arent right. this affects IAS and NPS. if your policy says, for example,
NAS-Port-Type = Wireless-802.11
an the packet doesnt have that attribute...or its not Wireless-802.11..then the packet is just silently dropped.
This violates RFC 2865. <sigh> Not that they care.
IAS and NPS are a mess with proxied RADIUS - especially when there are policies involved.
This should be on the Wiki, in a "what's wrong with other RADIUS servers" section. :) Alan DeKok.
On 24/03/2012 13:13, Alan Buxey wrote:
Hi,
there was never any more on this thread, so just to add some final info
Now, for whatever reason, the Windows box decides to discard some requests. Unfortunately, the error reporting is pretty weak ("discarding invalid request"). Our Windows guys are digging into this. It seems to be client specific, we suspect something with our recently changed certificate.
I don't see how. Normal RADIUS doesn't use certificates.
And if your home server *randomly* discards requests, then your priority should be to fix that. No amount of poking FreeRADIUS will make the home server magically work. No amount of poking FreeRADIUS will work around the fact that the home server is broken.
Microsoft decided, in their wisdom, to just discard packets that arent right. this affects IAS and NPS. if your policy says, for example,
NAS-Port-Type = Wireless-802.11
an the packet doesnt have that attribute...or its not Wireless-802.11..then the packet is just silently dropped. the RADIUS proxies throughout the proxy chain then think the server is dead.... status-server kicks in.... oh, guess what. they dont support that, so it stays marked dead. the remote proxies might be lucky...as their status-server will be answered by the proxy above them...which, if its FreeRADIUS or RADIATOR *will* respond in some way to show they are alive.
IAS and NPS are a mess with proxied RADIUS - especially when there are policies involved.
Further to what Alan says above IAS/NPS can report "invalid request" if it contains an attribute not in their dictionaries, or an attribute where the value does not match the type in their dictionaries. As NPS and IAS dictionaries are old, don't match the RFCs, and it seems MS never update the dictionaries, this means NPS and IAS discard a lot of valid packets! If you are proxying to IAS or NPS, filter the attributes very carefully before they hit the MS radius servers. Regards, James
participants (4)
-
Alan Buxey -
Alan DeKok -
James J J Hooper -
Norman Elton