Home servers constantly zombied, and I can't figure out how to fix it
Good morning, everybody. I have a problem. I have FreeRADIUS 2.1.3 servers that are proxying accounting information to two remote RADIUS servers (radiator, if it matters.) My problem is that the two servers I am sending to are constantly declared zombies. Perhaps related is that in packet traces on the RADIUS servers, I see my RADIUS servers sending duplicate packets. I do not know if the duplicate packets are because the NAS is sending duplicate packets to me (it is indeed sending duplicate packets, according to wireshark), or if it is something on the RADIUS server's end. Furthermore, in wireshark, I also see plenty of 'Malformed Packets', but I don't know if that's because the packet is *truly* Malformed, or if it is because wireshark is having some issues (the RADIUS servers are VMWare Virtual Machines, and I've seen previously that various things can cause wireshark to detect malformed packets when they actually are fine.) I have been making a lot of configuration changes (esp. with regard to the check interval, number of responses before alive, etc) - so if anything is seriously out of whack, let me know - but it seems that no matter what, those systems get marked as zombies by my RADIUS servers a half a dozen times a minute. I'm pasting some of my relevant configs below; if anything is way out of line, please let me know (it may be from my desperate tweaking). Any minor inconsistencies may be from me 'cleaning it up' so I'm not broadcasting any useful information. (Please note: I have my status_check as 'request' even though the other side DOES support the status-server, and will likely go back to that once this is all over, and it doesn't break things.) Adam radiusd.conf: max_request_time = 90 cleanup_delay = 10 max_requests = 262144 hostname_lookups = no thread pool { start_servers = 10 max_servers = 128 min_spare_servers = 3 max_spare_servers = 30 max_requests_per_server = 0 } ############################# sites-enabled/accounting.conf : home_server acct1 { type = acct ipaddr = xxx.xxx.xxx.xxx port = 1813 secret = xxxxxxxxxxx response_window = 60 zombie_period = 120 check_interval = 10 num_answers_to_alive = 1 status_check = request username = "xxxxxxxxxxxx" password = "xxxxxxxxxxxx" } home_server acct2 { type = acct ipaddr = yyy.yyy.yyy.yyy port = 1813 secret = yyyyyyyyyyyyy response_window = 60 zombie_period = 120 check_interval = 10 num_answers_to_alive = 1 status_check = request username = "xxxxx" password = "xxxxx" } home_server_pool acct-pool { type = load-balance home_server = acct1 home_server = acct2 } realm myacct { acct_pool = acct-pool } ################## detail-reader.conf : server acct-relay { listen { type = detail filename = (my detail file) load_factor = 100 } accounting { ok } pre-proxy { } post-proxy { }
Adam Bultman wrote:
I have FreeRADIUS 2.1.3 servers that are proxying accounting information to two remote RADIUS servers (radiator, if it matters.)
It could matter. http://www.cesnet.cz/doc/techzpravy/2008/eduroam-authentication-over-jammed-... 40% packet loss (client to server) => 78% success rate for PEAP-MSCHAPv2 40% packet loss (server to client) => 20% success rate for PEAP-MSCHAPv2 Huh? 40% packet loss means that when the client sends a request, 40% of the time it doesn't see a response to that request. It shouldn't matter if the request is lost, or of the response is lost. But it does. The paper says: "The more favorable results for client-to-server jamming are caused by the aggressive packet re-sending strategy of wpa_supplicant compared to the behavior of Radiator." I have no idea what that means. If the server receives a packet, it should respond to it. The graphs in the paper seems to indicate that client retransmits are being *ignored* by radiator. i.e. If there's packet loss client to server, then when a packet is lost, the server doesn't see it. The client retransmits, the server sees the packet, and responds. On the other hand, if there's packet loss server to client, the client sends a packet, the server responds. The client doesn't see the response, so it retransmits the packet. The server then sees the retransmit, and ignores the packet. As an indication that this is happening, see: http://www.open.com.au/radiator/history.html See Revision 4.0 (2008-01-14), reference to RFC 5080. They implemented the duplicate detection cache which has been in FreeRADIUS since day 1. (1999). My suggestion is to try replacing one of the home servers with FreeRADIUS. It will respond to any retransmit. So if there are packet loss problems, they should be less problematic.
My problem is that the two servers I am sending to are constantly declared zombies. Perhaps related is that in packet traces on the RADIUS servers, I see my RADIUS servers sending duplicate packets. I do not know if the duplicate packets are because the NAS is sending duplicate packets to me (it is indeed sending duplicate packets, according to wireshark), or if it is something on the RADIUS server's end.
The NAS is likely sending retransmits because it isn't seeing a response.
Furthermore, in wireshark, I also see plenty of 'Malformed Packets', but I don't know if that's because the packet is *truly* Malformed, or if it is because wireshark is having some issues (the RADIUS servers are VMWare Virtual Machines, and I've seen previously that various things can cause wireshark to detect malformed packets when they actually are fine.)
I'm not sure. If FreeRADIUS isn't complaining that the packets are malformed, then they should be OK. See the statistics (radmin / snmp) for counts of malformed packets.
I have been making a lot of configuration changes (esp. with regard to the check interval, number of responses before alive, etc) - so if anything is seriously out of whack, let me know - but it seems that no matter what, those systems get marked as zombies by my RADIUS servers a half a dozen times a minute.
The servers are marked zombie when the proxy sends a request, and doesn't see a response in 30s. That's a little aggressive. The current logic doesn't take into account if *other* packets have received responses. It should probably mark the home server "zombie" only if there have been *no* responses in the "zombie" time interval. I'd suggest replacing one of the home servers with FreeRADIUS. If that makes a big difference for the proxy, then the Radiator server is borked. Alan DeKok.
Alan DeKok wrote:
Adam Bultman wrote:
I have FreeRADIUS 2.1.3 servers that are proxying accounting information to two remote RADIUS servers (radiator, if it matters.)
It could matter.
Checking the status-server packets, it appears that their servers are stating they are radiator version 3.15 - which according to Radiator's web site, is from June of 2006. Holy crap.
As an indication that this is happening, see:
http://www.open.com.au/radiator/history.html
See Revision 4.0 (2008-01-14), reference to RFC 5080. They implemented the duplicate detection cache which has been in FreeRADIUS since day 1. (1999).
Yeah, they're still 1.5 years away from 4.0, if their server-status stuff is returning the correct version.
My suggestion is to try replacing one of the home servers with FreeRADIUS. It will respond to any retransmit. So if there are packet loss problems, they should be less problematic.
Er, I can't do that; it's not the same company, not the same state. They're tight-lipped about their radius server, their firewalls, their network setups - I have no clue what their systems are doing at any given time, so it's hard to know what's going on their end.
My problem is that the two servers I am sending to are constantly declared zombies. Perhaps related is that in packet traces on the RADIUS servers, I see my RADIUS servers sending duplicate packets. I do not know if the duplicate packets are because the NAS is sending duplicate packets to me (it is indeed sending duplicate packets, according to wireshark), or if it is something on the RADIUS server's end.
The NAS is likely sending retransmits because it isn't seeing a response.
I have been making a lot of configuration changes (esp. with regard to the check interval, number of responses before alive, etc) - so if anything is seriously out of whack, let me know - but it seems that no matter what, those systems get marked as zombies by my RADIUS servers a half a dozen times a minute.
The servers are marked zombie when the proxy sends a request, and doesn't see a response in 30s.
That's a little aggressive. The current logic doesn't take into account if *other* packets have received responses. It should probably mark the home server "zombie" only if there have been *no* responses in the "zombie" time interval.
How do I change that functionality? I'd *love* it if it didn't zombie their servers for no reason. When I do a radiusd -CXXX, I see options I don't see documented for the latest releases of freeradius: - ping_check - ping_interval - num_pings_to_alive - max_outstanding (I can't even find what this is for) I've googled for these (and some others) and not sure if they're used, or what - perhaps they are deprecated, but show up in the config check anyway. As it is, my *.work files are "stuck" (And I've googled for that, and found other list posts regarding that) which seems to indicate that the home servers aren't responding... except that even when my detail.work file is 'stuck' at 24k, and the detail file keeps growing, I'm still sending data to the other side. So something's working, but only sort of..
I'd suggest replacing one of the home servers with FreeRADIUS. If that makes a big difference for the proxy, then the Radiator server is borked.
I'm about to shoot an email to them to see if they can explain their 4 year old radius software, and perhaps maybe that's part of the problem. Thanks for your help. Adam -- Adam
Adam Bultman wrote:
How do I change that functionality? I'd *love* it if it didn't zombie their servers for no reason.
No.. it marks the servers zombie for a reason: they're not responding. But it may be too aggressive.
When I do a radiusd -CXXX, I see options I don't see documented for the latest releases of freeradius: - ping_check - ping_interval - num_pings_to_alive
Those are for backwards compatibility with pre-releases of 2.0. They should be removed. They are just different names for the status-server checks.
- max_outstanding (I can't even find what this is for)
You can put a limit on the total number of "outstanding" packets sent to a home server. i.e. put it at 256, and if there are 256 packets sent without a response, the proxy will *not* use that home server again, until it gets at least one response. This is a way to do load-limiting on home servers.
As it is, my *.work files are "stuck" (And I've googled for that, and found other list posts regarding that) which seems to indicate that the home servers aren't responding... except that even when my detail.work file is 'stuck' at 24k, and the detail file keeps growing, I'm still sending data to the other side. So something's working, but only sort of..
It's re-transmitting the same packet over and over. If you install 2.1.9, you can use "radmin" to see its progress in reading the detail file.
I'm about to shoot an email to them to see if they can explain their 4 year old radius software, and perhaps maybe that's part of the problem.
Yup. They can upgrade to a (cough) real radius server. :) Alan DeKok.
Alan DeKok wrote:
Adam Bultman wrote:
How do I change that functionality? I'd *love* it if it didn't zombie their servers for no reason.
No.. it marks the servers zombie for a reason: they're not responding. But it may be too aggressive.
When I do a radiusd -CXXX, I see options I don't see documented for the latest releases of freeradius: - ping_check - ping_interval - num_pings_to_alive
Those are for backwards compatibility with pre-releases of 2.0. They should be removed. They are just different names for the status-server checks.
Excellent; I was wondering if I was somehow not "seeing" something as I went through the documentation.
- max_outstanding (I can't even find what this is for)
You can put a limit on the total number of "outstanding" packets sent to a home server. i.e. put it at 256, and if there are 256 packets sent without a response, the proxy will *not* use that home server again, until it gets at least one response.
This is a way to do load-limiting on home servers.
As it is, my *.work files are "stuck" (And I've googled for that, and found other list posts regarding that) which seems to indicate that the home servers aren't responding... except that even when my detail.work file is 'stuck' at 24k, and the detail file keeps growing, I'm still sending data to the other side. So something's working, but only sort of..
It's re-transmitting the same packet over and over. If you install 2.1.9, you can use "radmin" to see its progress in reading the detail file.
After some work getting 2.1.9, and v2.1.x from the git repository up and running, I had to go back to 2.1.7-7, that is patched (hopefully, anyway!) for the "zombie" problem, via the patch you sent me. The 2.1.9 and 2.1.10 versions would die unexpectedly, right around the time the "Info: ... ... adding new socket command file /var/run/radiusd/radiusd.sock " would scroll through the debug. I couldn't figure it out for the life of me, and strace didn't give me too much - it'd just segfault right around that time. It also did it on vanilla installs of 2.1.10, too - so I just gave it up. At any rate, "radmin" *does* exist for 2.1.7-7 (from the redhat source, which I patched with the patch you gave me), but it's complaining about permissions on the sock file (which appear to be fine, but perhaps selinux is killing it, I have to take a gander) - once I get that ironed out, I'll take great pleasure in using radmin and seeing what it sees.
I'm about to shoot an email to them to see if they can explain their 4 year old radius software, and perhaps maybe that's part of the problem.
Yup. They can upgrade to a (cough) real radius server. :)
Turns out, they were a bit stand-offish. They didn't like their radius servers being implicated in the mix. "It's working for 30+ clients, so we have no plans to upgrade". One thing I also noticed was that it it doesn't look like freeradius is giving it very many tries on a packet before marking the system down. At least, that's the way it appears. I don't know how to use wireshark filters enough to find unacked packets, so I have to do that before I'll be able to piece that together. It is also noteworthy that upon pingscanning their network, I found two IP addresses that are up - and I'm getting packet loss to them. Between 4 and 7 percent, which while not a ton, might be enough to cause a problem if I'm relaying thousands of packets an hour. Thanks for the help, Alan. I appreciate it. -- Adam
Oh, I must apologize - I didn't know the 'detail' portion of radmin didn't exist until 2.1.9. Perhaps I'll work on compiling and testing that over the weekend. Adam Bultman wrote:
Alan DeKok wrote:
Adam Bultman wrote:
How do I change that functionality? I'd *love* it if it didn't zombie their servers for no reason. No.. it marks the servers zombie for a reason: they're not responding. But it may be too aggressive.
When I do a radiusd -CXXX, I see options I don't see documented for the latest releases of freeradius: - ping_check - ping_interval - num_pings_to_alive Those are for backwards compatibility with pre-releases of 2.0. They should be removed. They are just different names for the status-server checks.
Excellent; I was wondering if I was somehow not "seeing" something as I went through the documentation.
- max_outstanding (I can't even find what this is for) You can put a limit on the total number of "outstanding" packets sent to a home server. i.e. put it at 256, and if there are 256 packets sent without a response, the proxy will *not* use that home server again, until it gets at least one response.
This is a way to do load-limiting on home servers.
As it is, my *.work files are "stuck" (And I've googled for that, and found other list posts regarding that) which seems to indicate that the home servers aren't responding... except that even when my detail.work file is 'stuck' at 24k, and the detail file keeps growing, I'm still sending data to the other side. So something's working, but only sort of.. It's re-transmitting the same packet over and over. If you install 2.1.9, you can use "radmin" to see its progress in reading the detail file.
After some work getting 2.1.9, and v2.1.x from the git repository up and running, I had to go back to 2.1.7-7, that is patched (hopefully, anyway!) for the "zombie" problem, via the patch you sent me. The 2.1.9 and 2.1.10 versions would die unexpectedly, right around the time the "Info: ... ... adding new socket command file /var/run/radiusd/radiusd.sock " would scroll through the debug. I couldn't figure it out for the life of me, and strace didn't give me too much - it'd just segfault right around that time. It also did it on vanilla installs of 2.1.10, too - so I just gave it up.
At any rate, "radmin" *does* exist for 2.1.7-7 (from the redhat source, which I patched with the patch you gave me), but it's complaining about permissions on the sock file (which appear to be fine, but perhaps selinux is killing it, I have to take a gander) - once I get that ironed out, I'll take great pleasure in using radmin and seeing what it sees.
I'm about to shoot an email to them to see if they can explain their 4 year old radius software, and perhaps maybe that's part of the problem. Yup. They can upgrade to a (cough) real radius server. :)
Turns out, they were a bit stand-offish. They didn't like their radius servers being implicated in the mix. "It's working for 30+ clients, so we have no plans to upgrade".
One thing I also noticed was that it it doesn't look like freeradius is giving it very many tries on a packet before marking the system down. At least, that's the way it appears. I don't know how to use wireshark filters enough to find unacked packets, so I have to do that before I'll be able to piece that together.
It is also noteworthy that upon pingscanning their network, I found two IP addresses that are up - and I'm getting packet loss to them. Between 4 and 7 percent, which while not a ton, might be enough to cause a problem if I'm relaying thousands of packets an hour.
Thanks for the help, Alan. I appreciate it.
-- Adam
Adam Bultman wrote:
After some work getting 2.1.9, and v2.1.x from the git repository up and running, I had to go back to 2.1.7-7, that is patched (hopefully, anyway!) for the "zombie" problem, via the patch you sent me. The 2.1.9 and 2.1.10 versions would die unexpectedly, right around the time the "Info: ... ... adding new socket command file /var/run/radiusd/radiusd.sock " would scroll through the debug. I couldn't figure it out for the life of me, and strace didn't give me too much - it'd just segfault right around that time.
Don't use strace to track down SEGVs. It won't help. See doc/bugs for instructions on tracking down SEGVs. Those instructions work.
Yup. They can upgrade to a (cough) real radius server. :)
Turns out, they were a bit stand-offish. They didn't like their radius servers being implicated in the mix. "It's working for 30+ clients, so we have no plans to upgrade".
Exactly... "no one else has noticed a problem, so we're not going to fix it." I guess they don't fix leaks in the roof of their house. When it doesn't rain, it doesn't leak. When it rains, it's too wet to fix the leak.
One thing I also noticed was that it it doesn't look like freeradius is giving it very many tries on a packet before marking the system down.
FreeRADIUS doesn't retry packets when proxying. The *NAS* retries packets. FreeRADIUS retransmits only when it receives a packet from the NAS.
At least, that's the way it appears. I don't know how to use wireshark filters enough to find unacked packets, so I have to do that before I'll be able to piece that together.
Why use wireshark? The server has a debug mode...
It is also noteworthy that upon pingscanning their network, I found two IP addresses that are up - and I'm getting packet loss to them. Between 4 and 7 percent, which while not a ton, might be enough to cause a problem if I'm relaying thousands of packets an hour.
Yup.
Thanks for the help, Alan. I appreciate it.
It's what I do. Alan DeKok.
OK, so some more updates: 1. I re-compiled freeradius v2.1.x from the git sources, using a different spec file (I'm building RPMs from the source) and it's working much better now. 2. I've tuned my detail file placer and reader to do individual files based on attributes and a timestamp I've noticed that: 1. I'm getting pseudo-duplicate packets from my NAS 2. The pseudo-duplicate packets are subtly different in that some items are missing - in my case, Calling-Station-Id. 3. The "other side" tends to "zombie" on me with packets that don't contain the Calling-Station-Id (Although it'll zombie on me 4. I'm attempting to control my detail file reader to ignore packets that don't have a Calling-Station-Id attribute, but I'm failing. First, in my detail file reader, I want to make it so that if certain attributes are in the packet, that I send it on it's way to the other servers. If they aren't there, I want to simply ignore that packet, and make it go away. So, I have this in my accounting {} section: if (Calling-Station-Id > 0 && Framed-IP-Address > 0 && User-Name =~ /mydomain/i) { update control { Proxy-To-Realm := "my_acct" } } I'll see things like this: Mon Jul 19 12:18:52 2010 : Info: ? Evaluating (Calling-Station-Id > 0 ) -> TRUE Mon Jul 19 12:18:52 2010 : Info: ? Evaluating (Framed-IP-Address > 0 ) -> TRUE Mon Jul 19 12:18:52 2010 : Info: ? Evaluating (User-Name =~ /mydomain/i) -> TRUE Mon Jul 19 12:18:52 2010 : Info: ++? if (Calling-Station-Id > 0 && Framed-IP-Address > 0 && User-Name =~ /mydomain/i) -> TRUE Mon Jul 19 12:18:52 2010 : Info: +++[control] returns noop Mon Jul 19 12:18:52 2010 : Info: ++- if (Calling-Station-Id > 0 && Framed-IP-Address > 0 && User-Name =~ /mydomain/i) returns noop Mon Jul 19 12:18:52 2010 : Info: ++[ok] returns ok But, even when there's a packet that does NOT match, it still is trying to send it to the home servers - directly after the part where it says FALSE, the server is then immediately sending it on it's way. I'm not sure if I need to have an else {} section to do 'noop', or 'reject', or 'handled', or what, and that's where I'm stuck. I've been googling, but I'm not sure what I should be looking for. I'm going to keep trying in the meantime, and see if stripping those packets out are what I need to do. The other option I have is to put the realm and calling station ID in the detail packet names, and then simply ignore those. (I was doing that simply for packets with calling station IDs, but that creates a huge number of files freeRADIUS won't touch, which I then have to clean up after the fact.) Adam Alan DeKok wrote:
Adam Bultman wrote:
After some work getting 2.1.9, and v2.1.x from the git repository up and running, I had to go back to 2.1.7-7, that is patched (hopefully, anyway!) for the "zombie" problem, via the patch you sent me. The 2.1.9 and 2.1.10 versions would die unexpectedly, right around the time the "Info: ... ... adding new socket command file /var/run/radiusd/radiusd.sock " would scroll through the debug. I couldn't figure it out for the life of me, and strace didn't give me too much - it'd just segfault right around that time.
Don't use strace to track down SEGVs. It won't help.
See doc/bugs for instructions on tracking down SEGVs. Those instructions work.
Yup. They can upgrade to a (cough) real radius server. :) Turns out, they were a bit stand-offish. They didn't like their radius servers being implicated in the mix. "It's working for 30+ clients, so we have no plans to upgrade".
Exactly... "no one else has noticed a problem, so we're not going to fix it."
I guess they don't fix leaks in the roof of their house. When it doesn't rain, it doesn't leak. When it rains, it's too wet to fix the leak.
One thing I also noticed was that it it doesn't look like freeradius is giving it very many tries on a packet before marking the system down.
FreeRADIUS doesn't retry packets when proxying. The *NAS* retries packets. FreeRADIUS retransmits only when it receives a packet from the NAS.
At least, that's the way it appears. I don't know how to use wireshark filters enough to find unacked packets, so I have to do that before I'll be able to piece that together.
Why use wireshark? The server has a debug mode...
It is also noteworthy that upon pingscanning their network, I found two IP addresses that are up - and I'm getting packet loss to them. Between 4 and 7 percent, which while not a ton, might be enough to cause a problem if I'm relaying thousands of packets an hour.
Yup.
Thanks for the help, Alan. I appreciate it.
It's what I do.
Alan DeKok.
-- Adam
Adam Bultman wrote:
OK, so some more updates: 1. I'm getting pseudo-duplicate packets from my NAS
2. The pseudo-duplicate packets are subtly different in that some items are missing - in my case, Calling-Station-Id.
Your NAS looks to be pretty broken. If it doesn't do retransmits, it doesn't really implement RADIUS. My $0.02 is to go fix the NAS. Ask the manufacturer why it doesn't implement RADIUS. Ask them to implement the RFC 5080 retransmission algorithm. It's not hard.
3. The "other side" tends to "zombie" on me with packets that don't contain the Calling-Station-Id
i.e. packet which aren't being retransmitted.
4. I'm attempting to control my detail file reader to ignore packets that don't have a Calling-Station-Id attribute, but I'm failing.
# in detail file reader "virtual server" accounting { if (Calling-Station-Id) { # update proxies, etc. } else { ok # tell the server the packet was dealt with OK } }
First, in my detail file reader, I want to make it so that if certain attributes are in the packet, that I send it on it's way to the other servers. If they aren't there, I want to simply ignore that packet, and make it go away.
You need the "else ok" bit. Otherwise, the detail file reader assumes that the packet *wasn't* handled, and retransmits it. Alan DeKok.
participants (2)
-
Adam Bultman -
Alan DeKok