freeradius 2.1.8 dies Error: ASSERT FAILED event.c[1084]: home->ev != NULL
I recently upgraded our freeradius servers to 2.1.8 and over the past month it has died on one of the servers two times (spaced about two weeks apart I think). So fairly infrequently. A bit of background, We use this server predominantly to proxy requests. Every day for about 15 minutes, the two main home servers we proxy to stop responding (they are doing backups or maintenance during this time) so for those 15 minutes our clients (LNS/NAS) would be sending a very large number of accounting interim packets and some stop packets and would be resending these while the home servers are down. Some relevant proxy home server settings we currently use for the main home servers we proxy to: response_window = 14 zombie_period = 40 status_check = status-server check_interval = 30 num_answers_to_alive = 3 The times that freeradius has died has been near the end of the 15 minutes of the home servers downtime. The last time this happened, I noticed logs in the attached file. Ones that sound relevant as follows: Sun Mar 14 17:30:15 2010 : Proxy: Marking home server 10.0.1.48 port 1646 as zombie (it looks like it is dead). Sun Mar 14 17:30:16 2010 : Proxy: Marking home server 10.0.1.47 port 1646 as zombie (it looks like it is dead). Sun Mar 14 17:30:19 2010 : Proxy: Marking home server 10.0.1.47 port 1645 as zombie (it looks like it is dead). Sun Mar 14 17:30:19 2010 : Error: No response to status check 903535 for home server 10.0.1.48 port 1646 Sun Mar 14 17:30:20 2010 : Error: No response to status check 903536 for home server 10.0.1.47 port 1646 ... Sun Mar 14 17:30:32 2010 : Error: Internal sanity check failed for child state ... Fri Mar 19 17:30:54 2010 : Proxy: Failed to create a new socket for proxying requests. Fri Mar 19 17:30:54 2010 : Proxy: Failed to create a new socket for proxying requests. Fri Mar 19 17:30:54 2010 : Proxy: Failed to create a new socket for proxying requests. ... Fri Mar 19 17:30:56 2010 : Error: ASSERT FAILED event.c[1084]: home->ev != NULL That last one is where it dies I think. That last error seems a bit similar (but a bit different) to the following thread: http://www.mail-archive.com/freeradius-users@lists.freeradius.org/msg58052.h... ">Re: ASSERT FAILED event.c in 2.1.7
Alan DeKok Fri, 25 Sep 2009 01:19:40 -0700
Maja Wolniewicz wrote:
After the upgrade from 2.1.6 to 2.1.7 my two servers died 3-4 times daily with the following error:
Thu Sep 24 19:07:13 2009 : Error: Received conflicting packet from client AP-8 port 32777 - ID: 240 due to unfinished request 2396. Giving up on old request. Thu Sep 24 19:07:13 2009 : Error: ASSERT FAILED event.c[2682]: request->ev != NULL
I have to return to 2.1.6, which works smoothly.
The simplest thing to do in the short term is to delete the assertion.
Alan DeKok. "
That one was found to be a bug and was fixed - I don't know if my case is a bug though. Another thread that sounds useful for this is: http://www.mail-archive.com/freeradius-users@lists.freeradius.org/msg59985.h... "...
Until then, configuring status-checks && local detail files will definitely help. I would recommend doing that *anyways* for network stability.
Alan DeKok."
I don't currently use the robust proxy accounting that that thread suggests. I expect that would probably work around the issue of freeradius crashing in this case and I will give that a go. Just posting this to let you know that it _might_ be a bug and to ask for advice about whether you think this is a bug or not, and if I should follow up on that, or if you think it is just my configuration that needs some changes and what areas I should concentrate on if that is the case? Regards, Anthony
fab junkmail wrote:
I recently upgraded our freeradius servers to 2.1.8 and over the past month it has died on one of the servers two times (spaced about two weeks apart I think). So fairly infrequently.
OK.
A bit of background, We use this server predominantly to proxy requests. Every day for about 15 minutes, the two main home servers we proxy to stop responding (they are doing backups or maintenance during this time) so for those 15 minutes our clients (LNS/NAS) would be sending a very large number of accounting interim packets and some stop packets and would be resending these while the home servers are down.
You can configure the proxy to log accounting packets to disk when the home server is down. See raddb/sites-available/robust-proxy-accounting
Sun Mar 14 17:30:15 2010 : Proxy: Marking home server 10.0.1.48 port 1646 as zombie (it looks like it is dead). Sun Mar 14 17:30:16 2010 : Proxy: Marking home server 10.0.1.47 port 1646 as zombie (it looks like it is dead). Sun Mar 14 17:30:19 2010 : Proxy: Marking home server 10.0.1.47 port 1645 as zombie (it looks like it is dead). Sun Mar 14 17:30:19 2010 : Error: No response to status check 903535 for home server 10.0.1.48 port 1646 Sun Mar 14 17:30:20 2010 : Error: No response to status check 903536 for home server 10.0.1.47 port 1646 ... Sun Mar 14 17:30:32 2010 : Error: Internal sanity check failed for child state
Hmm... that's not good.
Fri Mar 19 17:30:54 2010 : Proxy: Failed to create a new socket for proxying requests.
Why is it running out of sockets? This shouldn't happen.
Fri Mar 19 17:30:54 2010 : Proxy: Failed to create a new socket for proxying requests. Fri Mar 19 17:30:54 2010 : Proxy: Failed to create a new socket for proxying requests. ... Fri Mar 19 17:30:56 2010 : Error: ASSERT FAILED event.c[1084]: home->ev != NULL
Well... after all of the previous errors, it's not surprising that something *worse* eventually goes wrong. It's like driving your car for 45 minutes after the tires are flat: not a good idea.
That last one is where it dies I think.
Yes.
That one was found to be a bug and was fixed - I don't know if my case is a bug though.
It's a bug, but the other problems you're seeing should be fixed, too.
I don't currently use the robust proxy accounting that that thread suggests. I expect that would probably work around the issue of freeradius crashing in this case and I will give that a go.
Yes.
Just posting this to let you know that it _might_ be a bug and to ask for advice about whether you think this is a bug or not, and if I should follow up on that, or if you think it is just my configuration that needs some changes and what areas I should concentrate on if that is the case?
You have a NAS which is sending large amounts of traffic to a proxy when the home server is down. The proxy isn't configured to do anything useful with the packets. This is a bug in the *architecture*. Alan DeKok.
Hi Alan, Thanks for your response. Alan DeKok wrote:
You can configure the proxy to log accounting packets to disk when the home server is down. See raddb/sites-available/robust-proxy-accounting
Ok I will definitely do this then.
Fri Mar 19 17:30:54 2010 : Proxy: Failed to create a new socket for proxying requests.
Why is it running out of sockets? This shouldn't happen.
Not sure but there is a _lot_ of attempted proxying going on - maybe it just went over the system limits like open file limits or something? In any case it probably won't be a problem when I implement the robust-proxy-accounting.
You have a NAS which is sending large amounts of traffic to a proxy when the home server is down. The proxy isn't configured to do anything useful with the packets. This is a bug in the *architecture*.
Understood. Thanks for your help Alan. Regards, Anthony
fab junkmail wrote:
Why is it running out of sockets? This shouldn't happen.
Not sure but there is a _lot_ of attempted proxying going on - maybe it just went over the system limits like open file limits or something? In any case it probably won't be a problem when I implement the robust-proxy-accounting.
Likely, yes. If the server is overloaded and unable to proxy packets... who knows what can happen. The *intent* is to have it still work, but it's a poorly tested code path. Alan DeKok.
participants (2)
-
Alan DeKok -
fab junkmail