FR 2.1.9 Frequent SegFault, didn't happen with FR 2.1.8
Hi Alan, All, Since upgrading to 2.1.9, FR is segfaulting frequently (every 20 minutes with load, every ~8 hours with less load). Attached -X at startup, and the last 100 lines before segfault. If someone can explain how to drive GDB (or any other method to track this down), I'm happy to try it. Thanks, James -- James J J Hooper Network Specialist Information Services University of Bristol http://www.wireless.bristol.ac.uk http://www.jamesjj.net --
On 09/06/2010 17:56, James J J Hooper wrote:
Hi Alan, All,
Since upgrading to 2.1.9, FR is segfaulting frequently (every 20 minutes with load, every ~8 hours with less load).
Attached -X at startup, and the last 100 lines before segfault.
If someone can explain how to drive GDB (or any other method to track this down), I'm happy to try it.
I found the manual (http://freeradius.org/radiusd/doc/bugs) so am RTFMing... I'll follow up with results if I find anything. -James
On 09/06/2010 21:17, James J J Hooper wrote:
On 09/06/2010 17:56, James J J Hooper wrote:
Hi Alan, All,
Since upgrading to 2.1.9, FR is segfaulting frequently (every 20 minutes with load, every ~8 hours with less load).
Attached -X at startup, and the last 100 lines before segfault.
If someone can explain how to drive GDB (or any other method to track this down), I'm happy to try it.
I found the manual (http://freeradius.org/radiusd/doc/bugs) so am RTFMing...
I'll follow up with results if I find anything.
OK - GDB log attached. This is from git branch v2.1.x, up to and including 0e9ae1698ba55b16b149 (Cleaned up debug output to be readable - about 7 hours ago), but with c703fd595cb86f51e309 (Install cryptpasswd as "radcrypt") reverted as it wouldn't 'make install' with this [see note below]. On linux, 2.6.9-89.0.23.ELsmp, CentOS release 4.8. -James -- James J J Hooper Network Specialist Information Services University of Bristol http://www.wireless.bristol.ac.uk http://www.jamesjj.net -- Note re: cryptpasswd: /usr/local/dnsnode/src/radiusd/git-20100609/freeradius-server/install-sh -c -m 755 cryptpassword /usr/local/bin/radcrypt install: cryptpassword does not exist gmake[2]: *** [install] Error 1
On Wed, Jun 09, 2010 at 10:00:14PM +0100, James J J Hooper wrote:
OK - GDB log attached.
Program received signal SIGSEGV, Segmentation fault. [Switching to Thread -1208649024 (LWP 2425)] 0x08067c64 in received_proxy_response (packet=0x8430a20) at event.c:3075 3075 } else if ((request->packet->code != request->proxy->code) && * 1 Thread -1208649024 (LWP 2425) 0x08067c64 in received_proxy_response (packet=0x8430a20) at event.c:3075
Thread 1 (Thread -1208649024 (LWP 2425)): #0 0x08067c64 in received_proxy_response (packet=0x8430a20) at event.c:3075
That code has indeed changed from 2.1.8 to 2.1.9, and it may be a simple bug - there is no existence check for request->packet before its dereference in the new CoA-related condition. There is an existence check for it below, but by then it's too late. It looks like that branching could do with some more reordering. As a temporary workaround, you can probably compile with -D WITHOUT_COA so that you lose that bit, assuming of course you don't use that feature :) -- 2. That which causes joy or happiness.
Josip Rodin wrote:
That code has indeed changed from 2.1.8 to 2.1.9, and it may be a simple bug - there is no existence check for request->packet before its dereference in the new CoA-related condition. There is an existence check for it below, but by then it's too late. It looks like that branching could do with some more reordering.
Yes. The code to fix a CoA bug caused another one. <sigh> The server is quickly getting to the point where it's doing so much that any small change can have large implications. The solution is to start re-organizing the code in the server core, and simplifying it. When 2.0 cam out, the code was much simpler than 1.x for the same functionality. Three years later, added functionality makes the code more complicated again. Alan DeKok.
On Thu, Jun 10, 2010 at 10:12:13AM +0200, Alan DeKok wrote:
Josip Rodin wrote:
That code has indeed changed from 2.1.8 to 2.1.9, and it may be a simple bug - there is no existence check for request->packet before its dereference in the new CoA-related condition. There is an existence check for it below, but by then it's too late. It looks like that branching could do with some more reordering.
Yes. The code to fix a CoA bug caused another one. <sigh>
The server is quickly getting to the point where it's doing so much that any small change can have large implications.
The solution is to start re-organizing the code in the server core, and simplifying it. When 2.0 cam out, the code was much simpler than 1.x for the same functionality. Three years later, added functionality makes the code more complicated again.
The Change of Authorization code hasn't really found its place, it looks a bit like a hack. Maybe it's the WITH_COA defines that make it look so. But more generally, after reading through the proxy code, I see it as a something of an unadjusted citizen :) yet CoA builds upon it and just ends up creating a bigger mess. The server started as a simple server that just responded to outside clients, whereas these two functions really want it to behave more like a client. I know that's a big readjustment :) -- 2. That which causes joy or happiness.
Josip Rodin wrote:
The Change of Authorization code hasn't really found its place, it looks a bit like a hack.
It's a hack.
Maybe it's the WITH_COA defines that make it look so. But more generally, after reading through the proxy code, I see it as a something of an unadjusted citizen :) yet CoA builds upon it and just ends up creating a bigger mess.
Yup.
The server started as a simple server that just responded to outside clients, whereas these two functions really want it to behave more like a client. I know that's a big readjustment :)
The solution for 2.2.0 is some major re-work of packet handling. Right now, all of the packets get processed by common code. This should be moved to per-type handling, with function callbacks. It increases the size of the REQUEST data structure (again). But it means that the "core" can just call a per-type handler. That handler can be set or reset, depending on what's going on. That's some of what I was trying to do in the git "master" branch, but that approach ended up being a dead end. Alan DeKok.
James J J Hooper wrote:
OK - GDB log attached. This is from git branch v2.1.x, up to and including 0e9ae1698ba55b16b149 (Cleaned up debug output to be readable - about 7 hours ago), but with c703fd595cb86f51e309 (Install cryptpasswd as "radcrypt") reverted as it wouldn't 'make install' with this [see note below].
OK. I fixed both problems. Thanks for tracking it down, it made the fix much simpler. Do a 'git pull' for the v2.1.x branch, and re-build. It should now be OK. Alan DeKok.
--On Thursday, June 10, 2010 10:10:05 +0200 Alan DeKok <aland@deployingradius.com> wrote:
James J J Hooper wrote:
OK - GDB log attached. This is from git branch v2.1.x, up to and including 0e9ae1698ba55b16b149 (Cleaned up debug output to be readable - about 7 hours ago), but with c703fd595cb86f51e309 (Install cryptpasswd as "radcrypt") reverted as it wouldn't 'make install' with this [see note below].
OK. I fixed both problems. Thanks for tracking it down, it made the fix much simpler.
Do a 'git pull' for the v2.1.x branch, and re-build. It should now be OK.
Hi Alan, Thanks for the swift fix - It's much happier now. -James -- James J J Hooper Network Specialist Information Services University of Bristol http://www.wireless.bristol.ac.uk http://www.jamesjj.net --
Hi,
OK. I fixed both problems. Thanks for tracking it down, it made the fix much simpler.
Do a 'git pull' for the v2.1.x branch, and re-build. It should now be OK.
hmm, this is interesting...James, do you use COA at all? we dont but this code is still in there and we havent had such a crash. could this be an issue that gets masked by a newer version of GCC (ours are generally CentOS 5.5 boxes...) - we have around 2,200 simultaneous users using 802.1X during the working day currently so show stopping bugs generally get seen alan
On 10/06/2010 22:20, Alan Buxey wrote:
Hi,
OK. I fixed both problems. Thanks for tracking it down, it made the fix much simpler.
Do a 'git pull' for the v2.1.x branch, and re-build. It should now be OK.
hmm, this is interesting...James, do you use COA at all? we dont but this code is still in there and we havent had such a crash. could this be an issue that gets masked by a newer version of GCC (ours are generally CentOS 5.5 boxes...) - we have around 2,200 simultaneous users using 802.1X during the working day currently so show stopping bugs generally get seen
No - we don't use COA on these boxes [yet]. How many of your users are home, and how many visiting (or do you do any other proxying)? - Proxy replies seem to have been the trigger for that code path bug. We seem to have a steady hundred or so users that get proxied up to the ORPS: http://www.wireless.bris.ac.uk/gfx/random/eduroamvisitors.png -James
On 10/06/2010 22:42, James J J Hooper wrote:
On 10/06/2010 22:20, Alan Buxey wrote:
Hi,
OK. I fixed both problems. Thanks for tracking it down, it made the fix much simpler.
Do a 'git pull' for the v2.1.x branch, and re-build. It should now be OK.
hmm, this is interesting...James, do you use COA at all? we dont but this code is still in there and we havent had such a crash. could this be an issue that gets masked by a newer version of GCC (ours are generally CentOS 5.5 boxes...) - we have around 2,200 simultaneous users using 802.1X during the working day currently so show stopping bugs generally get seen
No - we don't use COA on these boxes [yet]. How many of your users are home, and how many visiting (or do you do any other proxying)? - Proxy replies seem to have been the trigger for that code path bug.
We seem to have a steady hundred or so users that get proxied up to the ORPS: http://www.wireless.bris.ac.uk/gfx/random/eduroamvisitors.png
ORPS <=> NRPS (brain error)
James J J Hooper wrote:
Since upgrading to 2.1.9, FR is segfaulting frequently (every 20 minutes with load, every ~8 hours with less load).
Ouch.
Attached -X at startup, and the last 100 lines before segfault.
If someone can explain how to drive GDB (or any other method to track this down), I'm happy to try it.
doc/bugs is a good start. Alan DeKok.
participants (4)
-
Alan Buxey -
Alan DeKok -
James J J Hooper -
Josip Rodin