I am encountering an infrequent problem where FreeRadius crashes about once a week on a fairly busy server. I have a primary and secondary authentication server and just a primary accounting server. Both the primary and secondary crash at the same time (well the secondary is about 30 seconds behind the primary). Thus I know that it is an authentication request causing the problem. However, using the authentication requests that occurred just prior to the crash I am unable to replicate the problem. I have core dumps and have tried to find useful information on the cause but have not found anything. The trace always shows: (gdb) where #0 0x280a94ab in pthread_testcancel () from /usr/lib/libpthread.so.2 #1 0x280a1e3c in pthread_mutexattr_init () from /usr/lib/ libpthread.so.2 #2 0x2808b450 in ?? () FreeRadius 1.1.2 on FreeBSD 6.1 using libpthread. I have tried using libthr but that crashed instantly on receipt of any request so I suspect that was not intended to work. I am beginning to suspect that the problem may lie in libpthread. Is there anything that can be retrieved from the core files that might help? There are always several threads active at the time of the crash.
Doug Hardie wrote:
FreeRadius 1.1.2 on FreeBSD 6.1 using libpthread.
Upgrade to 1.1.6. It has a lot of fixes that may help. It looks like it's crashing when starting a new child thread. That may be a pthread issue in the underlying libraries. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
On May 8, 2007, at 00:49, Alan DeKok wrote:
Doug Hardie wrote:
FreeRadius 1.1.2 on FreeBSD 6.1 using libpthread.
Upgrade to 1.1.6. It has a lot of fixes that may help.
It looks like it's crashing when starting a new child thread. That may be a pthread issue in the underlying libraries.
I upgraded the secondary server to 1.1.6. We will see what happens the next time the primary crashes. It will probably be about another week before then as they crashed early yesterday. I expect though that the problem is in libpthread.
On May 8, 2007, at 00:49, Alan DeKok wrote:
Doug Hardie wrote:
FreeRadius 1.1.2 on FreeBSD 6.1 using libpthread.
Upgrade to 1.1.6. It has a lot of fixes that may help.
It looks like it's crashing when starting a new child thread. That may be a pthread issue in the underlying libraries.
Well, both the primary and backup machines crashed again today at the same time. Its not the FreeRadius changes that will fix it. I will be upgrading the OS in a few days. There appear to be some changes to pthread library. Will watch to see what happens after that.
On May 8, 2007, at 00:49, Alan DeKok wrote:
Doug Hardie wrote:
FreeRadius 1.1.2 on FreeBSD 6.1 using libpthread.
Upgrade to 1.1.6. It has a lot of fixes that may help.
It looks like it's crashing when starting a new child thread. That may be a pthread issue in the underlying libraries.
Upgraded to the latest of everything. Same problem except that it only took about an hour before the first crash. Any ideas how to figure out what is going on? Or at least to find the request that is in process when the crash occurs?
Hi Doug and everyone, In message <444BFFDD-3494-451C-AB44-BFC74DD130E8@lafn.org>, Doug Hardie <bc979@lafn.org> writes
On May 8, 2007, at 00:49, Alan DeKok wrote:
Doug Hardie wrote:
FreeRadius 1.1.2 on FreeBSD 6.1 using libpthread.
Upgrade to 1.1.6. It has a lot of fixes that may help.
It looks like it's crashing when starting a new child thread. That may be a pthread issue in the underlying libraries.
Upgraded to the latest of everything.
Including the OS? FreeBSD is up to 6.2-RELEASE(-p4) now. FWIW, I didn't find going from 6.1 to 6.2 that painful - though there's always the risk of something going wrong.
Same problem except that it only took about an hour before the first crash. Any ideas how to figure out what is going on? Or at least to find the request that is in process when the crash occurs?
Can I ask - especially as I'm the maintainer of the FreeBSD FreeRADIUS port - are you using the port or not? I've put in a lot of effort to tidy up the port over the last few versions, and I believe it is now a good quality and easily maintainable port, despite it missing one or two features I'd like to add when I get the time. Most notably, I've done away with unnecessary or irrelevant patches (in fact, the only patch that remains is to do with FreeBSD 4.x and will probably be ripped out soon). As an aside, I hope to create a 2.0.0-pre1 port soon - though I've been away and very busy, and there's still a ports freeze in place with the ongoing work to switch FreeBSD to XOrg 7.2. I don't know whether testing with 2.0.0-pre1 is of interest to you, but I intend to try running 2.0.0-pre1 on my site as soon as possible for testing purposes. Hopefully someone can give you some debugging advice, and we can figure out whether this is a FreeRADIUS or FreeBSD problem. I doubt that the port itself is to blame, as apart from the aforementioned source patch (which just adds a single #include line to one file), and a some patching to the build system to change the install location of raddb, the port simply wraps the contents of the tarball from the FreeRADIUS project. Best wishes - hope you can get this one sorted out, David -- David Wood david@wood2.org.uk
On May 19, 2007, at 16:34, David Wood wrote:
Hi Doug and everyone,
In message <444BFFDD-3494-451C-AB44-BFC74DD130E8@lafn.org>, Doug Hardie <bc979@lafn.org> writes
On May 8, 2007, at 00:49, Alan DeKok wrote:
Doug Hardie wrote:
FreeRadius 1.1.2 on FreeBSD 6.1 using libpthread.
Upgrade to 1.1.6. It has a lot of fixes that may help.
It looks like it's crashing when starting a new child thread. That may be a pthread issue in the underlying libraries.
Upgraded to the latest of everything.
Including the OS? FreeBSD is up to 6.2-RELEASE(-p4) now. FWIW, I didn't find going from 6.1 to 6.2 that painful - though there's always the risk of something going wrong.
Same problem except that it only took about an hour before the first crash. Any ideas how to figure out what is going on? Or at least to find the request that is in process when the crash occurs?
Can I ask - especially as I'm the maintainer of the FreeBSD FreeRADIUS port - are you using the port or not? I've put in a lot of effort to tidy up the port over the last few versions, and I believe it is now a good quality and easily maintainable port, despite it missing one or two features I'd like to add when I get the time.
Most notably, I've done away with unnecessary or irrelevant patches (in fact, the only patch that remains is to do with FreeBSD 4.x and will probably be ripped out soon).
As an aside, I hope to create a 2.0.0-pre1 port soon - though I've been away and very busy, and there's still a ports freeze in place with the ongoing work to switch FreeBSD to XOrg 7.2. I don't know whether testing with 2.0.0-pre1 is of interest to you, but I intend to try running 2.0.0-pre1 on my site as soon as possible for testing purposes.
Hopefully someone can give you some debugging advice, and we can figure out whether this is a FreeRADIUS or FreeBSD problem. I doubt that the port itself is to blame, as apart from the aforementioned source patch (which just adds a single #include line to one file), and a some patching to the build system to change the install location of raddb, the port simply wraps the contents of the tarball from the FreeRADIUS project.
I am using the port as of about a week ago. One thing I just noticed. The following is in radiusd.conf: thread pool { start_servers = 5 max_servers = 32 min_spare_servers = 3 max_spare_servers = 10 max_requests_per_server = 0 } However, ps -H shows only 3 active threads. I would have expected more. But perhaps its idle right now. I think 8 threads is the most I have ever seen. The last crash was trying to start the 8th thread. Perhaps upping min_spare_servers above 8 would help.
On May 19, 2007, at 17:27, Doug Hardie wrote:
One thing I just noticed. The following is in radiusd.conf:
thread pool { start_servers = 5 max_servers = 32 min_spare_servers = 3 max_spare_servers = 10 max_requests_per_server = 0 }
However, ps -H shows only 3 active threads. I would have expected more. But perhaps its idle right now. I think 8 threads is the most I have ever seen. The last crash was trying to start the 8th thread. Perhaps upping min_spare_servers above 8 would help.
Nope. Just tried the following: thread pool { start_servers = 5 max_servers = 32 min_spare_servers = 10 max_spare_servers = 20 max_requests_per_server = 0 } and the number of threads after several hours is still 3.
On May 8, 2007, at 00:49, Alan DeKok wrote:
Doug Hardie wrote:
FreeRadius 1.1.2 on FreeBSD 6.1 using libpthread.
Upgrade to 1.1.6. It has a lot of fixes that may help.
It looks like it's crashing when starting a new child thread. That may be a pthread issue in the underlying libraries.
The saga continues. Digging around through the core dumps I noticed that often one of my modules was active in another thread and always at a fprintf statement. I wondered if perhaps FreeBSD's fprintf statment was not always thread safe so I removed all of them. Not the problem. Now its dying on a simple assignment statement. However, thats obvious when you see the arguments to the authorize function: Both zeros. I didn't think that was supposed to happen. None of the included modules check for that condition. Is this whats causing my problem or is it the result of the thread that is not able to get started properly? I suspect the latter since the prior stack is corrupt. I am tempted to put a check for that right at the beginning of the authorize function and just return if it happens. Good idea? I am completely unable to replicate this situation on my test system. I can run thousands of requests via multiple radclients without any problems. I can drive the test system to overload and other than responses slow down a bit, it just works properly. #0 0x2830a6e8 in ?? () from /usr/local/lib/rlm_lafn.so #1 0x2830b9c0 in lafn_authorize (instance=0x0, request=0x0) at rlm_lafn.c:543 Previous frame inner to this frame (corrupt stack?)
Doug Hardie wrote:
I am completely unable to replicate this situation on my test system. I can run thousands of requests via multiple radclients without any problems. I can drive the test system to overload and other than responses slow down a bit, it just works properly.
#0 0x2830a6e8 in ?? () from /usr/local/lib/rlm_lafn.so #1 0x2830b9c0 in lafn_authorize (instance=0x0, request=0x0) at rlm_lafn.c:543
Umm... if you're using modules you wrote yourself, my guess would be that the problem lies in those modules. You probably have access a pointer after it's freed, which corrupts memory. The standard server as shipped in 1.1.6 does *not* have this problem. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
On May 25, 2007, at 01:24, Alan Dekok wrote:
Doug Hardie wrote:
I am completely unable to replicate this situation on my test system. I can run thousands of requests via multiple radclients without any problems. I can drive the test system to overload and other than responses slow down a bit, it just works properly.
#0 0x2830a6e8 in ?? () from /usr/local/lib/rlm_lafn.so #1 0x2830b9c0 in lafn_authorize (instance=0x0, request=0x0) at rlm_lafn.c:543
Umm... if you're using modules you wrote yourself, my guess would be that the problem lies in those modules. You probably have access a pointer after it's freed, which corrupts memory.
The standard server as shipped in 1.1.6 does *not* have this problem.
Nope. All memory that is used is local. Nothing is retained. Only the authorize module is used. Nothing is dynamically allocated in the module.
Doug Hardie wrote:
Nope. All memory that is used is local. Nothing is retained. Only the authorize module is used. Nothing is dynamically allocated in the module.
Are you sure there are no buffer overruns in your module? Are you sure you're calling the FreeRADIUS API correctly? My question is because we've had a full source code scan in 1.1.6, and a number of bugs have fixed. The result is I am very skeptical of there being memory-related bugs in the server. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
On May 25, 2007, at 01:24, Alan Dekok wrote:
Doug Hardie wrote:
I am completely unable to replicate this situation on my test system. I can run thousands of requests via multiple radclients without any problems. I can drive the test system to overload and other than responses slow down a bit, it just works properly.
#0 0x2830a6e8 in ?? () from /usr/local/lib/rlm_lafn.so #1 0x2830b9c0 in lafn_authorize (instance=0x0, request=0x0) at rlm_lafn.c:543
Umm... if you're using modules you wrote yourself, my guess would be that the problem lies in those modules. You probably have access a pointer after it's freed, which corrupts memory.
The standard server as shipped in 1.1.6 does *not* have this problem.
Should have pointed out that this module ran for over a year with 1.1.2 and FreeBSD 5.3 without any problems. Never once had a core dump.
I think I may have found the cause of my crashes. One of the proxy servers or NASs is occasionally sending me an incorrectly formatted authentication request. I have not been able to capture the entire packet yet but I did manage to log part of the last one just as the crash occurred and the part that was successfully flushed out of the buffers before the seg fault is definitely corrupt. Because my secondary server only handles requests when the primary is down, I can set it to capture all the packets. However, I am going to have to wait till I can upgrade its OS. Its also our news server and upgrading that is always a large pain.
participants (5)
-
A.L.M.Buxey＠lboro.ac.uk -
Alan DeKok -
Alan Dekok
-
David Wood -
Doug Hardie