1.1.4 stops responding to requests
Just a heads up: In our production environment here at Brooklyn Law School, 1.1.4 stops responding to radius requests after a few hours. We have used freeradius for some time (and love it) to authenticate wireless users to our ldap servers. We use it with two different wireless systems, trapeeze and colubris, which each do 802.1x authentication to freeradius servers using PEAP and mschapv2. We run it on three different servers. When vista came along, we had to upgrade from 1.1.4 to 1.0.5. We run Fedora Core 5 and Fedora Core 6 on these servers. Because there is no 1.1.4 rpm out yet for Core 5 or Core 6, we built freeradius from source. Since we are in a production environment, we were careful to use non-overlapping directory structures when we compiled and installed 1.1.4 so that if there was a problem, we could quickly go back to the rpm. 1.1.4 will run for a few hours and then either stop responding to requests or die. There is no seg fault warning in any log file. If I restart radius, it then begins answering again. Since it is a production environment in which 300-500 users are connected at any given time, we were unable to simply turn on debugging and look for problems. Once we realized the problem we had to quickly revert to 1.0.5 for now and make our relatively few Vista users for a little longer. I am going to try running it in debug mode over a weekend in a particular subset of the school's wireless network where not many users would be affected by a crash and see if I can collect any more information. I will do it on a system that never had any earlier version of freeradius installed on it, just to be safe. In the meantime, any advice would be appreciated. Thanks, Phil Allred
Hi,
1.1.4 will run for a few hours and then either stop responding to requests or die. There is no seg fault warning in any log file. If I restart radius, it then begins answering again. Since it is a production environment in which 300-500 users are connected at any given time, we were unable to simply turn on debugging and look for problems. Once we realized the problem we had to quickly revert to 1.0.5 for now and make our relatively few Vista users for a little longer.
Interesting... I've been seeing exactly this happening on our own system. In our case, I could track it down to the fact that it stopped responding shortly after accounting packets came in. Turning off accounting (already on NAS level) deterministically solved the problem for me, so I suspect the problem to be somewhere near there. I never followed this trace, because accounting is optional at our site (free wireless) and it was our prod environment, I didn't want to mess around without a good reason. So accounting is just off at the moment.
I am going to try running it in debug mode over a weekend in a particular subset of the school's wireless network where not many users would be affected by a crash and see if I can collect any more information. I will do it on a system that never had any earlier version of freeradius installed on it, just to be safe. In the meantime, any advice would be appreciated.
Is it an option to not do accounting? Or maybe queue the acct in files rather than a proper backend (for me, the issue happened in combination with mysql). I never tried if the hangs occur also when logging to a file. Greetings, Stefan Winter -- Stefan WINTER Fondation RESTENA - Réseau Téléinformatique de l'Education Nationale et de la Recherche - Ingénieur de recherche 6, rue Richard Coudenhove-Kalergi L-1359 Luxembourg
Hello, Same with me, but acct is necessary. The solution was radsqlrelay. Regards, Guilherme On 2/13/07, Stefan Winter <stefan.winter@restena.lu> wrote:
Hi,
1.1.4 will run for a few hours and then either stop responding to requests or die. There is no seg fault warning in any log file. If I restart radius, it then begins answering again. Since it is a production environment in which 300-500 users are connected at any given time, we were unable to simply turn on debugging and look for problems. Once we realized the problem we had to quickly revert to 1.0.5 for now and make our relatively few Vista users for a little longer.
Interesting... I've been seeing exactly this happening on our own system. In our case, I could track it down to the fact that it stopped responding shortly after accounting packets came in. Turning off accounting (already on NAS level) deterministically solved the problem for me, so I suspect the problem to be somewhere near there. I never followed this trace, because accounting is optional at our site (free wireless) and it was our prod environment, I didn't want to mess around without a good reason. So accounting is just off at the moment.
I am going to try running it in debug mode over a weekend in a particular subset of the school's wireless network where not many users would be affected by a crash and see if I can collect any more information. I will do it on a system that never had any earlier version of freeradius installed on it, just to be safe. In the meantime, any advice would be appreciated.
Is it an option to not do accounting? Or maybe queue the acct in files rather than a proper backend (for me, the issue happened in combination with mysql). I never tried if the hangs occur also when logging to a file.
Greetings,
Stefan Winter
-- Stefan WINTER
Fondation RESTENA - Réseau Téléinformatique de l'Education Nationale et de la Recherche - Ingénieur de recherche
6, rue Richard Coudenhove-Kalergi L-1359 Luxembourg
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Interesting. I don't think it is needed here. I'll check with our wireless vendor, then give it a shot. Thanks very much! Phil Guilherme Franco wrote:
Hello,
Same with me, but acct is necessary.
The solution was radsqlrelay.
Regards,
Guilherme
On 2/13/07, *Stefan Winter* <stefan.winter@restena.lu <mailto:stefan.winter@restena.lu>> wrote:
Hi,
> 1.1.4 will run for a few hours and then either stop responding to > requests or die. There is no seg fault warning in any log file. If I > restart radius, it then begins answering again. Since it is a > production environment in which 300-500 users are connected at any given > time, we were unable to simply turn on debugging and look for problems. > Once we realized the problem we had to quickly revert to 1.0.5 for now > and make our relatively few Vista users for a little longer.
Interesting... I've been seeing exactly this happening on our own system. In our case, I could track it down to the fact that it stopped responding shortly after accounting packets came in. Turning off accounting (already on NAS level) deterministically solved the problem for me, so I suspect the problem to be somewhere near there. I never followed this trace, because accounting is optional at our site (free wireless) and it was our prod environment, I didn't want to mess around without a good reason. So accounting is just off at the moment.
> I am going to try running it in debug mode over a weekend in a > particular subset of the school's wireless network where not many users > would be affected by a crash and see if I can collect any more > information. I will do it on a system that never had any earlier > version of freeradius installed on it, just to be safe. In the > meantime, any advice would be appreciated.
Is it an option to not do accounting? Or maybe queue the acct in files rather than a proper backend (for me, the issue happened in combination with mysql). I never tried if the hangs occur also when logging to a file.
Greetings,
Stefan Winter
-- Stefan WINTER
Fondation RESTENA - Réseau Téléinformatique de l'Education Nationale et de la Recherche - Ingénieur de recherche
6, rue Richard Coudenhove-Kalergi L-1359 Luxembourg
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html <http://www.freeradius.org/list/users.html>
------------------------------------------------------------------------
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
The fix for this is in the CVS HEAD and probably should be backported to the latest release branch. There was a race condition in the code where the server could clean up an accounting request before a thread actually got to it to process it. --Mike On Feb 13, 2007, at 1:36 PM, Phil Allred wrote:
Interesting. I don't think it is needed here. I'll check with our wireless vendor, then give it a shot.
Thanks very much!
Phil
Guilherme Franco wrote:
Hello,
Same with me, but acct is necessary.
The solution was radsqlrelay.
Regards,
Guilherme
On 2/13/07, *Stefan Winter* <stefan.winter@restena.lu <mailto:stefan.winter@restena.lu>> wrote:
Hi,
1.1.4 will run for a few hours and then either stop responding to requests or die. There is no seg fault warning in any log file. If I restart radius, it then begins answering again. Since it is a production environment in which 300-500 users are connected at any given time, we were unable to simply turn on debugging and look for problems. Once we realized the problem we had to quickly revert to 1.0.5 for now and make our relatively few Vista users for a little longer.
Interesting... I've been seeing exactly this happening on our own system. In our case, I could track it down to the fact that it stopped responding shortly after accounting packets came in. Turning off accounting (already on NAS level) deterministically solved the problem for me, so I suspect the problem to be somewhere near there. I never followed this trace, because accounting is optional at our site (free wireless) and it was our prod environment, I didn't want to mess around without a good reason. So accounting is just off at the moment.
I am going to try running it in debug mode over a weekend in a particular subset of the school's wireless network where not many users would be affected by a crash and see if I can collect any more information. I will do it on a system that never had any earlier version of freeradius installed on it, just to be safe. In the meantime, any advice would be appreciated.
Is it an option to not do accounting? Or maybe queue the acct in files rather than a proper backend (for me, the issue happened in combination with mysql). I never tried if the hangs occur also when logging to a file.
Greetings,
Stefan Winter
-- Stefan WINTER
Fondation RESTENA - Réseau Téléinformatique de l'Education Nationale et de la Recherche - Ingénieur de recherche
6, rue Richard Coudenhove-Kalergi L-1359 Luxembourg
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html <http://www.freeradius.org/list/users.html>
--------------------------------------------------------------------- ---
- List info/subscribe/unsubscribe? See http://www.freeradius.org/ list/users.html
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/ users.html
Hi,
The fix for this is in the CVS HEAD and probably should be backported to the latest release branch. There was a race condition in the code where the server could clean up an accounting request before a thread actually got to it to process it.
yes, that definitely deserves a backport IMO. Up to now, accounting is a dangerous endeavour in the 1.1 branch :-( Greetings, Stefan Winter -- Stefan WINTER RESTENA Foundation - Réseau Téléinformatique de l'Education Nationale et de la Recherche R&D Engineer 6, rue Richard Coudenhove-Kalergi L-1359 Luxembourg email: stefan.winter@restena.lu Tel.: +352 424409-1 http://www.restena.lu Fax: +352 422473
Michael Griego wrote:
The fix for this is in the CVS HEAD and probably should be backported to the latest release branch. There was a race condition in the code where the server could clean up an accounting request before a thread actually got to it to process it.
Remind me again where that was? In the thread queue, perhaps? Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Revision 1.79 to src/main/request_list.c --Mike On Feb 14, 2007, at 3:46 AM, Alan DeKok wrote:
Michael Griego wrote:
The fix for this is in the CVS HEAD and probably should be backported to the latest release branch. There was a race condition in the code where the server could clean up an accounting request before a thread actually got to it to process it.
Remind me again where that was? In the thread queue, perhaps?
Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/ users.html
Michael Griego wrote:
Revision 1.79 to src/main/request_list.c
But the code in branch_1_1 looks like the revision 1.79, not 1.78. So I'm unclear as to why it's an issue. Maybe something else? Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Hmm... hadn't looked at the actual code. I just looked to see if there were any log entries after mine for the branch, and I didn't see any. Interesting. I guess that's not it. --Mike On Feb 17, 2007, at 2:24 AM, Alan DeKok wrote:
Michael Griego wrote:
Revision 1.79 to src/main/request_list.c
But the code in branch_1_1 looks like the revision 1.79, not 1.78. So I'm unclear as to why it's an issue.
Maybe something else?
Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/ users.html
participants (5)
-
Alan DeKok -
Guilherme Franco -
Michael Griego -
Phil Allred -
Stefan Winter