issue with reaped processes timing out in rad_waitpid
Hi Found this https://www.marshut.net/kqyrtz/reaped-processes-still-timing-out-in-rad-wait... regarding the following error being seen in radius.log. We've just discovered that this is happening a lot here and wondered if anyone has had a chance to look at his proposed patch/ produced another one that rectified the problem. Rgds Alex ....I have hit upon a case where some ntlm_auth processes would return (and write the NT_KEY to the connecting pipe) but FR still complains that it failed and denies authentication (this is on 2.2.5). This manifests in the logs like the following: Tue Oct 28 11:10:15 2014 : Auth: Login incorrect (mschap: External script says NT_KEY: 4BCE6CA72058BA7EE500D1A68A8771C0): [tstRad9] (from client 155.98.204.47 port 0 cli 02-00-00-00-00-01 via TLS tunnel) Since this is actually the output for a valid and successful authentication, it appears that the exit code is the real issue. That exit code is either that of the process itself or 2 if rad_waitpid times out while waiting for the child. After adding some debugging statements and recompiling I found that there were cases where reap_children would reap a child process but the pid would not be found in thread_pool.waiters. This only happened when there were a significant number of auths per seconds and still not consistently. Some head scratching ensued and a colleague then suggested there may be a race condition between rad_fork (where it calls fr_hash_table_insert) and reap_children (where it calls fr_hash_table_finddata). .......
Alex Sharaz wrote:
regarding the following error being seen in radius.log. We've just discovered that this is happening a lot here and wondered if anyone has had a chance to look at his proposed patch/ produced another one that rectified the problem.
The patch has been added to all branches, and will be in the next release. Alan DeKok.
On 2014-10-31, at 7:17 , Alan DeKok <aland@deployingradius.com> wrote:
Alex Sharaz wrote:
regarding the following error being seen in radius.log. We've just discovered that this is happening a lot here and wondered if anyone has had a chance to look at his proposed patch/ produced another one that rectified the problem.
The patch has been added to all branches, and will be in the next release.
Alan DeKok.
In case anyone is tempted to use my initial patch, the new patch (the one that was accepted) is better than the original one. Don't use the one I sent. Recompile from the source if you need it. Regards, -- Louis Munro lmunro@inverse.ca :: www.inverse.ca +1.514.447.4918 x125 :: +1 (866) 353-6153 x125 Inverse inc. :: Leaders behind SOGo (www.sogo.nu) and PacketFence (www.packetfence.org)
So just to check, if I download the latest version of 2.x.x from git.freeradius.org as outined at freeradius.org/git it'll have your patch in it? Rgds Alex
On 2014-10-31, at 7:17 , Alan DeKok <aland@deployingradius.com> wrote:
Alex Sharaz wrote:
regarding the following error being seen in radius.log. We've just discovered that this is happening a lot here and wondered if anyone has had a chance to look at his proposed patch/ produced another one that rectified the problem.
The patch has been added to all branches, and will be in the next release.
Alan DeKok.
In case anyone is tempted to use my initial patch, the new patch (the one that was accepted) is better than the original one. Don't use the one I sent. Recompile from the source if you need it.
Regards, -- Louis Munro lmunro@inverse.ca :: www.inverse.ca +1.514.447.4918 x125 :: +1 (866) 353-6153 x125 Inverse inc. :: Leaders behind SOGo (www.sogo.nu) and PacketFence (www.packetfence.org) - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Alex Sharaz wrote:
So just to check, if I download the latest version of 2.x.x from git.freeradius.org as outined at freeradius.org/git it'll have your patch in it?
https://github.com/FreeRADIUS/freeradius-server/tree/v3.0.x Look at the right side of the screen. There's a button "download zip". Alan DeKok.
I just found that error in my logs myself. That's a new one I hadn't seen until you brought it up. We have a number of occurrences around our peak times now that I'm looking for it in the logs. - JohnD On 10/31/2014 04:54 AM, Alex Sharaz wrote:
Hi
Found this
https://www.marshut.net/kqyrtz/reaped-processes-still-timing-out-in-rad-wait...
regarding the following error being seen in radius.log. We've just discovered that this is happening a lot here and wondered if anyone has had a chance to look at his proposed patch/ produced another one that rectified the problem. Rgds Alex
....I have hit upon a case where some ntlm_auth processes would return (and write the NT_KEY to the connecting pipe) but FR still complains that it failed and denies authentication (this is on 2.2.5). This manifests in the logs like the following: Tue Oct 28 11:10:15 2014 : Auth: Login incorrect (mschap: External script says NT_KEY: 4BCE6CA72058BA7EE500D1A68A8771C0): [tstRad9] (from client 155.98.204.47 port 0 cli 02-00-00-00-00-01 via TLS tunnel) Since this is actually the output for a valid and successful authentication, it appears that the exit code is the real issue. That exit code is either that of the process itself or 2 if rad_waitpid times out while waiting for the child. After adding some debugging statements and recompiling I found that there were cases where reap_children would reap a child process but the pid would not be found in thread_pool.waiters. This only happened when there were a significant number of auths per seconds and still not consistently. Some head scratching ensued and a colleague then suggested there may be a race condition between rad_fork (where it calls fr_hash_table_insert) and reap_children (where it calls fr_hash_table_finddata).
.......
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
participants (4)
-
Alan DeKok -
Alex Sharaz -
John Douglass -
Louis Munro