Zombie accounting processes
Hello! I've been having a rather annoying problem with FreeRADIUS, including version 1.1.0. I've set up authentication and accounting via a Perl script. -- George-Cristian Bîrzan Network Engineer _______________________________________ RCS & RDS Constanta Tel.: +40341.400.401 / +40341.400.402 Fax: +40341.400.450 http://www.rcs-rds.ro _______________________________________ Privileged/Confidential Information may be contained in this message. If you are not the addressee indicated in this message (or responsible for delivery of the message to such person), you may not copy or deliver this message to anyone. In such a case, you should destroy this message and kindly notify the sender by reply e-mail.
On Mon, 2006-02-13 at 13:57 +0200, George-Cristian Bîrzan wrote:
Hello!
I've been having a rather annoying problem with FreeRADIUS, including version 1.1.0.
I've set up authentication and accounting via a Perl script.
Er. Ewps. Let's try again. :-) In users: DEFAULT Auth-Type = Accept Exec-Program-Wait = "/usr/local/etc/raddb/auth-ng.pl %u %{request:User-Password} %n %i", Fall-Through = 0 In acct_users: DEFAULT Acct-Status-Type = Stop Exec-Program = "/usr/local/etc/raddb/acct-ng.pl STOP %u %{Acct-Input-Octets} %{Acct-Output-Octets} %{Acct-Input-Packets} %{Acct-Output-Packets}" DEFAULT Acct-Status-Type = Start Exec-Program = "/usr/local/etc/raddb/acct-ng.pl START %u %i %{Framed-IP-Address} %n" If I don't compile FreeRADIUS with --with-threads, it all works fine, but it only will authenticate one user at a time. If I use threads, accounting stop processes remain as zombie processes. After a day or so, the server would die, not being able to create more processes. So, I've started restarting the RADIUS server hourly, but that's quite an ugly hack.
From a strace of a process that remains hung:
[pid 22052] exit_group(0) = ? Process 22052 detached --- SIGCHLD (Child exited) @ 0 (0) --- --- SIGRTMIN (Unknown signal 32) @ 0 (0) --- <... rt_sigsuspend resumed> ) = -1 EINTR (Interrupted system call) sigreturn() = ? (mask now [HUP INT QUIT TERM RTMIN]) waitpid(22058, NULL, WNOHANG) = -1 ECHILD (No child processes) 22058 exists, and is another process that hung. $ ps u 22058 22052 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 22052 0.0 0.0 0 0 ? Z 14:08 0:00 [acct-ng.pl] <defunct> root 22058 0.0 0.0 0 0 ? Z 14:08 0:00 [acct-ng.pl] <defunct> I'm trying to look at the source for hints, but I figured I should try asking... if anyone has any ideas, I'd greatly appreciate it. -- George-Cristian Bîrzan Network Engineer _______________________________________ RCS & RDS Constanta Tel.: +40341.400.401 / +40341.400.402 Fax: +40341.400.450 http://www.rcs-rds.ro _______________________________________ Privileged/Confidential Information may be contained in this message. If you are not the addressee indicated in this message (or responsible for delivery of the message to such person), you may not copy or deliver this message to anyone. In such a case, you should destroy this message and kindly notify the sender by reply e-mail.
George-Cristian Bîrzan wrote:
On Mon, 2006-02-13 at 13:57 +0200, George-Cristian Bîrzan wrote:
Hello!
I've been having a rather annoying problem with FreeRADIUS, including version 1.1.0.
What OS and kernel version is the server running?
From a strace of a process that remains hung:
[pid 22052] exit_group(0) = ? Process 22052 detached --- SIGCHLD (Child exited) @ 0 (0) --- --- SIGRTMIN (Unknown signal 32) @ 0 (0) --- <... rt_sigsuspend resumed> ) = -1 EINTR (Interrupted system call) sigreturn() = ? (mask now [HUP INT QUIT TERM RTMIN]) waitpid(22058, NULL, WNOHANG) = -1 ECHILD (No child processes)
Hmm. What does "ps -ejf" (on Linux - translate for closest equivalent on other systems) say for these?
22058 exists, and is another process that hung.
$ ps u 22058 22052 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 22052 0.0 0.0 0 0 ? Z 14:08 0:00 [acct-ng.pl] <defunct> root 22058 0.0 0.0 0 0 ? Z 14:08 0:00 [acct-ng.pl] <defunct>
I'm trying to look at the source for hints, but I figured I should try asking... if anyone has any ideas, I'd greatly appreciate it.
On Mon, 2006-02-13 at 12:56 +0000, Phil Mayers wrote:
George-Cristian Bîrzan wrote:
On Mon, 2006-02-13 at 13:57 +0200, George-Cristian Bîrzan wrote:
Hello!
I've been having a rather annoying problem with FreeRADIUS, including version 1.1.0.
What OS and kernel version is the server running?
It's a Slackware 10.1 (I think, could be 10) with a 2.6.14.3 kernel. Same happened with 2.4.29 and 2.6.12.
From a strace of a process that remains hung:
[pid 22052] exit_group(0) = ? Process 22052 detached --- SIGCHLD (Child exited) @ 0 (0) --- --- SIGRTMIN (Unknown signal 32) @ 0 (0) --- <... rt_sigsuspend resumed> ) = -1 EINTR (Interrupted system call) sigreturn() = ? (mask now [HUP INT QUIT TERM RTMIN]) waitpid(22058, NULL, WNOHANG) = -1 ECHILD (No child processes)
Hmm. What does "ps -ejf" (on Linux - translate for closest equivalent on other systems) say for these?
root 25065 24736 23880 23880 0 15:11 ? 00:00:00 [acct-ng.pl] <defunct> root 25073 23885 23880 23880 0 15:11 ? 00:00:00 [acct-ng.pl] <defunct> -- George-Cristian Bîrzan Network Engineer _______________________________________ RCS & RDS Constanta Tel.: +40341.400.401 / +40341.400.402 Fax: +40341.400.450 http://www.rcs-rds.ro _______________________________________ Privileged/Confidential Information may be contained in this message. If you are not the addressee indicated in this message (or responsible for delivery of the message to such person), you may not copy or deliver this message to anyone. In such a case, you should destroy this message and kindly notify the sender by reply e-mail.
George-Cristian Bîrzan wrote:
On Mon, 2006-02-13 at 12:56 +0000, Phil Mayers wrote:
George-Cristian Bîrzan wrote:
On Mon, 2006-02-13 at 13:57 +0200, George-Cristian Bîrzan wrote:
Hello!
I've been having a rather annoying problem with FreeRADIUS, including version 1.1.0. What OS and kernel version is the server running?
It's a Slackware 10.1 (I think, could be 10) with a 2.6.14.3 kernel. Same happened with 2.4.29 and 2.6.12.
Ok, so the FreeRadius executable should have only one PID since it's running the new threading, unless you've got an old libc?
Hmm. What does "ps -ejf" (on Linux - translate for closest equivalent on other systems) say for these?
root 25065 24736 23880 23880 0 15:11 ? 00:00:00 [acct-ng.pl] <defunct> root 25073 23885 23880 23880 0 15:11 ? 00:00:00 [acct-ng.pl] <defunct>
...so why do these have different parent PIDs (2nd column)? Could "acct-ng.pl" be itself forking? What process are those PIDs?
On Mon, 2006-02-13 at 14:36 +0000, Phil Mayers wrote:
It's a Slackware 10.1 (I think, could be 10) with a 2.6.14.3 kernel. Same happened with 2.4.29 and 2.6.12.
Ok, so the FreeRadius executable should have only one PID since it's running the new threading, unless you've got an old libc?
Hm. 2.3.4, not that old, but: root@radius:~# ps u $(pgrep radiusd) USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 5258 0.0 0.2 16908 2140 ? Ss 18:47 0:00 radiusd root 5259 0.0 0.2 16908 2140 ? S 18:47 0:00 radiusd root 5260 0.0 0.2 16908 2140 ? S 18:47 0:00 radiusd root 5261 0.0 0.2 16908 2140 ? S 18:47 0:00 radiusd root 5262 0.0 0.2 16908 2140 ? S 18:47 0:00 radiusd root 5263 0.0 0.2 16908 2140 ? S 18:47 0:00 radiusd root 5264 0.0 0.2 16908 2140 ? S 18:47 0:00 radiusd root 5978 0.0 0.2 16908 2140 ? S 18:57 0:00 radiusd
Hmm. What does "ps -ejf" (on Linux - translate for closest equivalent on other systems) say for these?
root 25065 24736 23880 23880 0 15:11 ? 00:00:00 [acct-ng.pl] <defunct> root 25073 23885 23880 23880 0 15:11 ? 00:00:00 [acct-ng.pl] <defunct>
...so why do these have different parent PIDs (2nd column)? Could "acct-ng.pl" be itself forking? What process are those PIDs?
radiusds. Also, as a sidenote, I've tried with an empty bash script, same thing happens, it gets stuck as a zombie when called for accounting stop. -- George-Cristian Bîrzan Network Engineer _______________________________________ RCS & RDS Constanta Tel.: +40341.400.401 / +40341.400.402 Fax: +40341.400.450 http://www.rcs-rds.ro _______________________________________ Privileged/Confidential Information may be contained in this message. If you are not the addressee indicated in this message (or responsible for delivery of the message to such person), you may not copy or deliver this message to anyone. In such a case, you should destroy this message and kindly notify the sender by reply e-mail.
George-Cristian =?ISO-8859-1?Q?B=EErzan?= <gcbirzan@constanta.rdsnet.ro> wrote:
If I use threads, accounting stop processes remain as zombie processes. After a day or so, the server would die, not being able to create more processes. So, I've started restarting the RADIUS server hourly, but that's quite an ugly hack.
I recall it being a bug in pthreads on some platforms. I know it works fine on my Linux systems (FC3, RH9), so it's a little difficult for me to debug it.
waitpid(22058, NULL, WNOHANG) = -1 ECHILD (No child processes)
22058 exists, and is another process that hung.
That's definitely a bug in your OS somewhere. If 22058 exists, and is "defunct", then the parent process calling "waitpid" on it should *never* get ECHILD. Alan DeKok.
On Mon, 2006-02-13 at 13:39 -0500, Alan DeKok wrote:
waitpid(22058, NULL, WNOHANG) = -1 ECHILD (No child processes)
22058 exists, and is another process that hung.
That's definitely a bug in your OS somewhere. If 22058 exists, and is "defunct", then the parent process calling "waitpid" on it should *never* get ECHILD.
It's not the parent, from what I can tell. At which point it should do that, no? -- George-Cristian Bîrzan Network Engineer _______________________________________ RCS & RDS Constanta Tel.: +40341.400.401 / +40341.400.402 Fax: +40341.400.450 http://www.rcs-rds.ro _______________________________________ Privileged/Confidential Information may be contained in this message. If you are not the addressee indicated in this message (or responsible for delivery of the message to such person), you may not copy or deliver this message to anyone. In such a case, you should destroy this message and kindly notify the sender by reply e-mail.
George-Cristian =?ISO-8859-1?Q?B=EErzan?= <gcbirzan@constanta.rdsnet.ro> wrote:
It's not the parent, from what I can tell. At which point it should do that, no?
Yes, but then something else is seriously broken. If it's not the parent, then how the heck did it get the PID of the child? Alan DeKok.
George-Cristian Bîrzan wrote:
On Mon, 2006-02-13 at 13:39 -0500, Alan DeKok wrote:
waitpid(22058, NULL, WNOHANG) = -1 ECHILD (No child processes) 22058 exists, and is another process that hung. That's definitely a bug in your OS somewhere. If 22058 exists, and is "defunct", then the parent process calling "waitpid" on it should *never* get ECHILD.
It's not the parent, from what I can tell. At which point it should do that, no?
No it should not do that, that's why you are having problems. Posix MANDATES that any thread in a thread group can reap any child process, not just the thread that created the process. The Linux manpage for waitpid says this was fixed in the 2.4 kernel, but clearly not - possibly the problem lies in the pthread library (LinuxThreads) you're using, which is bundled with glibc. There's also the morass of SIGCHLD handling... I would suggest trying a system with proper NPTL threading (2.6 kernel, recent libc compiled appropriately) which will almost certainly work. Upgrading your main system to that may be a little more... involved.
Does anyone know of any freeRadius training classes? Thanks Archana
participants (4)
-
Alan DeKok -
Archana Vemulapalli -
George-Cristian Bîrzan -
Phil Mayers