Unstable FreeRadius

[George C. Kaplan]

George C. Kaplan wrote:

> I can't speak to the MySQL problems, but we've observed the same lock-up
> behavior of the daemon:  unresponsive to RADIUS requests, 98% CPU usage,
> only a 'kill -9' will break it loose.  (We're running FR 1.0.5 on
> FreeBSD 5.5).
> 
> In our case, the daemon appears to get wedged only if a signal (HUP,
> e.g.) arrives just as it's handling a kerberos authentication request.
> If I can speculate, perhaps the signal-handling bug is not just in the
> rlm_krb5 module, but a more general problem that can also affect rlm_sql.
> 
> When I asked about our problem back in March, it was suggested that we
> upgrade to 1.1.0 (now 1.1.1), as that release has some signal handling
> bug fixes.  We're finally ready to upgrade (tomorrow), so we'll see if
> that helps.

Apparently 1.1.1 has the same problem, but at least I've found a way to
trigger the lockup at will:

- Configure freeradius to authenticate to a kerberos server

- Set up a dummy kerberos server that just accepts TCP connections on
port 88 but doesn't send anything back.  (I just used 'nc -l 88').

- Change /etc/krb5.conf on the freeradius server to point to the dummy
kerberos server.

- Use 'radtest' to send an authentication request to freeradius.  If you
just leave it alone, radiusd will timeout after several seconds, sending
an Access-Reject, and logging a "Cannot contact any KDC..." message.

- Before it times out, send a HUP to the radiusd process.  After a few
seconds, the CPU utilization will start to climb, eventually reaching
about 98%.  At this point the daemon will not respond to any RADIUS
requests, even for huntgroups that don't use kerberos.  The only way out
is to kill the daemon and restart it.  (Under 1.0.5 I generally had to
do 'kill -9', but now a 'kill -TERM' seems to work).

This appears to be related to threaded operation, since the daemon does
*not* get wedged if it's running with -s or -X options.  I haven't tried
this with any other authentication modules.

My current system:

   freeradius 1.1.1, compiled from ports with MIT kerberos support
   FreeBSD 5.5-PRERELEASE

I'll file a bug report once my bugzilla password comes through.  In the
meantime, suggestions for more detailed troubleshooting here are welcome.

-- 
George C. Kaplan                            gckaplan at ack.berkeley.edu
Communication & Network Services            510-643-0496
University of California at Berkeley