George C. Kaplan wrote:

I can't speak to the MySQL problems, but we've observed the same lock-up behavior of the daemon: unresponsive to RADIUS requests, 98% CPU usage, only a 'kill -9' will break it loose. (We're running FR 1.0.5 on FreeBSD 5.5).

In our case, the daemon appears to get wedged only if a signal (HUP, e.g.) arrives just as it's handling a kerberos authentication request. If I can speculate, perhaps the signal-handling bug is not just in the rlm_krb5 module, but a more general problem that can also affect rlm_sql.

When I asked about our problem back in March, it was suggested that we upgrade to 1.1.0 (now 1.1.1), as that release has some signal handling bug fixes. We're finally ready to upgrade (tomorrow), so we'll see if that helps.

Apparently 1.1.1 has the same problem, but at least I've found a way to trigger the lockup at will: - Configure freeradius to authenticate to a kerberos server - Set up a dummy kerberos server that just accepts TCP connections on port 88 but doesn't send anything back. (I just used 'nc -l 88'). - Change /etc/krb5.conf on the freeradius server to point to the dummy kerberos server. - Use 'radtest' to send an authentication request to freeradius. If you just leave it alone, radiusd will timeout after several seconds, sending an Access-Reject, and logging a "Cannot contact any KDC..." message. - Before it times out, send a HUP to the radiusd process. After a few seconds, the CPU utilization will start to climb, eventually reaching about 98%. At this point the daemon will not respond to any RADIUS requests, even for huntgroups that don't use kerberos. The only way out is to kill the daemon and restart it. (Under 1.0.5 I generally had to do 'kill -9', but now a 'kill -TERM' seems to work). This appears to be related to threaded operation, since the daemon does *not* get wedged if it's running with -s or -X options. I haven't tried this with any other authentication modules. My current system: freeradius 1.1.1, compiled from ports with MIT kerberos support FreeBSD 5.5-PRERELEASE I'll file a bug report once my bugzilla password comes through. In the meantime, suggestions for more detailed troubleshooting here are welcome. -- George C. Kaplan gckaplan@ack.berkeley.edu Communication & Network Services 510-643-0496 University of California at Berkeley

Alan DeKok wrote:

"George C. Kaplan" <gckaplan@ack.berkeley.edu> wrote:

...

freeradius 1.1.1, compiled from ports with MIT kerberos support FreeBSD 5.5-PRERELEASE

I'll file a bug report once my bugzilla password comes through. In the meantime, suggestions for more detailed troubleshooting here are welcome.

I don't run kerberos myself, so I'm unsure as to how to reproduce it. Could you attach example krb.conf files, and the "krb5{}" section from radiusd.conf?

I put some notes on a minimal radius config in bug report #365. A bare-bones version of our krb5.conf is attached. Substitute the FQHN of your system running a dummy kerberos server ('nc -l 88' works) in the 'kdc' line. If you've got it set up right, a 'radtest' will time out after several seconds with a "Cannot contact any KDC..." message in radius.log. If you HUP the radiusd before it times out, that should trigger the lockup.

Or, you could use gdb to attach to the running process. e.g.

$ radiusd ... $ ps -ef | grep radiusd $ nice +15 <radiusd-pid> <reproduce the problem> $ gdb `which radiusd` <radiusd-pid>

$ thread apply all bt full $ cont

You may have to do the last two steps a few times to see where the server is locked.

OK, I'll try to do this in the next day or so. -- George C. Kaplan gckaplan@ack.berkeley.edu Communication & Network Services 510-643-0496 University of California at Berkeley [libdefaults] default_realm = BERKELEY.EDU [realms] BERKELEY.EDU = { kdc = kerberos.berkeley.edu:88 }