rlm_perl/radius issues with db connections

[Meyers, Dan]

I'm having a few issues with FreeRadius 2.1.4 (2.1.6 isn't in ports yet)
and MySQL 5.1 on FreeBSD 6.2.

Specifically, I am using the CLONE method in my perl script to create my
database handles every time a new thread is spawned. If I start radius
and then do 'show processlist' on the MySQL server I see no connections
from the radius server. If I then send a single update from radclient I
get a connection in show processlist, and a timely response to my
request.

However in our live environment the radius server is fairly heavily used
by a large number of NASes, so as soon as it is turned on it gets a
large influx of requests, and the logs fill with 'Error: WARNING:
Unresponsive child for request X' messages. This goes on for a good
10-20 seconds or so, during which the processlist on the MySQL server
still shows no connections. Then, suddenly, the error messages stop and
all my connections appear in the processlist. I'm *think* this delay is
in instantiating perl for each radius thread, as the error messages
still get spammed into the log for 10-20 seconds even if my authenticate
and accounting perl functions simply contain 'return RLM_MODULE_OK;'. I
am guessing that once perl has been instantiated for the radius thread
the creation of db connections and such occurs quite quickly, but if
radius suddenly tries to start a large number of perl processes at once
the server gets bogged down.

This behaviour can be replicated in our test environment using
radclient. No matter how long I wait after starting the radius server, I
always get these error messages as soon as I start bombarding the server
with requests. And of course until I start bombarding it it doesn't need
to start any extra threads.

The initial issue was that we were getting these error messages in the
live environment after some time of the server running, and they weren't
ever going away. You had to restart radius to get rid of them, and there
was a reasonable chance that they'd reappear as soon as you restarted
it. I *believe* the issue there was actually to do with mysql requests
hanging as well as perl starting, although I am still unsure as to why,
as our client was compiled with the thread safe options and the server
doesn't appear to have resource issues. That was, however, with 2.0.3.
The issue of the threads never becoming responsive again doesn't
*appear* to exist in 2.1.4, from my testing. An issue we are still
seeing, however, is that sometimes no results are returned from the db.
DBD doesn't throw an exception or error, we just get an empty result set
back occasionally. I'm investigating that at the mo.

I was just wondering if anyone else had come across any similar
problems, and how you got around them. Ideally, i'd like for the perl
process to be instantiated before radius starts trying to use the thread
for requests. That way the radius server would simply not respond to a
request if no threads were ready and free, and my NASes could retry/fail
over/whatever, whereas at the moment it seems to be sending back rejects
while perl is still instantiating in the thread. I could also then tell
radius to start with a reasonable number of threads to handle 'normal'
load, and it wouldn't actually say it was ready to process requests, and
start sending anything back, until all my normal required threads and
their respective perl processes had been started.

--
Dan Meyers
Network Specialist, Lancaster University
E-Mail: d.meyers at lancaster.ac.uk