This is a long shot, but if anyone has the time to read this, I'd appreciate any suggestions! I'm running FR 1.x on the same RHEL4 box that handles POP3/IMAP proxying (using 'perdition') and authenticated SMTP (using sendmail). I'm in the process of migrating from Funk/Juniper, so my other RADIUS servers are otherwise occupied. In the same general time frame I've been running FR on this box, we've started to notice random timeout problems when sending or reading mail. I've been trying to track this down for a month now, and I'm fresh out of ideas. The problem is so transient, only lasts a minute or so, so it's really hard to get a handle on whats happening. And its not as simple as it refuses all connections during that time . it's like (some) existing sessions get hung up, but new ones will work. And it doesn't effect all the people all the time. The machines has gobs of spare memory (only about 3GB of 8GB used) and cycles (I don't think I've ever seen the load average go over 2), even though it is fairly busy with mail. Maybe 40 or 50 concurrent POP3/IMAP proxy sessions and a dozen or so sendmails. The actual mailbox server the proxy talks to has similar amounts of headroom. The RADIUS load is negligible, just a dozen or so sectors of wireless users doing PPPOE. Call it a couple of dozen queries a minute or so. MySQL runs on another server, on the same gigabit switch, no network logjams anywhere. Local iostats show no disk bottlenecks anywhere. If it makes any difference, I run radiusd in -X mode, because it crashes when running as a service (valgrind showed Bad Things happening). About the only thing I can think of which the mail and RADIUS have in common is that they all use PAM/winbind to authenticate against a Windows AD. I have heard about issues with PAM, mostly reports of memory leaks, but it has always worked perfectly for me. This box has been running for about 18 months without a reboot. Until last week, when I tried the Big Stick approach, but the problem is still there after a reboot. My next step is to switch RADIUS over to using ntlm_auth instead of PAM/winbind. It already does ntlm_auth for MSCHAP requests, it's only the plain text that uses PAM. I initially configured it to use PAM because I couldn't get ntlm_auth to work outside of MSCHAP, using clear text, but I may have solved that one. Anyway . if nothing else, I'll let you know how it goes, in case someone with the same problem ever googles into this. -- hugh
On Jul 20, 2007, at 12:55, Hugh Messenger wrote:
If it makes any difference, I run radiusd in –X mode, because it crashes when running as a service (valgrind showed Bad Things happening).
While that may not be all of the issues, debug mode uses a lot of disk I/O. You might be getting delays accessing mail files from this. You need to figure out why it doesn't run as a service. I have been using it in service mode for years with no problems.
Doug Hardie said:
While that may not be all of the issues, debug mode uses a lot of disk I/O. You might be getting delays accessing mail files from this. You need to figure out why it doesn't run as a service. I have been using it in service mode for years with no problems.
Yup, that occurred to me as I wrote it. As is usually the case, summarizing a problem for other people tends help see things clearer yourself. So that's definitely the next step, ahead of cutting PAM out of the loop. If I understand how ntlm_auth works, it'll still use winbind, but I hope I can at least remove PAM from the equation. I'm about to fire up valgrind and see if 1.1.7 fixes the run-as-service problem. I know 2.0 does, I have a test setup on the hot spare for the mail/radius box, but I'm just not quite brave enough to run 2.0 live yet. I did report the run-as-service problem on this list a month or so back, but it boiled down to a case of "Hmmm, it shouldn't do that". -- hugh
On Fri 20 Jul 2007, Hugh Messenger wrote:
This is a long shot, but if anyone has the time to read this, I'd appreciate any suggestions!
I'm running FR 1.x on the same RHEL4 box that handles POP3/IMAP proxying (using 'perdition') and authenticated SMTP (using sendmail). I'm in the process of migrating from Funk/Juniper, so my other RADIUS servers are otherwise occupied.
Are you doing lots of SSL mail connections? Are you also doing Wireless EAP? I have heard of otherwise underloaded boxes doing "random" things like this when they run out of randomness while doing to many crypto sessions. (Too "many", not too "much"... ie. Not a CPU issue..) -- Peter Nixon http://peternixon.net/
participants (3)
-
Doug Hardie -
Hugh Messenger -
Peter Nixon