FR 2.1.7 Exits for no reason

[Gary Gatten]

You seem fairly intelligent (and a Ninja) so I figured you'd be familiar with gdb and just needed a nudge that direction.

Gdb will have a lot of good info at abend time: what each thread was executing, stack info, etc. It may give you a clue or more importantly Alen.

If you google "gdb mini tutorial" I'm sure you'll find info. I found mine in the nTop FAQ. If I was near my system I'd send you more details - I will in the AM if you can't find it.

G

From: McNutt, Justin M. [mailto:McNuttJ at missouri.edu]
Sent: Tuesday, March 08, 2011 10:12 PM
To: FreeRadius users mailing list <freeradius-users at lists.freeradius.org>
Subject: RE: FR 2.1.7 Exits for no reason

You must realize that "gdb" by itself is an answer that is of very little use.  While I am aware that gdb is the GNU Debugger, you have no way of knowing that I do, and you gave no other context or other information that would help me use gdb to gather anything.

So let me be more clear:

What EXACTLY do I need to do to get more information about this phenomenon, and under what circumstances do I need to do it, and once I have some output, what should I be looking for in it?  Running production RADIUS servers with "strace radiusd -X" is probably impractical (and highly insecure), and may even alter the runtime environment such that the fatal event never occurs.  I've never observed the failure in either of the two test servers I run, and their configurations are identical, so I must assume that radiusd dies after receiving some sort of improper/unexpected data, or when it gets into some weird state, or other such thing.

But it can't be fixed if I can't figure out how to reproduce it.  It'll happen eventually, but a server that is no longer running doesn't tell me much either.  How is gdb going to help me figure out why something isn't working any more?

--J

________________________________
From: freeradius-users-bounces+mcnuttj=missouri.edu at lists.freeradius.org [mailto:freeradius-users-bounces+mcnuttj=missouri.edu at lists.freeradius.org] On Behalf Of Gary Gatten
Sent: Tuesday, March 08, 2011 5:06 PM
To: 'freeradius-users at lists.freeradius.org'
Subject: Re: FR 2.1.7 Exits for no reason

Gdb

From: McNutt, Justin M. [mailto:McNuttJ at missouri.edu]
Sent: Tuesday, March 08, 2011 04:59 PM
To: freeradius-users at lists.freeradius.org <freeradius-users at lists.freeradius.org>
Subject: FR 2.1.7 Exits for no reason

Hey all,

So the host-based auth stuff is working well now, but we've discovered another problem.

We have four FR 2.1.7 servers running on RHEL 5 (fully patched).  Every now and then, for no apparent reason, radiusd just stops.  It exits with "Exiting normally." to syslog.  They don't all exit at the same time.  Since there are four of them behind a load balancer, it usually doesn't result in a service outage, and we've been lucky so far that only a couple of them have been down at once.  But it's still disconcerting.

The servers tend to all be started within a minute of each other, since I make changes to Server #1, and then use an rsync script to replicate /etc/raddb to the other servers and restart them.  So they all start within seconds of one another.  This week, Server #3 stopped within about 8 hours of being started (went from 1130 to 1930).  Server #1 failed last week at 2330.  Server #4 hasn't failed yet.  It's very odd.

Any ideas on how I can troubleshoot this?

Thanks!

Justin McNutt
Network Systems Analyst - Ninja
DNPS, Mizzou Telecom
(573) 882-5183

"Do you have a concussion?"

Ping is NOT a service.  You don't need it.  Use a real test.


"This email is intended to be reviewed by only the intended recipient and may contain information that is privileged and/or confidential. If you are not the intended recipient, you are hereby notified that any review, use, dissemination, disclosure or copying of this email and its attachments, if any, is strictly prohibited. If you have received this email in error, please immediately notify the sender by return email and delete this email from your system."





<font size="1">
<div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
</div>
"This email is intended to be reviewed by only the intended recipient
 and may contain information that is privileged and/or confidential.
 If you are not the intended recipient, you are hereby notified that
 any review, use, dissemination, disclosure or copying of this email
 and its attachments, if any, is strictly prohibited.  If you have
 received this email in error, please immediately notify the sender by
 return email and delete this email from your system."
</font>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freeradius.org/pipermail/freeradius-users/attachments/20110308/f62a9d67/attachment.html>