cannot get core dump of crashing freeradius

[Alan DeKok]

Jakob Hirsch wrote:
> we have freeradius 2.1.8 running on a couple of servers and are very
> happy with it. But every few days FR crashes on one of the servers (a
> random one, not always the same). The load is significant (average 150
> requests/s per server, 400/s peak) but sureley not too high. So
> everything seems to run fine besides the annoying crashes, which alarms
> people and make the weekly availibility reports look bad (even though FR
> is restarted automatically, of course). The previous 1.1.8 installation
> we upgraded 6 months ago from did not have this problem.

  Hmm... I've run it at 20K pps for *days*....

> Anyways, I really want to find out what's going wrong, so I wanted to
> get core dumps of these crashes. Only that I just don't get them.
> - radiusd.conf has allow_core_dumps = yes (and FR says "Info: Core dumps
> are enabled." at startup)
> - /proc/sys/kernel/core_pattern is set to '/tmp/core.%t.%e.%p', so core
> dumps can be written to disk (tested with a little programm that forces
> a segfault)
> - I put "ulimit -c unlimited" in the startup script.
> cat /proc/$(pidof freeradius)/limits shows "unlimited" for soft and hard
> limit of "Max core file size"

  Often 'root' can't core dump, and programs that change uid can't core
dump.  It's hard to know what's going on with the OS.

> So what's missing? The only indication of the crash is this line in syslog:
> 
>> Apr 10 17:57:19 xxxxxxxx kernel: [12268615.000288] freeradius[14846]: segfault at 73818 ip 00007f0cb40e875e sp 00007fff9c6304c0 error 4 in libfreeradius-radius-2.1.8.so[7f0cb40d1000+1f000]
> 
> (This is debian lenny x86_64, btw.)
>
> Any hints?

  doc/bugs.  You'll need symbols to find out what's going on.

  Alan DeKok.