"Exiting normally" on FreeBSD -- Synopsis?

[Brian A. Seklecki]

> Next step must be ktrace(8)/kdump(8) or GDB [1].
>
> ~BAS
>


So it turns out, since April, there have been two distinctive types of 
crashes.

The unexplained SIGHUP, which we eventually tracked down to faulty 
logging configurations (now using SYSLOG instead of file logging), and 
an ongoing Sig11.


We finally got around to setting up core dump archival and here's what 
we found:


% sudo gdb ../local/sbin/radiusd /var/core/133.radiusd.core

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Core was generated by `radiusd'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from 
/usr/local/lib/freeradius-2.1.9/libfreeradius-radius-2.1.9.so...done.
Loaded symbols for 
/usr/local/lib/freeradius-2.1.9/libfreeradius-radius-2.1.9.so
Reading symbols from /lib/libcrypt.so.5...done.
Loaded symbols for /lib/libcrypt.so.5
Reading symbols from /usr/local/lib/libltdl.so.7...done.
Loaded symbols for /usr/local/lib/libltdl.so.7
Reading symbols from /usr/lib/libssl.so.6...done.
Loaded symbols for /usr/lib/libssl.so.6
Reading symbols from /lib/libcrypto.so.6...done.
Loaded symbols for /lib/libcrypto.so.6
Reading symbols from /lib/libthr.so.3...done.
Loaded symbols for /lib/libthr.so.3
Reading symbols from /lib/libc.so.7...done.
Loaded symbols for /lib/libc.so.7
Reading symbols from /usr/local/lib/freeradius-2.1.9/rlm_exec.so...done.
Loaded symbols for /usr/local/lib/freeradius-2.1.9/rlm_exec.so
Reading symbols from /usr/local/lib/freeradius-2.1.9/rlm_expr.so...done.
Loaded symbols for /usr/local/lib/freeradius-2.1.9/rlm_expr.so
Reading symbols from /usr/local/lib/freeradius-2.1.9/rlm_chap.so...done.
Loaded symbols for /usr/local/lib/freeradius-2.1.9/rlm_chap.so
Reading symbols from /usr/local/lib/freeradius-2.1.9/rlm_mschap.so...done.
Loaded symbols for /usr/local/lib/freeradius-2.1.9/rlm_mschap.so
Reading symbols from /usr/local/lib/freeradius-2.1.9/rlm_pap.so...done.
Loaded symbols for /usr/local/lib/freeradius-2.1.9/rlm_pap.so
Reading symbols from /usr/local/lib/freeradius-2.1.9/rlm_realm.so...done.
Loaded symbols for /usr/local/lib/freeradius-2.1.9/rlm_realm.so
Reading symbols from /usr/local/lib/freeradius-2.1.9/rlm_ldap.so...done.
Loaded symbols for /usr/local/lib/freeradius-2.1.9/rlm_ldap.so
Reading symbols from /usr/local/lib/libldap_r-2.4.so.7...done.
Loaded symbols for /usr/local/lib/libldap_r-2.4.so.7
Reading symbols from /usr/local/lib/liblber-2.4.so.7...done.
Loaded symbols for /usr/local/lib/liblber-2.4.so.7
Reading symbols from 
/usr/local/lib/freeradius-2.1.9/rlm_preprocess.so...done.
Loaded symbols for /usr/local/lib/freeradius-2.1.9/rlm_preprocess.so
Reading symbols from 
/usr/local/lib/freeradius-2.1.9/rlm_acct_unique.so...done.
Loaded symbols for /usr/local/lib/freeradius-2.1.9/rlm_acct_unique.so
Reading symbols from /usr/local/lib/freeradius-2.1.9/rlm_files.so...done.
Loaded symbols for /usr/local/lib/freeradius-2.1.9/rlm_files.so
Reading symbols from /usr/local/lib/freeradius-2.1.9/rlm_detail.so...done.
Loaded symbols for /usr/local/lib/freeradius-2.1.9/rlm_detail.so
Reading symbols from /usr/local/lib/freeradius-2.1.9/rlm_radutmp.so...done.
Loaded symbols for /usr/local/lib/freeradius-2.1.9/rlm_radutmp.so
Reading symbols from /libexec/ld-elf.so.1...done.
Loaded symbols for /libexec/ld-elf.so.1
#0  0x0000000000409c49 in rad_authenticate (request=0x80138ba40) at 
auth.c:667
667			request->reply->code = PW_AUTHENTICATION_REJECT;
[New Thread 801208740 (LWP 100229)]
[New Thread 801208900 (LWP 100220)]
[New Thread 801208ac0 (LWP 100135)]
[New Thread 801208c80 (LWP 100132)]
[New Thread 801208e40 (LWP 100108)]
[New Thread 8012021c0 (LWP 100395)]




And:


(gdb) bt
#0  0x0000000000409c49 in rad_authenticate (request=0x80138ba40) at 
auth.c:667
#1  0x000000000043241c in radius_handle_request (request=0x80138ba40, 
fun=0x409700 <rad_authenticate>) at event.c:3735
#2  0x0000000000425311 in request_handler_thread (arg=0x8013799c0) at 
threads.c:493
#3  0x0000000800e509ed in pthread_create () from /lib/libthr.so.3
#4  0x0000000000000000 in ?? ()
Cannot access memory at address 0x7fffff5fc000



This is where I normally defer to Steve/Alan for a bit of GDB/C skills


I'd be happy to provide a redacted radiusd.conf(5) and and run any GDB 
commands, we just have to keep the contents of memory under wraps to 
protect passwords (We use LDAP, so.)



This is:


[bseklecki at blackout ~]$ radiusd -v
radiusd: FreeRADIUS Version 2.1.9, for host amd64-portbld-freebsd8.0, 
built on Jun 30 2010 at 11:46:51
Copyright (C) 1999-2009 The FreeRADIUS server project and contributors.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
You may redistribute copies of FreeRADIUS under the terms of the
GNU General Public License.
For more information about these matters, see the file named COPYRIGHT.

Happy holidays, all,
  ~BAS