radiusd stop responding. deadlock?

Fri Jun 22 07:22:53 CEST 2007

Hi,
I'm *sorry* that I am not good at English
because I'm Japanese.

We using freeradius 1.1.0 for PEAP authentication,
and it is working well almost.
but sometime, radiusd stops responding.CPU usage is 100%.
(need to radiusd stop/start).

following is result of ps.
--------------------------------
#ps -efml | grep radius
1 S root     10134 10127  0  69   0    -  7395 148a46 Jun19 ?       
00:00:00 /usr/dot1x/sbin/radiusd -d /e
tc/raddb -l syslog -g local0 -x
1 S root     17737 10134  0  69   0    -  7395 10800b Jun19 ?       
00:00:01 /usr/dot1x/sbin/radiusd -d /e
tc/raddb -l syslog -g local0 -x
1 S root     17835 10134  0  68   0    -  7395 10800b Jun19 ?       
00:00:00 /usr/dot1x/sbin/radiusd -d /e
tc/raddb -l syslog -g local0 -x
0 R root     10127     1 47  80   0    -  7395      - Jun19 ?       
19:24:13 /usr/dot1x/sbin/radiusd -d /e
tc/raddb -l syslog -g local0 -x
1 S root     17769 10134  0  69   0    -  7395 10800b Jun19 ?       
00:00:01 /usr/dot1x/sbin/radiusd -d /e
tc/raddb -l syslog -g local0 -x
1 R root     17800 10134 49  76   0    -  7395      - Jun19 ?       
19:19:02 /usr/dot1x/sbin/radiusd -d /e
tc/raddb -l syslog -g local0 -x
1 S root     17713 10134  0  69   0    -  7395 10800b Jun19 ?       
00:00:01 /usr/dot1x/sbin/radiusd -d /e
tc/raddb -l syslog -g local0 -x
--------------------------------

pid:10127(main) eats 47% CPU.
pid:17800 eats 49% CPU.

following is result of gdb about these thread.
--------------------------------
# gdb
GNU gdb 5.3
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you
are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-slackware-linux".
(gdb) attach 10127
Attaching to process 10127
Reading symbols from /usr/dot1x/sbin/radiusd...(no debugging symbols
found)...
done.
....(snip)...
0x4019a072 in pthread_once () from /lib/libpthread.so.0
(gdb) where
#0  0x4019a072 in pthread_once () from /lib/libpthread.so.0
#1  0x4027df03 in malloc () from /lib/libc.so.6
#2  0x401eae87 in rad_recv () from /usr/dot1x/lib/libradius-1.1.0.so
#3  0x0804d78b in main ()
#4  0x4021cd06 in __libc_start_main () from /lib/libc.so.6
(gdb) detach
Detaching from program: /usr/dot1x/sbin/radiusd, process 10127
(gdb) 
(gdb) 
(gdb) attach 10127
Attaching to program: /usr/dot1x/sbin/radiusd, process 10127
Symbols already loaded for /lib/libcrypt.so.1
....(snip)...
0x401998cc in pthread_mutex_trylock () from /lib/libpthread.so.0
(gdb) whrere
#0  0x401998cc in pthread_mutex_trylock () from /lib/libpthread.so.0
#1  0x4027c65f in _IO_file_xsputn () from /lib/libc.so.6
#2  0x4027df03 in malloc () from /lib/libc.so.6
#3  0x401eae87 in rad_recv () from /usr/dot1x/lib/libradius-1.1.0.so
#4  0x0804d78b in main ()
#5  0x4021cd06 in __libc_start_main () from /lib/libc.so.6
(gdb) detach
Detaching from program: /usr/dot1x/sbin/radiusd, process 10127
(gdb) 
(gdb) attach 17800
Attaching to program: /usr/dot1x/sbin/radiusd, process 17800
Symbols already loaded for /lib/libcrypt.so.1
....(snip)...
0x4027f4b0 in mallopt () from /lib/libc.so.6
(gdb) 
(gdb) where
#0  0x4027f4b0 in mallopt () from /lib/libc.so.6
#1  0x4027ed83 in mallopt () from /lib/libc.so.6
#2  0x4027df1a in malloc () from /lib/libc.so.6
#3  0x4038a2e9 in ber_memalloc_x () from /usr/dot1x/lib/liblber-2.2.so.7
#4  0x4038a478 in ber_memrealloc_x () from /usr/dot1x/lib/liblber-2.2.so.7
#5  0x40388790 in ber_realloc () from /usr/dot1x/lib/liblber-2.2.so.7
#6  0x403886b4 in ber_write () from /usr/dot1x/lib/liblber-2.2.so.7
#7  0x40387179 in ber_put_tag () from /usr/dot1x/lib/liblber-2.2.so.7
#8  0x40387404 in ber_put_int_or_enum () from
/usr/dot1x/lib/liblber-2.2.so.7
#9  0x40388416 in ber_printf () from /usr/dot1x/lib/liblber-2.2.so.7
#10 0x403612f2 in ldap_build_search_req () from
/usr/dot1x/lib/libldap-2.2.so.7
#11 0x403611da in ldap_search () from /usr/dot1x/lib/libldap-2.2.so.7
#12 0x40361432 in ldap_search_st () from /usr/dot1x/lib/libldap-2.2.so.7
#13 0x4033f803 in _init () from /usr/dot1x/lib/rlm_ldap-1.1.0.so
#14 0x403400d6 in _init () from /usr/dot1x/lib/rlm_ldap-1.1.0.so
#15 0x08050c19 in vp_listdebug ()
#16 0x08050fde in paircmp ()
#17 0x403f820f in _init () from /usr/dot1x/lib/rlm_files-1.1.0.so
#18 0x08057c99 in module_post_auth ()
#19 0x080582df in modcall ()
#20 0x08057d0f in module_post_auth ()
#21 0x08057e06 in module_post_auth ()
#22 0x08058254 in modcall ()
#23 0x08057d0f in module_post_auth ()
#24 0x08057e06 in module_post_auth ()
#25 0x08058254 in modcall ()
#26 0x08056fff in find_module_instance ()
#27 0x080578f0 in module_authorize ()
#28 0x08053aa3 in rad_authenticate ()
#29 0x0804dcf8 in rad_respond ()
#30 0x0805b5a0 in radius_xlat ()
#31 0x40198ca3 in pthread_detach () from /lib/libpthread.so.0
(gdb) detach
Detaching from program: /usr/dot1x/sbin/radiusd, process 17800
(gdb) 
--------------------------------

it look like deadlock occured when malloc().

is it fixed FreeRADIUS 1.1.6 ?
or other problem ?
any idea ?

  we use PEAP, rlm_ldap

  glibc 2.3.2
  openssl 0.9.7g 
  openldap 2.2.27
  gcc 3.2.3

Best Regards,
S.Obayashi.

--------------------------------------
Start Yahoo! Auction now! Check out the cool campaign
http://pr.mail.yahoo.co.jp/auction/