<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<META NAME="Generator" CONTENT="MS Exchange Server version 6.5.7654.12">
<TITLE>Freeradius and memory usage part deux</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->
<P><FONT SIZE=2 FACE="Arial">Greetings,</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial">I started this email few months back but due to other projects that took priority, I had to put this troubleshooting on hold. To recap:</FONT></P>
<P><FONT SIZE=2 FACE="Arial">1. Our radius servers are having some memory issue where its memory usage would increase by 1% every 30 - 45 minutes until it crashes and restarts.</FONT></P>
<P><FONT SIZE=2 FACE="Arial">2. I created a script that will temporarily fix the issue by monitoring its memory usage and restart after it reaches the threshold.</FONT></P>
<P><FONT SIZE=2 FACE="Arial">3. Changed detail log configuration so that it creates a new one every hour but memory usage still increasing but at a slower rate.</FONT></P>
<P><FONT SIZE=2 FACE="Arial">4. Upgraded to 2.1.6 from 2.1.5 but the problem still exist. I recently tested the latest 2.1.8 as well but still have the same issue.</FONT></P>
<P><FONT SIZE=2 FACE="Arial">Our server spec is as follow:</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial">OS: CentOS (5.4)</FONT>
<BR><FONT SIZE=2 FACE="Arial">Freeradius version: 2.1.6</FONT>
<BR><FONT SIZE=2 FACE="Arial">CPU: single quad core Xeon</FONT>
<BR><FONT SIZE=2 FACE="Arial">Memory: 4GB</FONT>
<BR><FONT SIZE=2 FACE="Arial">HD: FC drives on EMC</FONT>
<BR><FONT SIZE=2 FACE="Arial">Database: MySQL on a dedicated MySQL server</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial">We also have a simple perl module script that takes IMSI passes by our PDSN and checks it against the username. I did ran valgrind yesterday and the output is below:</FONT></P>
<P><FONT SIZE=2 FACE="Arial">==17709== Memcheck, a memory error detector.</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== Copyright (C) 2002-2006, and GNU GPL'd, by Julian Seward et al.</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== Using LibVEX rev 1658, a library for dynamic binary translation.</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== Copyright (C) 2004-2006, and GNU GPL'd, by OpenWorks LLP.</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== Using valgrind-3.2.1, a dynamic binary instrumentation framework.</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== Copyright (C) 2000-2006, and GNU GPL'd, by Julian Seward et al.</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== For more details, rerun with: -v</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709==</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709==</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 78 from 1)</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== malloc/free: in use at exit: 3,013,615 bytes in 35,591 blocks.</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== malloc/free: 45,091 allocs, 9,500 frees, 3,698,764 bytes allocated.</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== For counts of detected errors, rerun with: -v</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== searching for pointers to 35,591 not-freed blocks.</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== checked 2,451,476 bytes.</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709==</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== 5 bytes in 1 blocks are definitely lost in loss record 1 of 36</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== at 0x40053C0: malloc (vg_replace_malloc.c:149)</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== by 0x561B2E: Perl_safesysmalloc (in /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so)</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== by 0x5647E7: Perl_savepvn (in /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so)</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== by 0x51CE15: Perl_gv_fetchpv (in /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so)</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== by 0x51481F: (within /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so)</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== by 0x518608: perl_parse (in /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so)</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== by 0x435E393: perl_instantiate (rlm_perl.c:463)</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== by 0x805E8EB: find_module_instance (modules.c:501)</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== by 0x805FC2E: do_compile_modsingle (modcall.c:1777)</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== by 0x806096E: do_compile_modgroup (modcall.c:1997)</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== by 0x805DA21: load_component_section (modules.c:666)</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== by 0x805DEC4: load_byserver (modules.c:987)</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709==</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== LEAK SUMMARY:</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== definitely lost: 5 bytes in 1 blocks.</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== possibly lost: 0 bytes in 0 blocks.</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== still reachable: 3,013,610 bytes in 35,590 blocks.</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== suppressed: 0 bytes in 0 blocks.</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== Reachable blocks (those to which a pointer was found) are not shown.</FONT>
<BR><FONT SIZE=2 FACE="Arial">==17709== To see them, rerun with: --show-reachable=yes</FONT>
</P>
<BR>
<P><FONT SIZE=2 FACE="Arial">I hope those information helps. We also have two freeradius servers in our remote datacenter still running 2.1.2 without any issue but I don't see a point rolling back to the older version so hopefully we can fix this issue with the current version if possible.</FONT></P>
<BR>
<BR>
<P><FONT SIZE=2 FACE="Arial">Cheers,</FONT>
<BR><FONT FACE="Arial"> </FONT>
<BR><FONT SIZE=2 FACE="Arial">Roy Kartadinata</FONT>
<BR><FONT FACE="Arial"> </FONT>
</P>
</BODY>
</HTML>