Freeradius and memory usage part deux

[Roy Kartadinata]

Greetings,

I started this email few months back but due to other projects that took priority, I had to put this troubleshooting on hold. To recap:

1. Our radius servers are having some memory issue where its memory usage would increase by 1% every 30 - 45 minutes until it crashes and restarts.
2. I created a script that will temporarily fix the issue by monitoring its memory usage and restart after it reaches the threshold.
3. Changed detail log configuration so that it creates a new one every hour but memory usage still increasing but at a slower rate.
4. Upgraded to 2.1.6 from 2.1.5 but the problem still exist. I recently tested the latest 2.1.8 as well but still have the same issue.

Our server spec is as follow:

OS: CentOS (5.4)
Freeradius version: 2.1.6
CPU: single quad core Xeon
Memory: 4GB
HD: FC drives on EMC
Database: MySQL on a dedicated MySQL server

We also have a simple perl module script that takes IMSI passes by our PDSN and checks it against the username. I did ran valgrind yesterday and the output is below:

==17709== Memcheck, a memory error detector.
==17709== Copyright (C) 2002-2006, and GNU GPL'd, by Julian Seward et al.
==17709== Using LibVEX rev 1658, a library for dynamic binary translation.
==17709== Copyright (C) 2004-2006, and GNU GPL'd, by OpenWorks LLP.
==17709== Using valgrind-3.2.1, a dynamic binary instrumentation framework.
==17709== Copyright (C) 2000-2006, and GNU GPL'd, by Julian Seward et al.
==17709== For more details, rerun with: -v
==17709==
==17709==
==17709== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 78 from 1)
==17709== malloc/free: in use at exit: 3,013,615 bytes in 35,591 blocks.
==17709== malloc/free: 45,091 allocs, 9,500 frees, 3,698,764 bytes allocated.
==17709== For counts of detected errors, rerun with: -v
==17709== searching for pointers to 35,591 not-freed blocks.
==17709== checked 2,451,476 bytes.
==17709==
==17709== 5 bytes in 1 blocks are definitely lost in loss record 1 of 36
==17709==    at 0x40053C0: malloc (vg_replace_malloc.c:149)
==17709==    by 0x561B2E: Perl_safesysmalloc (in /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so)
==17709==    by 0x5647E7: Perl_savepvn (in /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so)
==17709==    by 0x51CE15: Perl_gv_fetchpv (in /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so)
==17709==    by 0x51481F: (within /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so)
==17709==    by 0x518608: perl_parse (in /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so)
==17709==    by 0x435E393: perl_instantiate (rlm_perl.c:463)
==17709==    by 0x805E8EB: find_module_instance (modules.c:501)
==17709==    by 0x805FC2E: do_compile_modsingle (modcall.c:1777)
==17709==    by 0x806096E: do_compile_modgroup (modcall.c:1997)
==17709==    by 0x805DA21: load_component_section (modules.c:666)
==17709==    by 0x805DEC4: load_byserver (modules.c:987)
==17709==
==17709== LEAK SUMMARY:
==17709==    definitely lost: 5 bytes in 1 blocks.
==17709==      possibly lost: 0 bytes in 0 blocks.
==17709==    still reachable: 3,013,610 bytes in 35,590 blocks.
==17709==         suppressed: 0 bytes in 0 blocks.
==17709== Reachable blocks (those to which a pointer was found) are not shown.
==17709== To see them, rerun with: --show-reachable=yes


I hope those information helps. We also have two freeradius servers in our remote datacenter still running 2.1.2 without any issue but I don't see a point rolling back to the older version so hopefully we can fix this issue with the current version if possible.



Cheers,
 
Roy Kartadinata
 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freeradius.org/pipermail/freeradius-users/attachments/20100104/8af271e0/attachment.html>