Freeradius and memory usage part deux
Greetings, I started this email few months back but due to other projects that took priority, I had to put this troubleshooting on hold. To recap: 1. Our radius servers are having some memory issue where its memory usage would increase by 1% every 30 - 45 minutes until it crashes and restarts. 2. I created a script that will temporarily fix the issue by monitoring its memory usage and restart after it reaches the threshold. 3. Changed detail log configuration so that it creates a new one every hour but memory usage still increasing but at a slower rate. 4. Upgraded to 2.1.6 from 2.1.5 but the problem still exist. I recently tested the latest 2.1.8 as well but still have the same issue. Our server spec is as follow: OS: CentOS (5.4) Freeradius version: 2.1.6 CPU: single quad core Xeon Memory: 4GB HD: FC drives on EMC Database: MySQL on a dedicated MySQL server We also have a simple perl module script that takes IMSI passes by our PDSN and checks it against the username. I did ran valgrind yesterday and the output is below: ==17709== Memcheck, a memory error detector. ==17709== Copyright (C) 2002-2006, and GNU GPL'd, by Julian Seward et al. ==17709== Using LibVEX rev 1658, a library for dynamic binary translation. ==17709== Copyright (C) 2004-2006, and GNU GPL'd, by OpenWorks LLP. ==17709== Using valgrind-3.2.1, a dynamic binary instrumentation framework. ==17709== Copyright (C) 2000-2006, and GNU GPL'd, by Julian Seward et al. ==17709== For more details, rerun with: -v ==17709== ==17709== ==17709== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 78 from 1) ==17709== malloc/free: in use at exit: 3,013,615 bytes in 35,591 blocks. ==17709== malloc/free: 45,091 allocs, 9,500 frees, 3,698,764 bytes allocated. ==17709== For counts of detected errors, rerun with: -v ==17709== searching for pointers to 35,591 not-freed blocks. ==17709== checked 2,451,476 bytes. ==17709== ==17709== 5 bytes in 1 blocks are definitely lost in loss record 1 of 36 ==17709== at 0x40053C0: malloc (vg_replace_malloc.c:149) ==17709== by 0x561B2E: Perl_safesysmalloc (in /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so) ==17709== by 0x5647E7: Perl_savepvn (in /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so) ==17709== by 0x51CE15: Perl_gv_fetchpv (in /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so) ==17709== by 0x51481F: (within /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so) ==17709== by 0x518608: perl_parse (in /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so) ==17709== by 0x435E393: perl_instantiate (rlm_perl.c:463) ==17709== by 0x805E8EB: find_module_instance (modules.c:501) ==17709== by 0x805FC2E: do_compile_modsingle (modcall.c:1777) ==17709== by 0x806096E: do_compile_modgroup (modcall.c:1997) ==17709== by 0x805DA21: load_component_section (modules.c:666) ==17709== by 0x805DEC4: load_byserver (modules.c:987) ==17709== ==17709== LEAK SUMMARY: ==17709== definitely lost: 5 bytes in 1 blocks. ==17709== possibly lost: 0 bytes in 0 blocks. ==17709== still reachable: 3,013,610 bytes in 35,590 blocks. ==17709== suppressed: 0 bytes in 0 blocks. ==17709== Reachable blocks (those to which a pointer was found) are not shown. ==17709== To see them, rerun with: --show-reachable=yes I hope those information helps. We also have two freeradius servers in our remote datacenter still running 2.1.2 without any issue but I don't see a point rolling back to the older version so hopefully we can fix this issue with the current version if possible. Cheers, Roy Kartadinata
Roy Kartadinata wrote:
1. Our radius servers are having some memory issue where its memory usage would increase by 1% every 30 - 45 minutes until it crashes and restarts.
2. I created a script that will temporarily fix the issue by monitoring its memory usage and restart after it reaches the threshold.
You can also use "monit". The latest version from SVN has a RADIUS plugin.
3. Changed detail log configuration so that it creates a new one every hour but memory usage still increasing but at a slower rate.
Hmm... I'm not sure why that would change anything.
4. Upgraded to 2.1.6 from 2.1.5 but the problem still exist. I recently tested the latest 2.1.8 as well but still have the same issue.
That's not nice.
We also have a simple perl module script that takes IMSI passes by our PDSN and checks it against the username. I did ran valgrind yesterday and the output is below: ... ==17709== 5 bytes in 1 blocks are definitely lost in loss record 1 of 36
That shouldn't be anything to worry about. i.e. the problem is likely elsewhere.
==17709== LEAK SUMMARY: ==17709== definitely lost: 5 bytes in 1 blocks. ==17709== possibly lost: 0 bytes in 0 blocks. ==17709== still reachable: 3,013,610 bytes in 35,590 blocks.
Run "radiusd -m" to have it clean up memory on exit.
I hope those information helps. We also have two freeradius servers in our remote datacenter still running 2.1.2 without any issue but I don't see a point rolling back to the older version so hopefully we can fix this issue with the current version if possible.
If 2.1.2 doesn't have the issue, then it should be straightforward to track the problem down. Find a simple configuration that reproduces the problem (ideally one that can be run on a stand-alone system), and post it. The "git bisect" command will track the problem down very quicjly.
participants (2)
-
Alan DeKok -
Roy Kartadinata