memory leaks and other problems with rlm_ippool
Hi Guys I am consistently finding that if you create more than one IP Pool bigger than a class B network that FreeRADIUS fails to start with an error saying that it cannot fork. It can however be run in debug mode with this config. I have tested this on a number of different servers with a number of different versions of FreeRADIUS and on SUSE Linux Enterprise 9, SUSE Pro 9.2, SUSE Pro 9.3 and some random version of Turbo Linux that a customer had. Also, one customer who had only a single class B IP Pool with about 10,000 clients connecting and disconnecting on a regular basis (Around 300 Auth requests and associated Accounting packets per minute between 8am and midnight) is seeing their FreeRADIUS grow over the course of a couple of days until the Linux OOM killer kills it. (Making their customers very unhappy until they restart their server) Their server is fully patched SLES9 running FreeRADIUS 1.0.4 Does anyone have any ideas? Surely someone else must have seen these problems. I can't be the only person running multiple, large IP Pools... -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc
Peter Nixon wrote:
I am consistently finding that if you create more than one IP Pool bigger than a class B network that FreeRADIUS fails to start with an error saying that it cannot fork. It can however be run in debug mode with this config. I have tested this on a number of different servers with a number of different versions of FreeRADIUS and on SUSE Linux Enterprise 9, SUSE Pro 9.2, SUSE Pro 9.3 and some random version of Turbo Linux that a customer had.
I'm not using rlm_ippool, but I've seen a similar bug report on the bugzilla. http://bugs.freeradius.org/show_bug.cgi?id=238 -- Nicolas Baradakis
On Wednesday 24 August 2005 18:07, Nicolas Baradakis wrote:
Peter Nixon wrote:
I am consistently finding that if you create more than one IP Pool bigger than a class B network that FreeRADIUS fails to start with an error saying that it cannot fork. It can however be run in debug mode with this config. I have tested this on a number of different servers with a number of different versions of FreeRADIUS and on SUSE Linux Enterprise 9, SUSE Pro 9.2, SUSE Pro 9.3 and some random version of Turbo Linux that a customer had.
I'm not using rlm_ippool, but I've seen a similar bug report on the bugzilla.
Yep. Slightly different simptoms but it definitely appears to be related. -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc
Nicolas Baradakis <nbk@sitadelle.com> wrote:
I'm not using rlm_ippool, but I've seen a similar bug report on the bugzilla.
It looks like gdb is allocating large amounts amounts of data for it's cache. Maybe setting "cache_size" to a smaller value would help? Alan DeKok.
I've been seeing the same issue and have been trying to track it down for over a year. I'm not running ippools, though, but I get a large number of EAP conversations. Until I could track it down, I've got a script in place to kill/restart the daemon when it reaches 400MB. I've also been kinda watching for others seeing similar issues, and, honestly, this is the first thing I've seen that comes close. My setup is Fedora Core 1, fully patched, on a Dell PowerEdge 650 (2.4GHz P4, 512MB RAM). The daemon grows in memory usage unchecked over time even in full -X debug mode. I'd be very interested in comparing notes. I've been valgrinding the heck out of the server and can't seem to figure out where the leak is, even though I've found and fixed a couple of leaks in my own checking. Those haven't fixed the memory growth problem. I'm convinced that something somewhere has an off-by-one bug or something similar and is horking the memory manager. I have yet to be able to track this down though. --Mike Peter Nixon wrote:
Hi Guys
I am consistently finding that if you create more than one IP Pool bigger than a class B network that FreeRADIUS fails to start with an error saying that it cannot fork. It can however be run in debug mode with this config. I have tested this on a number of different servers with a number of different versions of FreeRADIUS and on SUSE Linux Enterprise 9, SUSE Pro 9.2, SUSE Pro 9.3 and some random version of Turbo Linux that a customer had.
Also, one customer who had only a single class B IP Pool with about 10,000 clients connecting and disconnecting on a regular basis (Around 300 Auth requests and associated Accounting packets per minute between 8am and midnight) is seeing their FreeRADIUS grow over the course of a couple of days until the Linux OOM killer kills it. (Making their customers very unhappy until they restart their server) Their server is fully patched SLES9 running FreeRADIUS 1.0.4
Does anyone have any ideas? Surely someone else must have seen these problems. I can't be the only person running multiple, large IP Pools...
Michael Griego <mgriego@utdallas.edu> wrote:
I'd be very interested in comparing notes. I've been valgrinding the heck out of the server and can't seem to figure out where the leak is,
It sounds like it's not a memory leak, then. i.e. the memory is still accounted for, but it *should* have been freed. I have a line on a tool that tracks these issues, and I'll see if I can post it on a web page. Alan DeKok.
Peter Nixon <listuser@peternixon.net> wrote:
I am consistently finding that if you create more than one IP Pool bigger than a class B network that FreeRADIUS fails to start with an error saying that it cannot fork. It can however be run in debug mode with this config.
Maybe the modules should be initialized *after* forking, not before.
Also, one customer who had only a single class B IP Pool with about 10,000 clients connecting and disconnecting on a regular basis (Around 300 Auth requests and associated Accounting packets per minute between 8am and midnight) is seeing their FreeRADIUS grow over the course of a couple of days until the Linux OOM killer kills it.
Maybe the DB library has a memory leak? In the short term, I would suggest HUP'ing their server every day or so, even if the config hasn't changed. That should tear down the modules & restart them, which might help. Alan DeKok.
participants (4)
-
Alan DeKok -
Michael Griego -
Nicolas Baradakis -
Peter Nixon