Very critical: Memory leak in freeradius-1.1.6
Hi, I am seeing a very serious memory leak issue with freeradius-1.1.6. The memory usage of freeradius gone from 3386Byte to 64MB when i was trying to connect 16 clients with roaming interval of 1 second. More Access-Requests are coming and we keep saving those requests until cleanup_delay. After my initial investigation in the souce code, we keep cleaning the requests only if the select() fails, which means no more request to handle. But in my case the clients are keep sending the request i think the cached request_list is not cleared properly or misses out some requests or something happens i guess. I may be wrong too, so please let me know what could be the root cause. Awaiting for any earliest reply. Thanks, Nikitha
nikitha george wrote:
I am seeing a very serious memory leak issue with freeradius-1.1.6. The memory usage of freeradius gone from 3386Byte to 64MB when i was trying to connect 16 clients with roaming interval of 1 second. More Access-Requests are coming and we keep saving those requests until cleanup_delay.
Yes... that's how the server is supposed to work.
After my initial investigation in the souce code, we keep cleaning the requests only if the select() fails,
No. It cleans the requests after it's examined all of the sockets for data.
which means no more request to handle. But in my case the clients are keep sending the request i think the cached request_list is not cleared properly or misses out some requests or something happens i guess. I may be wrong too, so please let me know what could be the root cause.
No. People are running the server with sustain loads of 100's of packets per second. There is no memory leak like you say. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
So why is it happening in my case then? I can see all the requests gets cleaned up ( log message was put) but still so much memory is consumed by radiusd. You want me post the huge log file..? I badly need this fix now.. Configuration wise i am using the default configuration except users and clients entry added newly. Hoping to get some positive reply. Thanks, Nikitha On 5/18/07, Alan DeKok <aland@deployingradius.com> wrote:
nikitha george wrote:
I am seeing a very serious memory leak issue with freeradius-1.1.6. The memory usage of freeradius gone from 3386Byte to 64MB when i was trying to connect 16 clients with roaming interval of 1 second. More Access-Requests are coming and we keep saving those requests until cleanup_delay.
Yes... that's how the server is supposed to work.
After my initial investigation in the souce code, we keep cleaning the requests only if the select() fails,
No. It cleans the requests after it's examined all of the sockets for data.
which means no more request to handle. But in my case the clients are keep sending the request i think the cached request_list is not cleared properly or misses out some requests or something happens i guess. I may be wrong too, so please let me know what could be the root cause.
No. People are running the server with sustain loads of 100's of packets per second. There is no memory leak like you say.
Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
nikitha george wrote:
So why is it happening in my case then? I can see all the requests gets cleaned up ( log message was put) but still so much memory is consumed by radiusd.
When the server caches the requests, it uses memory to do that. When it frees the requests, the memory does *not* get returned to the system. This is usually due to the "malloc/free" implementation on your system, and cannot be controlled by the server.
You want me post the huge log file..? I badly need this fix now.. Configuration wise i am using the default configuration except users and clients entry added newly.
If your server really is that busy, then it needs that much memory to ensure it maintains quality service. If you make it use less memory, requests will be lost, and your network will suffer. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Do you have any idea what kind of malloc/free implementation on our system will work out in this scenario? Thanks, Nikitha On 5/19/07, Alan DeKok <aland@deployingradius.com> wrote:
nikitha george wrote:
So why is it happening in my case then? I can see all the requests gets cleaned up ( log message was put) but still so much memory is consumed by radiusd.
When the server caches the requests, it uses memory to do that. When it frees the requests, the memory does *not* get returned to the system. This is usually due to the "malloc/free" implementation on your system, and cannot be controlled by the server.
You want me post the huge log file..? I badly need this fix now.. Configuration wise i am using the default configuration except users and clients entry added newly.
If your server really is that busy, then it needs that much memory to ensure it maintains quality service. If you make it use less memory, requests will be lost, and your network will suffer.
Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 nikitha george wrote:
So why is it happening in my case then? I can see all the requests gets cleaned up ( log message was put) but still so much memory is consumed by radiusd.
Memory usage under Linux is a tricky thing. It depends if you're looking at RSS (RES in top), which is the actual RAM used by the process, or virtual memory (VIRT in top) which actually may map directly to disk and/or to RAM, and/or be shared with another process! A typical usage of a process the RES will rise, and then hit a plateau. Check the memory usage after a day, it should be the same the following week. If the RES raise linearly for days until the system trashes, you may have detected a memory leak. But to make sure a leak happens, usually tools like Valgrind helps better than looking at what the kernel reports, as there is a lot more hidden than what shows. - -- ============== +---------------------------------------------+ Martin Gadbois | "Please answer by yes or no. | Sr. SW Designer | Uncooperative user waste precious CPU time" | Colubris Networks Inc. | -- The Andromeda Strain, M. Crichton, 1969 | -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGThWW9Y3/iTTCEDkRAnOMAJ47LwlA9o48V/49KnahEtaaNWf+4QCfXTL1 HAza0Po1crhG3tHzf6OQIcY= =q2Gp -----END PGP SIGNATURE-----
Hi Martin, Thanks for your reply. The memory usage of radiusd in our system went from 6687KB to 160MB which is 50.1% of memory of our system and radiusd got re-started. I had run the Valgrind to see the resource leak. Definitely lost memory was 380 Bytes but still reachable are more. I think this is the one which is leaking memory. I dont have access to the log files today, i will post it by tomorrow for your reference. Thanks, Nikitha On 5/19/07, Martin Gadbois <martin.gadbois@colubris.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
nikitha george wrote:
So why is it happening in my case then? I can see all the requests gets cleaned up ( log message was put) but still so much memory is consumed by radiusd.
Memory usage under Linux is a tricky thing. It depends if you're looking at RSS (RES in top), which is the actual RAM used by the process, or virtual memory (VIRT in top) which actually may map directly to disk and/or to RAM, and/or be shared with another process!
A typical usage of a process the RES will rise, and then hit a plateau. Check the memory usage after a day, it should be the same the following week.
If the RES raise linearly for days until the system trashes, you may have detected a memory leak.
But to make sure a leak happens, usually tools like Valgrind helps better than looking at what the kernel reports, as there is a lot more hidden than what shows.
- -- ============== +---------------------------------------------+ Martin Gadbois | "Please answer by yes or no. | Sr. SW Designer | Uncooperative user waste precious CPU time" | Colubris Networks Inc. | -- The Andromeda Strain, M. Crichton, 1969 | -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFGThWW9Y3/iTTCEDkRAnOMAJ47LwlA9o48V/49KnahEtaaNWf+4QCfXTL1 HAza0Po1crhG3tHzf6OQIcY= =q2Gp -----END PGP SIGNATURE----- - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
nikitha george wrote:
Hi Martin,
Thanks for your reply. The memory usage of radiusd in our system went from 6687KB to 160MB which is 50.1% of memory of our system and radiusd got re-started.
I had run the Valgrind to see the resource leak. Definitely lost memory was 380 Bytes but still reachable are more. I think this is the one which is leaking memory. I dont have access to the log files today, i will post it by tomorrow for your reference.
You seem pretty sure you've got a memory leak. As Alan has said, we're pretty sure there are no major leaks in the server core, so logically it's either your system libraries or an infrequently-used bit of the server. We need to know/see the following: * What OS are you on? Versions of the OS, kernel, major libraries (libc, threading) and a rough idea of how many clients and requests/sec you're getting * your radiusd.conf * a debug output of a *few* (not thousands!) of requests from your server - run "radiusd -X" A couple of wild guesses: Are you using PAM? Are you using rlm_perl or rlm_python? Are you using a database that's a bit too slow? I've you've actually run the "valgrind --memcheck --leak-check=yes" then let's take a look at that too.
nikitha george wrote:
The memory usage of radiusd in our system went from 6687KB to 160MB which is 50.1% of memory of our system and radiusd got re-started.
160Mb does sound like a lot.
I had run the Valgrind to see the resource leak. Definitely lost memory was 380 Bytes but still reachable are more. I think this is the one which is leaking memory. I dont have access to the log files today, i will post it by tomorrow for your reference.
The 380 bytes "definitely lost" is from libltdl. It contributed almost nothing to the 160M total memory used by the server. If valgrind doesn't say that the memory is lost, then the memory is very likely still being used. i.e. It's likely needed for something. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
On 5/20/07, Alan DeKok <aland@deployingradius.com> wrote:
If valgrind doesn't say that the memory is lost, then the memory is very likely still being used. i.e. It's likely needed for something.
something..? Why do radiusd need to hold the memory even after cleaning up the request? Even after stopping the request, the memory never gets freed. Why is this strange result of huge memory usage by radiusd? On 5/20/07, Alan DeKok <aland@deployingradius.com> wrote:
nikitha george wrote:
The memory usage of radiusd in our system went from 6687KB to 160MB which is 50.1% of memory of our system and radiusd got re-started.
160Mb does sound like a lot.
I had run the Valgrind to see the resource leak. Definitely lost memory was 380 Bytes but still reachable are more. I think this is the one which is leaking memory. I dont have access to the log files today, i will post it by tomorrow for your reference.
The 380 bytes "definitely lost" is from libltdl. It contributed almost nothing to the 160M total memory used by the server.
If valgrind doesn't say that the memory is lost, then the memory is very likely still being used. i.e. It's likely needed for something.
Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
nikitha george wrote:
On 5/20/07, Alan DeKok <aland@deployingradius.com> wrote:
If valgrind doesn't say that the memory is lost, then the memory is very likely still being used. i.e. It's likely needed for something.
something..?
Did you read my earlier explanations?
Why do radiusd need to hold the memory even after cleaning up the request?
Did you read my earlier message explaining this?
Even after stopping the request, the memory never gets freed. Why is this strange result of huge memory usage by radiusd?
Did you read my earlier message explaining this? Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
participants (5)
-
Alan DeKok -
Alan Dekok
-
Martin Gadbois -
nikitha george -
Phil Mayers