Freeradius and memory usage
Hello, Our radius server is currently having some memory issue where its memory usage would increase by 1% every 30-45 minutes. Eventually the server will crash and restart because of out of memory. We've been using freeradius for a couple of years and this just started on Thursday so I upgraded our radius to the latest 2.1.6 from 2.1.5 but the problem still there. Has anyone ever experiencing this issue before? Cheers, Roy Kartadinata
Is there anything strange in radius log files and are you sure that freeradius is the reason for your problem ? Database? web server ?.... Roy Kartadinata wrote:
Hello,
Our radius server is currently having some memory issue where its memory usage would increase by 1% every 30-45 minutes. Eventually the server will crash and restart because of out of memory. We’ve been using freeradius for a couple of years and this just started on Thursday so I upgraded our radius to the latest 2.1.6 from 2.1.5 but the problem still there. Has anyone ever experiencing this issue before?
Cheers,
Roy Kartadinata
------------------------------------------------------------------------
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Oh it's the radius from ps output and top. This server is a dedicated radius server, no web server running in it and we use external database servers. Cheers, Roy Kartadinata -----Original Message----- From: freeradius-users-bounces+rkartadinata=pocket.com@lists.freeradius.org [mailto:freeradius-users-bounces+rkartadinata=pocket.com@lists.freeradiu s.org] On Behalf Of Marinko Tarlac Sent: Thursday, July 23, 2009 2:17 PM To: FreeRadius users mailing list Subject: Re: Freeradius and memory usage Is there anything strange in radius log files and are you sure that freeradius is the reason for your problem ? Database? web server ?.... Roy Kartadinata wrote:
Hello,
Our radius server is currently having some memory issue where its memory usage would increase by 1% every 30-45 minutes. Eventually the server will crash and restart because of out of memory. We've been using freeradius for a couple of years and this just started on Thursday so I upgraded our radius to the latest 2.1.6 from 2.1.5 but the problem still there. Has anyone ever experiencing this issue
before?
Cheers,
Roy Kartadinata
------------------------------------------------------------------------
- List info/subscribe/unsubscribe? See
http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Roy Kartadinata wrote:
Our radius server is currently having some memory issue where its memory usage would increase by 1% every 30-45 minutes. Eventually the server will crash and restart because of out of memory. We’ve been using freeradius for a couple of years and this just started on Thursday so I upgraded our radius to the latest 2.1.6 from 2.1.5 but the problem still there. Has anyone ever experiencing this issue before?
The only thing like that I'm aware of is with the detail file, which was fixed in 2.1.6. Look at the logs to see what the server is doing. What errors is it producing? Alan DeKok.
Alan DeKok wrote:
Roy Kartadinata wrote:
Our radius server is currently having some memory issue where its memory usage would increase by 1% every 30-45 minutes. Eventually the server will crash and restart because of out of memory. We've been using freeradius for a couple of years and this just started on Thursday so I upgraded our radius to the latest 2.1.6 from 2.1.5 but the problem still there. Has anyone ever experiencing this issue before?
The only thing like that I'm aware of is with the detail file, which was fixed in 2.1.6.
Look at the logs to see what the server is doing. What errors is it producing?
I didn't see any error on error log, it looks clean. But this is what the log looks like when it started to run out of memory the other night: Wed Jul 22 22:03:42 2009 : Error: Rejecting request 16183416 due to lack of any response from home server xxx.xxx.xxx.xxx port 1813 Wed Jul 22 22:03:42 2009 : Error: Rejecting request 16183418 due to lack of any response from home server xxx.xxx.xxx.xxx port 1813 Wed Jul 22 22:03:42 2009 : Error: Rejecting request 16183420 due to lack of any response from home server xxx.xxx.xxx.xxx port 1813 Wed Jul 22 22:03:42 2009 : Error: WARNING: Unresponsive child for request 16183422, in module detail component pre-proxy Wed Jul 22 22:03:42 2009 : Error: Rejecting request 16183423 due to lack of any response from home server xxx.xxx.xxx.xxx port 1813 Wed Jul 22 22:03:42 2009 : Error: WARNING: Unresponsive child for request 16183425, in module component pre-proxy Wed Jul 22 22:03:42 2009 : Error: WARNING: Unresponsive child for request 16183427, in module component pre-proxy Wed Jul 22 22:03:42 2009 : Error: WARNING: Unresponsive child for request 16183428, in module component pre-proxy Wed Jul 22 22:03:42 2009 : Error: WARNING: Unresponsive child for request 16183430, in module component pre-proxy Wed Jul 22 22:03:42 2009 : Error: WARNING: Unresponsive child for request 16183432, in module component pre-proxy Wed Jul 22 22:03:42 2009 : Error: WARNING: Unresponsive child for request 16183434, in module component pre-proxy Wed Jul 22 22:03:43 2009 : Error: WARNING: Unresponsive child for request 16183436, in module component pre-proxy Wed Jul 22 22:03:43 2009 : Error: WARNING: Unresponsive child for request 16183438, in module component pre-proxy Wed Jul 22 22:03:43 2009 : Error: WARNING: Unresponsive child for request 16183440, in module component pre-proxy Wed Jul 22 22:03:43 2009 : Error: WARNING: Unresponsive child for request 16183441, in module component pre-proxy Wed Jul 22 22:03:43 2009 : Error: WARNING: Unresponsive child for request 16183443, in module component pre-proxy Wed Jul 22 22:03:43 2009 : Error: WARNING: Unresponsive child for request 16183445, in module component pre-proxy Wed Jul 22 22:03:43 2009 : Error: WARNING: Unresponsive child for request 16183447, in module component pre-proxy And radius eventually died few seconds later. During all these, the memory usage was at around 75% but authentication which we don't proxy continued to work. My current band aid was a simple script that checks its memory usage regularly and restart after it passes certain percentage, not a good solution but this will do for now. Cheers, Roy Kartadinata
Roy Kartadinata wrote:
I didn't see any error on error log, it looks clean. But this is what the log looks like when it started to run out of memory the other night:
Wed Jul 22 22:03:42 2009 : Error: Rejecting request 16183416 due to lack of any response from home server xxx.xxx.xxx.xxx port 1813 Wed Jul 22 22:03:42 2009 : Error: Rejecting request 16183418 due to lack of any response from home server xxx.xxx.xxx.xxx port 1813 Wed Jul 22 22:03:42 2009 : Error: Rejecting request 16183420 due to lack of any response from home server xxx.xxx.xxx.xxx port 1813 Wed Jul 22 22:03:42 2009 : Error: WARNING: Unresponsive child for request 16183422, in module detail component pre-proxy
Ok... so it's blocked writing to the detail file.
Wed Jul 22 22:03:42 2009 : Error: Rejecting request 16183423 due to lack of any response from home server xxx.xxx.xxx.xxx port 1813 Wed Jul 22 22:03:42 2009 : Error: WARNING: Unresponsive child for request 16183425, in module component pre-proxy
Module "" means "server core". That's weird... Is the detail file on an NFS mount? It really shouldn't block writing to the detail file unless the underlying file system goes away.
And radius eventually died few seconds later. During all these, the memory usage was at around 75% but authentication which we don't proxy continued to work.
Because it's going crazy, and dying... Alan DeKok.
Maybe detail file is to large ? I had a similar problem when server goes off because of this... Alan DeKok wrote:
Roy Kartadinata wrote:
I didn't see any error on error log, it looks clean. But this is what the log looks like when it started to run out of memory the other night:
Wed Jul 22 22:03:42 2009 : Error: Rejecting request 16183416 due to lack of any response from home server xxx.xxx.xxx.xxx port 1813 Wed Jul 22 22:03:42 2009 : Error: Rejecting request 16183418 due to lack of any response from home server xxx.xxx.xxx.xxx port 1813 Wed Jul 22 22:03:42 2009 : Error: Rejecting request 16183420 due to lack of any response from home server xxx.xxx.xxx.xxx port 1813 Wed Jul 22 22:03:42 2009 : Error: WARNING: Unresponsive child for request 16183422, in module detail component pre-proxy
Ok... so it's blocked writing to the detail file.
Wed Jul 22 22:03:42 2009 : Error: Rejecting request 16183423 due to lack of any response from home server xxx.xxx.xxx.xxx port 1813 Wed Jul 22 22:03:42 2009 : Error: WARNING: Unresponsive child for request 16183425, in module component pre-proxy
Module "" means "server core".
That's weird...
Is the detail file on an NFS mount? It really shouldn't block writing to the detail file unless the underlying file system goes away.
And radius eventually died few seconds later. During all these, the memory usage was at around 75% but authentication which we don't proxy continued to work.
Because it's going crazy, and dying...
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
An update on this issue, I was finally able to split the detail file log into hourly by adding another entry to modules/detail.log file. So far the memory usage is still increasing but in a much slower rate. Marinko Tarlac wrote:
Maybe detail file is to large ?
I had a similar problem when server goes off because of this...
How did you fix your issue?
Alan DeKok wrote:
Roy Kartadinata wrote:
I didn't see any error on error log, it looks clean. But this is what the log looks like when it started to run out of memory the other night:
Wed Jul 22 22:03:42 2009 : Error: Rejecting request 16183416 due to lack of any response from home server xxx.xxx.xxx.xxx port 1813 Wed Jul 22 22:03:42 2009 : Error: Rejecting request 16183418 due to lack of any response from home server xxx.xxx.xxx.xxx port 1813 Wed Jul 22 22:03:42 2009 : Error: Rejecting request 16183420 due to lack of any response from home server xxx.xxx.xxx.xxx port 1813 Wed Jul 22 22:03:42 2009 : Error: WARNING: Unresponsive child for request 16183422, in module detail component pre-proxy
Ok... so it's blocked writing to the detail file.
Wed Jul 22 22:03:42 2009 : Error: Rejecting request 16183423 due to lack of any response from home server xxx.xxx.xxx.xxx port 1813 Wed Jul 22 22:03:42 2009 : Error: WARNING: Unresponsive child for request 16183425, in module component pre-proxy
Module "" means "server core".
That's weird...
Is the detail file on an NFS mount? It really shouldn't block writing to the detail file unless the underlying file system goes away.
And radius eventually died few seconds later. During all these, the memory usage was at around 75% but authentication which we don't proxy continued to work.
Because it's going crazy, and dying...
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Cheers, Roy Kartadinata
Hi Well I don't use is.. Only in case I need to debug something... I have a small problem with triggering AccountingOff query (I don't know what causes this) which closes all opened sessions in database... Now I found a work around for this... You can try to increase Interim-update intervals on your NAS(es) (5 min and up). This will help and your server will take some breath :) Also it would be good if you can tell us something about your system... OS, CPU, HDD type ... Best regards Roy Kartadinata wrote:
An update on this issue, I was finally able to split the detail file log into hourly by adding another entry to modules/detail.log file. So far the memory usage is still increasing but in a much slower rate.
Marinko Tarlac wrote:
Maybe detail file is to large ?
I had a similar problem when server goes off because of this...
How did you fix your issue?
Marinko Tarlac wrote:
Hi
Well I don't use is.. Only in case I need to debug something... I have a small problem with triggering AccountingOff query (I don't know what causes this) which closes all opened sessions in database... Now I found a work around for this...
You can try to increase Interim-update intervals on your NAS(es) (5 min and up). This will help and your server will take some breath :)
Well even if I force our NASes to send all traffic to our secondary AAA, memory usage on primary server didn't go down.
Also it would be good if you can tell us something about your system... OS, CPU, HDD type ...
Server info is as follow: OS - Linux (CentOS 5.3) Freeradius - 2.1.6 CPU - single quad core Xeon Memory - 4GB HDD - FC drives on EMC
Best regards
Roy Kartadinata wrote:
An update on this issue, I was finally able to split the detail file log into hourly by adding another entry to modules/detail.log file. So far the memory usage is still increasing but in a much slower rate.
Marinko Tarlac wrote:
Maybe detail file is to large ?
I had a similar problem when server goes off because of this...
How did you fix your issue?
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Cheers, Roy Kartadinata
Roy Kartadinata wrote:
An update on this issue, I was finally able to split the detail file log into hourly by adding another entry to modules/detail.log file. So far the memory usage is still increasing but in a much slower rate.
That helps narrow it down a bit. But it's still a bit worrying. I'll see if I can find out any leaks in the code... Alan DeKok.
Alan DeKok wrote:
Roy Kartadinata wrote:
An update on this issue, I was finally able to split the detail file log into hourly by adding another entry to modules/detail.log file. So far the memory usage is still increasing but in a much slower rate.
That helps narrow it down a bit. But it's still a bit worrying.
I'll see if I can find out any leaks in the code...
Do you need any other information from me to help you with finding the leaks?
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Cheers, Roy Kartadinata
Roy Kartadinata wrote:
Do you need any other information from me to help you with finding the leaks?
Ideally, run it under valgrind, with options where it shows and tracks memory usage: $ script leak.txt $ valgrind --tool=memcheck --leak-check=full radiusd -f $ exit Let it run for a few hours, and then stop it. It should dump a report to the screen, which will get captured in "leak.txt". Then, mail it to me. If it doesn't show any leaks, then the issue is that it's tracking too many things, and not cleaning up. That's a little harder problem to solve. Alan DeKok.
Alan DeKok wrote:
Roy Kartadinata wrote:
Do you need any other information from me to help you with finding the leaks?
Ideally, run it under valgrind, with options where it shows and tracks memory usage:
$ script leak.txt $ valgrind --tool=memcheck --leak-check=full radiusd -f $ exit
Will radius still be operational (taking authentication request) while we run it under valgrind?
Let it run for a few hours, and then stop it. It should dump a report to the screen, which will get captured in "leak.txt". Then, mail it to me.
If it doesn't show any leaks, then the issue is that it's tracking too many things, and not cleaning up. That's a little harder problem to solve.
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Cheers, Roy Kartadinata Pocket Communications Phone: 210-858-2559 Fax: 210-678-8101
Alan Buxey wrote:
Hi,
Will radius still be operational (taking authentication request) while we run it under valgrind?
yes - but slower - so make sure you've got the horsepower...any time critical things might not operate as usual.
Cool, I'll just do it during our slowest hour then.
alan - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Cheers, Roy Kartadinata
Alan DeKok wrote:
Roy Kartadinata wrote:
I didn't see any error on error log, it looks clean. But this is what the log looks like when it started to run out of memory the other night:
Wed Jul 22 22:03:42 2009 : Error: Rejecting request 16183416 due to lack of any response from home server xxx.xxx.xxx.xxx port 1813 Wed Jul 22 22:03:42 2009 : Error: Rejecting request 16183418 due to lack of any response from home server xxx.xxx.xxx.xxx port 1813 Wed Jul 22 22:03:42 2009 : Error: Rejecting request 16183420 due to lack of any response from home server xxx.xxx.xxx.xxx port 1813 Wed Jul 22 22:03:42 2009 : Error: WARNING: Unresponsive child for request 16183422, in module detail component pre-proxy
Ok... so it's blocked writing to the detail file.
Wed Jul 22 22:03:42 2009 : Error: Rejecting request 16183423 due to lack of any response from home server xxx.xxx.xxx.xxx port 1813 Wed Jul 22 22:03:42 2009 : Error: WARNING: Unresponsive child for request 16183425, in module component pre-proxy
Module "" means "server core".
That's weird...
Is the detail file on an NFS mount? It really shouldn't block writing to the detail file unless the underlying file system goes away.
No, the detail file is on a local partition. I think its because of the size, but for some reason it won't use settings on modules/detail file where I separated it by hour.
And radius eventually died few seconds later. During all these, the memory usage was at around 75% but authentication which we don't proxy continued to work.
Because it's going crazy, and dying...
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Cheers, Roy Kartadinata Pocket Communications Phone: 210-858-2559 Fax: 210-678-8101
participants (4)
-
Alan Buxey -
Alan DeKok -
Marinko Tarlac -
Roy Kartadinata