The radiusd keeps closing and i have to restart it. I am running 2.1.11 on this server but i have recently upgraded to this and it was happening before. The program uses up all the memory on the computer which is 4GB and I have enclosed a picture of this. I can send what ever config files you need to help me discover this problem. We are using a MySql database to keep track of customer accounts and usage. free -m total used free shared buffers cached Mem: 3291 1462 1829 0 123 331 -/+ buffers/cache: 1006 2284 Swap: 6427 0 6427 -- John Babineaux Network Admin Radio Communications Service Crowley, Louisiana
john wrote:
The radiusd keeps closing and i have to restart it. I am running 2.1.11 on this server but i have recently upgraded to this and it was happening before. The program uses up all the memory on the computer which is 4GB and I have enclosed a picture of this. I can send what ever config files you need to help me discover this problem. We are using a MySql database to keep track of customer accounts and usage.
Try the v2.1.x branch from http://git.freeradius.org That will become 2.1.12 soon. Alan DeKok.
Alan DeKok <aland@deployingradius.com> wrote:
The radiusd keeps closing and i have to restart it. I am running 2.1.11 on this server but i have recently upgraded to this and it was happening before.
The program uses up all the memory on the computer which is 4GB and I have enclosed a picture of this. I can send what ever config files you need to help me discover this problem. We are using a MySql database to keep track of customer accounts and usage.
Try the v2.1.x branch from http://git.freeradius.org
That will become 2.1.12 soon.
Well...been running v2.1.x (8e71524f) for a while now and not seen any problems. I was running 2.1.11 on the other node and that still died with "unable to insert event", interestingly again just after a Status-Server message was received. RAM, CPU, IO usage at the time and for the whole time since FreeRADIUS was started was all normal (from my torrus[1] graphs). Will keep you posted if anything crops up...touch wood it seems okay. Cheers [1] http://torrus.org/ is amazing, especially combined with snmpd on hosts too -- Alexander Clouter .sigmonster says: HOST SYSTEM RESPONDING, PROBABLY UP...
Alexander Clouter wrote:
Well...been running v2.1.x (8e71524f) for a while now and not seen any problems.
OK. I'll release 2.1.12 this week, then.
I was running 2.1.11 on the other node and that still died with "unable to insert event", interestingly again just after a Status-Server message was received. RAM, CPU, IO usage at the time and for the whole time since FreeRADIUS was started was all normal (from my torrus[1] graphs).
Weird...
[1] http://torrus.org/ is amazing, especially combined with snmpd on hosts too
It looks very nice. Alan DeKok.
Alan DeKok <aland@deployingradius.com> wrote:
[1] http://torrus.org/ is amazing, especially combined with snmpd on hosts too
It looks very nice.
The nice bit is that it is trivial to configure. All you do is load up a command with the IP addresses you want to poke along with the SNMP communitities and you quickly have five minutely graphs for *every* port on your network; and various server with SNMPd running. Simples -- Alexander Clouter .sigmonster says: "Apathy is not the problem, it's the solution"
john <john@rcsaccess.net> wrote:
The radiusd keeps closing and i have to restart it. I am running 2.1.11 on this server but i have recently upgraded to this and it was happening before.
The program uses up all the memory on the computer which is 4GB and I have enclosed a picture of this. I can send what ever config files you need to help me discover this problem. We are using a MySql database to keep track of customer accounts and usage.
IIRC the fix you need it: https://github.com/alandekok/freeradius-server/commit/731e733b I recommend, if possible, to just use the v2.1.x tree which is 'stable' just not a marked official release. http://git.freeradius.org/ I am though currently trying to pin down a bug where FreeRADIUS just closes it's-self down for no reason at all. I have run tcpdump during the clean shutdown, and see it is not malformed traffic causing the problem, RAM usage is normal, open FD's is sane, etc etc. Caught the event many times with gdb, but it's not a SIG, just a regular exit(). Currently now running FreeRADIUS in production with -X to see if there is anything in the full debug logs... Just a warning, but I would imagine there would be other grumblings on the list (or I have missed them and it's already fixed...). Cheers -- Alexander Clouter .sigmonster says: I can't stand squealers; hit that guy. -- Albert Anastasia
Alexander Clouter <alex@digriz.org.uk> wrote:
I am though currently trying to pin down a bug where FreeRADIUS just closes it's-self down for no reason at all. I have run tcpdump during the clean shutdown, and see it is not malformed traffic causing the problem, RAM usage is normal, open FD's is sane, etc etc. Caught the event many times with gdb, but it's not a SIG, just a regular exit().
Currently now running FreeRADIUS in production with -X to see if there is anything in the full debug logs...
Caught it! ---- [snipped] rlm_sql (sql.dot1x): Reserving sql socket id: 1 rlm_sql_postgresql: Status: PGRES_COMMAND_OK rlm_sql_postgresql: query affected rows = 1 rlm_sql (sql.dot1x): Released sql socket id: 1 ++[sql.dot1x] returns ok ++? if (invalid) ? Evaluating (invalid) -> FALSE ++? if (invalid) -> FALSE ++? if (failed) ? Evaluating (failed) -> TRUE ++? if (failed) -> TRUE ++- entering if (failed) {...} +++? if (Acct-Status-Type == Stop && (!(Acct-Session-Time) || Acct-Session-Time == 0) && Packet-Transmit-Counter > 5) ? Evaluating (Acct-Status-Type == Stop ) -> FALSE ??? Skipping (Acct-Session-Time) ?? Skipping (Acct-Session-Time == 0) ? Skipping (Packet-Transmit-Counter > 5) +++? if (Acct-Status-Type == Stop && (!(Acct-Session-Time) || Acct-Session-Time == 0) && Packet-Transmit-Counter > 5) -> FALSE ++- if (failed) returns ok } # server dot1x.decoupled-accounting Finished request 10642. Cleaning up request 10642 ID 25817 with timestamp +5748 Going to the next request Detail listener /var/log/freeradius/radacct/journal/dot1x/detail.acct.* state replied signalled 0 waiting 0.214551 sec Waking up in 0.1 seconds. Waking up in 0.1 seconds. rad_recv: Status-Server packet from host 127.0.0.1 port 50412, id=38, length=38 [event.c:3002] Failed to insert event ---- There seem to be a bunch of malloc()'s where it could fail lurking behind INSERT_EVENT(). I am pretty sure that the system is not running out of RAM (it is a 512MB box) but I am now priming up snmpd and RRD to track this over time. Any ideas? Cheers -- Alexander Clouter .sigmonster says: The faster we go, the rounder we get. -- The Grateful Dead
On Thu, Jul 28, 2011 at 4:42 PM, Alexander Clouter <alex@digriz.org.uk> wrote:
rad_recv: Status-Server packet from host 127.0.0.1 port 50412, id=38, length=38 [event.c:3002] Failed to insert event ----
There seem to be a bunch of malloc()'s where it could fail lurking behind INSERT_EVENT(). I am pretty sure that the system is not running out of RAM (it is a 512MB box) but I am now priming up snmpd and RRD to track this over time.
Any ideas?
What happens when you send Status-Server packet manually (see "man radclient" for example)? Does the failure happen? -- Fajar
Fajar A. Nugraha <list@fajar.net> wrote:
On Thu, Jul 28, 2011 at 4:42 PM, Alexander Clouter <alex@digriz.org.uk> wrote:
rad_recv: Status-Server packet from host 127.0.0.1 port 50412, id=38, length=38 [event.c:3002] Failed to insert event ----
There seem to be a bunch of malloc()'s where it could fail lurking behind INSERT_EVENT(). I am pretty sure that the system is not running out of RAM (it is a 512MB box) but I am now priming up snmpd and RRD to track this over time.
Any ideas?
What happens when you send Status-Server packet manually (see "man radclient" for example)? Does the failure happen?
Status-Server is sent from localhost once a second as part of the failover system I use: http://www.digriz.org.uk/ha-ospf-anycast The script used is: http://www.digriz.org.uk/ha-ospf-anycast?action=AttachFile&do=get&target=rad... I do not think it is related to it as sometimes days can pass between the daemon exiting and on this occasion it was only an hour or two. If it was related to the cumulative number of requests being processed, I would expect a roughly regular 'death' interval. I do not think it is load related either as we have had the system die at all hours of the day. Cheers -- Alexander Clouter .sigmonster says: Unix soit qui mal y pense [Unix to him who evil thinks?]
Alexander Clouter wrote:
rad_recv: Status-Server packet from host 127.0.0.1 port 50412, id=38, length=38 [event.c:3002] Failed to insert event
Ouch.
There seem to be a bunch of malloc()'s where it could fail lurking behind INSERT_EVENT(). I am pretty sure that the system is not running out of RAM (it is a 512MB box) but I am now priming up snmpd and RRD to track this over time.
Any ideas?
Hmm... 512MB isn't a lot for a modern system. And on Linux, malloc() never fails. The other alternative is some kind of internal API problem. But those should all be fixed in git head. Alan DeKok.
Alan DeKok <aland@deployingradius.com> wrote:
Alexander Clouter wrote:
rad_recv: Status-Server packet from host 127.0.0.1 port 50412, id=38, length=38 [event.c:3002] Failed to insert event
Ouch.
Indeed. It did only start to happen once I upgraded to 2.1.11 from 2.1.10. Of course I was originally plagued by the OP's problem of the memory leak when using git v2.1.x between these releases; which might have hidden this particular problem. Before 2.1.11, FreeRADIUS ran fine for weeks.
There seem to be a bunch of malloc()'s where it could fail lurking behind INSERT_EVENT(). I am pretty sure that the system is not running out of RAM (it is a 512MB box) but I am now priming up snmpd and RRD to track this over time.
Any ideas?
Hmm... 512MB isn't a lot for a modern system. And on Linux, malloc() never fails.
...plenty though. Over nearly 12 hours of use, RAM usaged for FreeRADIUS is still at 15MB for one of my nodes and the other is 17MB. Linux is using the 280MB for filesystem cache and still has 180MB free!
The other alternative is some kind of internal API problem. But those should all be fixed in git head.
If you think there is something relevent in v2.1.x since 2.1.11 then I'll give it a go sooner rather than later. Can you think of something that might not be system RAM related but maybe caused by another possible RAM limit, heap, stack whatever it is (not quite my forte)? Cheers -- Alexander Clouter .sigmonster says: Money may buy friendship but money cannot buy love.
Alexander Clouter wrote:
Indeed. It did only start to happen once I upgraded to 2.1.11 from 2.1.10. Of course I was originally plagued by the OP's problem of the memory leak when using git v2.1.x between these releases; which might have hidden this particular problem.
Before 2.1.11, FreeRADIUS ran fine for weeks.
Hmm... looking at the changes, I don't see much that could have caused this. I'll see if I can come up with a patch to help narrow it down.
If you think there is something relevent in v2.1.x since 2.1.11 then I'll give it a go sooner rather than later.
Can you think of something that might not be system RAM related but maybe caused by another possible RAM limit, heap, stack whatever it is (not quite my forte)?
If the crash is consistently in the same place, then it's a coding bug, not an overflow. Alan DeKok.
participants (4)
-
Alan DeKok -
Alexander Clouter -
Fajar A. Nugraha -
john