radrelay (mode) from cvs head
Hi Guys I am testing radrelay from current cvs on a production server in the hope it will smooth out the traffic "bumps" however it seems to make them worse due to the way ot reads and sends huge chunks of packets to the backend (sql) at the same time. What happens when the following is logged? Is the packet deleted from the spool and dropped completely or is it sent again a little later? Wed Oct 4 10:05:37 2006 : Error: Dropping request (1024 is too many): from client UNKNOWN-CLIENT port 27272 - ID: 161 Wed Oct 4 10:05:37 2006 : Info: WARNING: Please check the radrelay.conf file. The value for 'max_requests' is probably set too low. I have set max_outstanding = 1 in an attempt to limit the number is requests but it doesn't seem to work. Basically I dont care is accounting requests are a few seconds late but they should always get to the sql server, and having > 1024 SQL sockets is unrealistic (Currently running with 50 but for this type of queued situation the fastest config SHOULD be 1-5 sockets depending on number of cpus and disks in the DB server). I am also running radrelay in single threaded mode in an attempt to slow it down. (That and the fact it crashes with "Error: Assertion failed in listen.c, line 620" when running threaded) Cheers -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc
Peter Nixon <listuser@peternixon.net> wrote:
What happens when the following is logged? Is the packet deleted from the spool and dropped completely or is it sent again a little later?
It's dropped.
I have set max_outstanding = 1 in an attempt to limit the number is requests but it doesn't seem to work.
Hmm... that *should* make a difference. And accounting packets shouldn't affect mac_requests, because they're deleted from the request list as soon as the reply is sent.
I am also running radrelay in single threaded mode in an attempt to slow it down. (That and the fact it crashes with "Error: Assertion failed in listen.c, line 620" when running threaded)
Arg. Where's it being called from? Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
On Wed 04 Oct 2006 18:04, Alan DeKok wrote:
Peter Nixon <listuser@peternixon.net> wrote:
What happens when the following is logged? Is the packet deleted from the spool and dropped completely or is it sent again a little later?
It's dropped.
Hmm. Thats bad :-(
I have set max_outstanding = 1 in an attempt to limit the number is requests but it doesn't seem to work.
Hmm... that *should* make a difference.
I am testing this further and will let you know.
And accounting packets shouldn't affect mac_requests, because they're deleted from the request list as soon as the reply is sent.
I am also running radrelay in single threaded mode in an attempt to slow it down. (That and the fact it crashes with "Error: Assertion failed in listen.c, line 620" when running threaded)
Arg. Where's it being called from?
When setting "proxy_requests = no" the radrelay crash in threaded mode goes away (At least hasn't occured for the last 2 hours, and it was occuring almost instantaneously before) Do you need a core or something or is that enough info? It would seem to be the same bug that is bitting us in radiusd when running proxied. Cheers -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc
Peter Nixon <listuser@peternixon.net> wrote:
It's dropped.
Hmm. Thats bad :-(
That's what configurable_failover is for. Not in SQL? Drop back into a "detail" file, and process it again.
When setting "proxy_requests = no" the radrelay crash in threaded mode goes away (At least hasn't occured for the last 2 hours, and it was occuring almost instantaneously before)
Hmm... I don't see that at all... And I *do* see it paying attention to "max_outstanding".
Do you need a core or something or is that enough info? It would seem to be the same bug that is bitting us in radiusd when running proxied.
Output from gdb & the core would help. It does appear to be the same kind of bug. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
On Wed 04 Oct 2006 18:04, Alan DeKok wrote:
Peter Nixon <listuser@peternixon.net> wrote:
What happens when the following is logged? Is the packet deleted from the spool and dropped completely or is it sent again a little later?
It's dropped.
I have set max_outstanding = 1 in an attempt to limit the number is requests but it doesn't seem to work.
Hmm... that *should* make a difference.
And accounting packets shouldn't affect mac_requests, because they're deleted from the request list as soon as the reply is sent.
* We are only relaying accounting to sql. * max_outstanding = 1 * We have turned off the proxy module which stopped the crash in multithreaded mode. In multi threaded mode we constantly get: Mon Oct 9 15:48:46 2006 : Info: rlm_sql (sql-acct): There are no DB handles to use! skipped 0, tried to connect 0 Mon Oct 9 15:48:46 2006 : Info: rlm_sql (sql-acct): There are no DB handles to use! skipped 0, tried to connect 0 Mon Oct 9 15:48:46 2006 : Info: rlm_sql (sql-acct): There are no DB handles to use! skipped 0, tried to connect 0 I have tried setting: thread pool { start_servers = 1 max_servers = 1 min_spare_servers = 1 max_spare_servers = 1 } This gives: Mon Oct 9 15:52:43 2006 : Info: The maximum number of threads (1) are active, cannot spawn new thread to handle request Mon Oct 9 15:52:43 2006 : Info: The maximum number of threads (1) are active, cannot spawn new thread to handle request Mon Oct 9 15:52:43 2006 : Info: The maximum number of threads (1) are active, cannot spawn new thread to handle request Mon Oct 9 15:52:43 2006 : Info: The maximum number of threads (1) are active, cannot spawn new thread to handle request I cannot figure out any way to make radrelay do anything close to sane behaviour except for when its running in single theaded mode (-s command line switch) Is there a config option I am missing? If not we need to change this behaviour as it is clearly insane to assume that a backend (even detail backend, let alone sql) can WRITE to disk as fast as radrelay can READ from disk.. For now I am leaving my startup script (and the one in cvs) set to single threaded mode... Which may not actually be a stupid as it sounds.. At lease where only one sql backend is in use. Cheers -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc
On Oct 9, 2006, at 8:01 AM, Peter Nixon wrote:
In multi threaded mode we constantly get:
Mon Oct 9 15:48:46 2006 : Info: rlm_sql (sql-acct): There are no DB handles to use! skipped 0, tried to connect 0 Mon Oct 9 15:48:46 2006 : Info: rlm_sql (sql-acct): There are no DB handles to use! skipped 0, tried to connect 0 Mon Oct 9 15:48:46 2006 : Info: rlm_sql (sql-acct): There are no DB handles to use! skipped 0, tried to connect 0
I just recently started switching from a MySQL backend to a PostgreSQL backend, and I'm getting these as well now. Something, IMO, seems to be up with the PGSQL module code as I don't recall having this problem with the MySQL backend. The symptoms at the moment are the server will leak memory like crazy, and I'll start getting the above when enough memory has been leaked that the disk starts going crazy swapping pages in and out, causing the PGSQL processes to stay in an iowait state until they can get a shot at the disk. I have 2 SQL modules, one for autz, which is running against a Slony slave/replica database that FR only ever performs SELECTs on, and ane for accounting. I start seeing the above against my autz database. I upped the number of SQL sockets for that module to 10, but it didn't really do anything since the real problem seems to be the leak, and running out of sockets is only really a symptom brought on by disk activity. MySQL only ever had the default 5 sockets, and it had no problem keeping up. The tables are, of course, properly indexed and tested for query speed. Like I said, my current suspicion is something in the rlm_sql_postgresql module code, maybe a per-query leak, but I'm not sure. Oh, and this is all with the CVS HEAD. --Mike
On Mon 09 Oct 2006 16:44, Michael Griego wrote:
On Oct 9, 2006, at 8:01 AM, Peter Nixon wrote:
In multi threaded mode we constantly get:
Mon Oct 9 15:48:46 2006 : Info: rlm_sql (sql-acct): There are no DB handles to use! skipped 0, tried to connect 0 Mon Oct 9 15:48:46 2006 : Info: rlm_sql (sql-acct): There are no DB handles to use! skipped 0, tried to connect 0 Mon Oct 9 15:48:46 2006 : Info: rlm_sql (sql-acct): There are no DB handles to use! skipped 0, tried to connect 0
I just recently started switching from a MySQL backend to a PostgreSQL backend, and I'm getting these as well now. Something, IMO, seems to be up with the PGSQL module code as I don't recall having this problem with the MySQL backend. The symptoms at the moment are the server will leak memory like crazy, and I'll start getting the above when enough memory has been leaked that the disk starts going crazy swapping pages in and out, causing the PGSQL processes to stay in an iowait state until they can get a shot at the disk.
Hmm. We don't have the swapping etc but maybe thats because we are keeping too close an eye on it. When radiusd restarts it consumes around 4M or ram but a check of my primary server shows it using 59M at present but still happily authing users (from postgres) and writing detail files. I quick restart dropped it back down again. On the other hand radrelay always seems to stop reading the file and just now it was using 29M or ram, but with a 35M unprocessed spool file. A restart causes it to process the file as expected but by the time it finished the spool it was back up to 24M of ram (inside a minute) so there definately seems to be a per request memory leak. Both are running single threaded.
I have 2 SQL modules, one for autz, which is running against a Slony slave/replica database that FR only ever performs SELECTs on, and ane for accounting.
As I state above I see it for both autz (radiusd) AND accounting (radrelay)
I start seeing the above against my autz database. I upped the number of SQL sockets for that module to 10, but it didn't really do anything since the real problem seems to be the leak, and running out of sockets is only really a symptom brought on by disk activity.
I run 50 sockets per instance.
MySQL only ever had the default 5 sockets, and it had no problem keeping up. The tables are, of course, properly indexed and tested for query speed.
Like I said, my current suspicion is something in the rlm_sql_postgresql module code, maybe a per-query leak, but I'm not sure.
I have not tested with any other database type as yet so I haven't confirmed your postgres suspision although I will try to do that today.
Oh, and this is all with the CVS HEAD.
Sure :-) -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc
participants (3)
-
Alan DeKok -
Michael Griego -
Peter Nixon