Major impact on authentication!
Hi, Freeradius 1.1.4 is randomly losing connection to both databases and it's causing total loss in the authentication process: Info: rlm_sql (sql): There are no DB handles to use! skipped 0, tried to connect 0 Info: rlm_sql (sql_postgresql): There are no DB handles to use! skipped 0, tried to connect 0 Info: The maximum number of threads (32) are active, cannot spawn new thread to handle request Running either in multi or single threaded mode, that messages appeared 47.099,00 times since Jan 27! Freeradius is configured with 32 max_servers and 32 connections to each DB. There's no starving since no accounting is being used and the server have to handle just 3 auths per second. Every time this happens, no one can authenticate and doing a restart in Freeradius solves the problem. To circumvent the problem, I've added a cron.hourly job so each hour a "service radiusd restart" is issued. As this is random, it's hard to debug, but at the same time freeradius loses the connection, several other applications can successfully connect/ maintain previous established connections to the databases. I've enabled all sorts of debug in the databases trying to better understand why freeradius is doing this, but there was no luck. I've installed the latest CVS and the same problem appeared, please help!
Guilherme Franco wrote:
Freeradius 1.1.4 is randomly losing connection to both databases and it's causing total loss in the authentication process:
The messages you posted show it's NOT losing connection to the database.
Info: rlm_sql (sql): There are no DB handles to use! skipped 0, tried to connect 0 Info: rlm_sql (sql_postgresql): There are no DB handles to use! skipped 0, tried to connect 0 Info: The maximum number of threads (32) are active, cannot spawn new thread to handle request
This means that ALL threads are blocked trying to access the DB, because ALL DB handles are in use. i.e. the DB is slow, and is not responding to FreeRADIUS. Fix the DB, fix the problem.
Running either in multi or single threaded mode, that messages appeared 47.099,00 times since Jan 27! Freeradius is configured with 32 max_servers and 32 connections to each DB. There's no starving since no accounting is being used and the server have to handle just 3 auths per second.
And how long does it take the server to respond to a request? If it's more than 1/3 of a second, it will quickly reach the state you see.
As this is random, it's hard to debug, but at the same time freeradius loses the connection, several other applications can successfully connect/ maintain previous established connections to the databases.
FreeRADIUS is NOT losing its connection to the DB. If you think that's happening, you will try to fix a problem that doesn't exist, and will NOT solve the real problem.
I've enabled all sorts of debug in the databases trying to better understand why freeradius is doing this, but there was no luck.
Find out why the database isn't responding to FreeRADIUS. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Alan DeKok wrote:
As this is random, it's hard to debug, but at the same time freeradius loses the connection, several other applications can successfully connect/ maintain previous established connections to the databases.
FreeRADIUS is NOT losing its connection to the DB. If you think that's happening, you will try to fix a problem that doesn't exist, and will NOT solve the real problem.
I've enabled all sorts of debug in the databases trying to better understand why freeradius is doing this, but there was no luck.
Find out why the database isn't responding to FreeRADIUS.
I had similar issues at one time with MySQL and FreeRADIUS. There is an app out there for MySQL called Mytop which is basically like the unix "top" command, but looks at MySQL processes. This makes it very easy to watch and see what processes are taking too long and holding up the rest. I'm not sure if there is a similar app out there for postgresql, but it'd be worth a look. -- Dennis Skinner Systems Administrator BlueFrog Internet http://www.bluefrog.com
I'm testing PostgreSQL performance with various tools. Thanks! On 2/5/07, Dennis Skinner <dskinner@bluefrog.com> wrote:
Alan DeKok wrote:
As this is random, it's hard to debug, but at the same time freeradius loses the connection, several other applications can successfully connect/ maintain previous established connections to the databases.
FreeRADIUS is NOT losing its connection to the DB. If you think that's happening, you will try to fix a problem that doesn't exist, and will NOT solve the real problem.
I've enabled all sorts of debug in the databases trying to better understand why freeradius is doing this, but there was no luck.
Find out why the database isn't responding to FreeRADIUS.
I had similar issues at one time with MySQL and FreeRADIUS. There is an app out there for MySQL called Mytop which is basically like the unix "top" command, but looks at MySQL processes. This makes it very easy to watch and see what processes are taking too long and holding up the rest.
I'm not sure if there is a similar app out there for postgresql, but it'd be worth a look.
-- Dennis Skinner Systems Administrator BlueFrog Internet http://www.bluefrog.com - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Hi,
Freeradius 1.1.4 is randomly losing connection to both databases and it's causing total loss in the authentication process:
from a historical perspective you may find that is wasnt the 1.1.4 upgrade that has broken things - your database may have finally become too big and unwieldy. this has certainly been the case in many such cases. I would check how long your database queries/inserts are taking. perhaps vacuum/optimise the tables, move/drop older entries, create better KEYs for the purposes you need. alan
On Mon 05 Feb 2007 13:05, A.L.M.Buxey@lboro.ac.uk wrote:
Hi,
Freeradius 1.1.4 is randomly losing connection to both databases and it's causing total loss in the authentication process:
from a historical perspective you may find that is wasnt the 1.1.4 upgrade that has broken things - your database may have finally become too big and unwieldy. this has certainly been the case in many such cases. I would check how long your database queries/inserts are taking. perhaps vacuum/optimise the tables, move/drop older entries, create better KEYs for the purposes you need.
The _random_ problems don't coincide with a user running a usage report from a web interface by any chance do they?? Regards -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc
Hello, Thank you all for your prompt answers! The database takes between 15ms and 40ms to answer to freeradius and has only 40.000 entries there, so it isn't big. PostgreSQL is updated to it's latest version and vaccum runs every night. The queries are from sqlippool.conf, so... Thanks!
Guilherme Franco wrote:
The database takes between 15ms and 40ms to answer to freeradius
Sometimes it takes a LOT longer than that. Find out why. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
The database takes between 15ms and 40ms to answer
to freeradius
Sometimes it takes a LOT longer than that.
I don't konw how you measure the processing time. I don't know what your request does, but if it does one or several commit(s), is the commit time counted? I have used Oracle/TkProf (tkprof is a request profiling tool from Oracle) and the time for commit is not given. The only way to have it is to measure it as a black-box. But maybe it's what you already do. An maybe that prostgre return the commit time? Geoff. ___________________________________________________________________________ Découvrez une nouvelle façon d'obtenir des réponses à toutes vos questions ! Profitez des connaissances, des opinions et des expériences des internautes sur Yahoo! Questions/Réponses http://fr.answers.yahoo.com
Sorry Mr. Geoffroy, your message arrived only now in my e-mail. Somehow, every list message takes a large amount of time to come into my e-mail. Thank you for your concerns and yes, postgresql has debug/ analyze commands that displays the time that commit or anything else take to complete. On 2/5/07, Geoffroy Arnoud <garnoud@yahoo.co.uk> wrote:
The database takes between 15ms and 40ms to answer
to freeradius
Sometimes it takes a LOT longer than that.
I don't konw how you measure the processing time. I don't know what your request does, but if it does one or several commit(s), is the commit time counted? I have used Oracle/TkProf (tkprof is a request profiling tool from Oracle) and the time for commit is not given. The only way to have it is to measure it as a black-box.
But maybe it's what you already do. An maybe that prostgre return the commit time?
Geoff.
___________________________________________________________________________ Découvrez une nouvelle façon d'obtenir des réponses à toutes vos questions ! Profitez des connaissances, des opinions et des expériences des internautes sur Yahoo! Questions/Réponses http://fr.answers.yahoo.com - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On Mon 05 Feb 2007 23:03, Guilherme Franco wrote:
Sorry Mr. Geoffroy, your message arrived only now in my e-mail.
Somehow, every list message takes a large amount of time to come into my e-mail.
I am also have big delays on some emails from the list (Up to 5 days!) while others arrive within seconds.. -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc
participants (6)
-
A.L.M.Buxey＠lboro.ac.uk -
Alan DeKok -
Dennis Skinner -
Geoffroy Arnoud -
Guilherme Franco -
Peter Nixon