OK... the code to handle the detail file is now in. radiusd.conf: listen { type = detail filename = foo max_outstanding = 10 # this will be deleted later } If one thread is waiting for something to do, then ONE packet is read from the detail file, and handed to a child thread. This means that if the server is busy handling authentication and/or accounting requests, it completely ignores the detail file. As it should. I've also updated the "detail" module so that in the post-proxy section, it writes the accounting packet IF there's no reply from the home server. This lets the detail module sit in Post-Proxy-Type Fail, and Just Do The Right Thing. Please test. I've done some basic work, and it should be OK. It doesn't hit assertions, it doesn't crash, and I've run 1000's of packets through it. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
On Fri 04 May 2007, Alan DeKok wrote:
OK... the code to handle the detail file is now in.
radiusd.conf:
listen { type = detail filename = foo max_outstanding = 10 # this will be deleted later }
If one thread is waiting for something to do, then ONE packet is read from the detail file, and handed to a child thread.
This means that if the server is busy handling authentication and/or accounting requests, it completely ignores the detail file. As it should.
I've also updated the "detail" module so that in the post-proxy section, it writes the accounting packet IF there's no reply from the home server. This lets the detail module sit in Post-Proxy-Type Fail, and Just Do The Right Thing.
Please test. I've done some basic work, and it should be OK. It doesn't hit assertions, it doesn't crash, and I've run 1000's of packets through it.
Sounds great. I will test it asap. Only thing that comes to mind is that you may want to add a "delay" option that pauses the read thread for X milliseconds in between packets to allow throttling of packets. This is not critical, but rather a "nice to have" option that should be trivial to implement while you are digging around in that section of the code.. Cheers -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc
Peter Nixon wrote:
Only thing that comes to mind is that you may want to add a "delay" option that pauses the read thread for X milliseconds in between packets to allow throttling of packets. This is not critical, but rather a "nice to have" option that should be trivial to implement while you are digging around in that section of the code..
OK. Right now, it reads the packets as fast as the back-end can process them. Throttling means that the machine is still responsive in the event of a large backlog of accounting packets. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
On Fri 04 May 2007, Alan DeKok wrote:
Peter Nixon wrote:
Only thing that comes to mind is that you may want to add a "delay" option that pauses the read thread for X milliseconds in between packets to allow throttling of packets. This is not critical, but rather a "nice to have" option that should be trivial to implement while you are digging around in that section of the code..
OK. Right now, it reads the packets as fast as the back-end can process them. Throttling means that the machine is still responsive in the event of a large backlog of accounting packets.
Right. My production PostgreSQL server which has a redundant pair of FreeRADIUS 2.0pre servers with SQL auth, accounting and dual sqlippools each poking at it sits on a 15min system load average of 0.12 When running detail2db.pl (the perl import script I posted to the list a few weeks ago) which only uses a single thread and a single DB socket, from a remote machine (same machine as freeradius, not same as PostgreSQL) the DB server load spikes to 2+ and the DB sometimes responds slow enough that FreeRADIUS Auth queries fail and errors are logged (not always.. depends on traffic levels) For that reason I have added a 3000 usec sleep in between processing of each packet, which keeps the DB load at about 0.8 while processing a months worth of detail files. Note that perl should be a bit slower than C so a good default sleep time is probably around 5000usec.. Cheers -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc
Peter Nixon wrote:
When running detail2db.pl (the perl import script I posted to the list a few weeks ago) which only uses a single thread and a single DB socket, from a remote machine (same machine as freeradius, not same as PostgreSQL) the DB server load spikes to 2+ and the DB sometimes responds slow enough that FreeRADIUS Auth queries fail and errors are logged (not always.. depends on traffic levels)
Ouch. That's not nice. I had thought that the priority queuing would ensure that authentication packets get handled before detail files. On the other hand, if the CPU is pegged from handling the detail packets, it won't have time to notice that an authentication packet has arrived.
For that reason I have added a 3000 usec sleep in between processing of each packet, which keeps the DB load at about 0.8 while processing a months worth of detail files. Note that perl should be a bit slower than C so a good default sleep time is probably around 5000usec..
Hmm... Hard-coded sleep times don't adapt well to changing circumstances. Maybe changing the code to require *2* waiting threads would be better. That way, the "max active threads" limit wouldn't be reached. Since that's not reached, more threads won't ever be created to handle the flood of detail packets. And, there will always be one waiting thread, which can handle any authentication packet that comes in. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
On Sat 05 May 2007, Alan DeKok wrote:
Peter Nixon wrote:
When running detail2db.pl (the perl import script I posted to the list a few weeks ago) which only uses a single thread and a single DB socket, from a remote machine (same machine as freeradius, not same as PostgreSQL) the DB server load spikes to 2+ and the DB sometimes responds slow enough that FreeRADIUS Auth queries fail and errors are logged (not always.. depends on traffic levels)
Ouch. That's not nice. I had thought that the priority queuing would ensure that authentication packets get handled before detail files.
On the other hand, if the CPU is pegged from handling the detail packets, it won't have time to notice that an authentication packet has arrived.
For that reason I have added a 3000 usec sleep in between processing of each packet, which keeps the DB load at about 0.8 while processing a months worth of detail files. Note that perl should be a bit slower than C so a good default sleep time is probably around 5000usec..
Hmm... Hard-coded sleep times don't adapt well to changing circumstances. Maybe changing the code to require *2* waiting threads would be better. That way, the "max active threads" limit wouldn't be reached. Since that's not reached, more threads won't ever be created to handle the flood of detail packets. And, there will always be one waiting thread, which can handle any authentication packet that comes in.
Hi Alan I think you may have misunderstood the problem and my solution slightly. The CPU on the RADIUS servers NEVER gets pegged. Its the system load on the backend DB server which goes high, slowing everything down. Now in fact its not even the CPU that is being pegged, but rather the hard disk(s). Now, you can always add more, faster harddisks and try to trim down the amount of data you keep in your DB (so you seek less, and have smaller indexes), but that fact remains that even with a dedicated DB server like I have, your hard disks will NEVER be able to keep up with a similar speed RADIUS server poking at it full speed with multiple threads, unless you have at least one disk per radius thread in a RAID mirror so that each thread gets its own disk head to seek with. This is an unlikely situation as its extremely expensive. Having 2 waiting threads before the detail file reader kicks in is a good idea.. Maybe it should even be 3, or configurable... But it still doesn't solve the problem above.. I have the following queries hitting the database during normal operation: Auth-Request: authorize_check_query - select from stored procedure with multiple joins and selects authorize_reply_query - select from stored procedure with multiple joins and selects authorize_group_check_query - default FR select which includes join authorize_group_reply_query - default FR select which includes join authenticate_query - never gets hit in my setup... duplicate_session_killer - single select from radacct sqlippool_dynamic - modified default FR select sqlippool_static - modified default FR select group_membership_query - default FR select postauth_query - default FR insert Now, this chunk of queries happens for EVERY auth packet. At the same time, on other threads you could have multiple Accounting start, update or stop packets which are poking new data into both the radacct table as well as the sqlippool tables slowing things down by locking bits of those tables and doing index updates... One possible optimisation I could make it to have my duplicate_session_killer use data from the sqlippool tables (which are fixed in length and therefore should be faster to update indexes) than radacct. I need t benchmark this though, and haven't seen the need yet to break a working system. My current config uses 2 separate sql modules, one for auth queries and one for acct. Both hit the same database. This ensures that auth can never overwhelm acct and visa versa.. (Maybe some of your new changes will make this redundant but it certainly improved things when I did it 6 months ago) Now, whileever you have a relatively even mix of radius packets, spread apart in time everythings works well. (As I said my DB has a 15min load average of 0.12 and in normal operation never goes over 0.3) However when you have a single thread poking accounting data into the system at full speed things start to break down. A single thread can push data into the radacct table at > 1000 records per second, which pegs the hard disks (and works the CPU pretty well also) making the DB spend all its time updating indexes (as well as writing to disk of course). This works slows down the multi-query auth process to the point where it can start to internally time out, and certainly replies late to the NAS. So, again.. Having the detail reader not inject packets back into the system unless there is > X threads free is a good idea, but I still think that that reader needs a configurable delay in between each packet that it injects... Its no the lack of threads thats the issue (although that IS an issue), its the amount of load one dedicated thread can create... Cheers -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc
Peter Nixon wrote:
I think you may have misunderstood the problem and my solution slightly. The CPU on the RADIUS servers NEVER gets pegged. Its the system load on the backend DB server which goes high, slowing everything down. Now in fact its not even the CPU that is being pegged, but rather the hard disk(s).
Ah, OK.
One possible optimisation I could make it to have my duplicate_session_killer use data from the sqlippool tables (which are fixed in length and therefore should be faster to update indexes) than radacct. I need t benchmark this though, and haven't seen the need yet to break a working system.
I like Kostas' idea http://kkalev.wordpress.com/2007/03/25/radius-server-performance-tips/ Always believe the accounting DB for duplicate logins, but write duplicate logins to a separate DB, too. Have another process read the second DB, check the NAS, and generate "stop" records if the user isn't really online. Or maybe that's a separate problem from what you have.
My current config uses 2 separate sql modules, one for auth queries and one for acct. Both hit the same database. This ensures that auth can never overwhelm acct and visa versa.. (Maybe some of your new changes will make this redundant but it certainly improved things when I did it 6 months ago)
That's the hope. We'll have to see if it works in practice. I suspect that the prioritization of auth versus acct matters only in high-load situations. If you're only getting 10-20 packets per second, there's always a gap between packets, and the auth/acct prioritization doesn't matter. . However when you have a single thread poking accounting data into the system
at full speed things start to break down. A single thread can push data into the radacct table at > 1000 records per second, which pegs the hard disks (and works the CPU pretty well also) making the DB spend all its time updating indexes (as well as writing to disk of course).
The goal, then, is to write to the DB in such a way that there's bandwidth left over for authentication packets. That's hard for the RADIUS server to know.
So, again.. Having the detail reader not inject packets back into the system unless there is > X threads free is a good idea, but I still think that that reader needs a configurable delay in between each packet that it injects...
Probably, yes.
Its no the lack of threads thats the issue (although that IS an issue), its the amount of load one dedicated thread can create...
All right. I'd like to rate-limit the detail packets, but also keep it adaptable to changing DB/hardware issues. We can perhaps fix this by having only one detail packet "in-flight" at a time. We also keep track of round trip time to the DB. We send the "next" detail packet only after 2*RTT, ensuring that the load factor on the DB due to detail file handling is always less than 50%. That should be relatively easy to implement. The configuration should be "load_factor", expressed in percentage 1..100. The server then calculates "delay = (RTT * (100-load_factor))/load_factor" That's adaptable, and should be easy for administrators to understand and configure. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
On Sun 06 May 2007, Alan DeKok wrote:
Peter Nixon wrote:
I think you may have misunderstood the problem and my solution slightly. The CPU on the RADIUS servers NEVER gets pegged. Its the system load on the backend DB server which goes high, slowing everything down. Now in fact its not even the CPU that is being pegged, but rather the hard disk(s).
Ah, OK.
One possible optimisation I could make it to have my duplicate_session_killer use data from the sqlippool tables (which are fixed in length and therefore should be faster to update indexes) than radacct. I need t benchmark this though, and haven't seen the need yet to break a working system.
I like Kostas' idea
http://kkalev.wordpress.com/2007/03/25/radius-server-performance-tips/
Yes. Thats a nice piece of writing (which I had read before). I have been planning to do some performance tests on "it" vs my current setup.
Always believe the accounting DB for duplicate logins, but write duplicate logins to a separate DB, too. Have another process read the second DB, check the NAS, and generate "stop" records if the user isn't really online.
Or maybe that's a separate problem from what you have.
Yep. Lets say its orthogonal. When a new auth request comes in, I use the data in the radacct table to sent a radius disconnect request to the NAS if data matching the new request is already "online". My indexes are pretty highly optimised, so I am unsure how much quick a new table would be (it may actually be slower as I would then be updating an extra index or 2 for each request)
My current config uses 2 separate sql modules, one for auth queries and one for acct. Both hit the same database. This ensures that auth can never overwhelm acct and visa versa.. (Maybe some of your new changes will make this redundant but it certainly improved things when I did it 6 months ago)
That's the hope. We'll have to see if it works in practice. I suspect that the prioritization of auth versus acct matters only in high-load situations. If you're only getting 10-20 packets per second, there's always a gap between packets, and the auth/acct prioritization doesn't matter.
. However when you have a single thread poking accounting data into the system
at full speed things start to break down. A single thread can push data into the radacct table at > 1000 records per second, which pegs the hard disks (and works the CPU pretty well also) making the DB spend all its time updating indexes (as well as writing to disk of course).
The goal, then, is to write to the DB in such a way that there's bandwidth left over for authentication packets. That's hard for the RADIUS server to know.
So, again.. Having the detail reader not inject packets back into the
system
unless there is > X threads free is a good idea, but I still think that that reader needs a configurable delay in between each packet that it injects...
Probably, yes.
Its no the lack of threads thats the issue (although that IS an issue), its the amount of load one dedicated thread can create...
All right. I'd like to rate-limit the detail packets, but also keep it adaptable to changing DB/hardware issues. We can perhaps fix this by having only one detail packet "in-flight" at a time. We also keep track of round trip time to the DB. We send the "next" detail packet only after 2*RTT, ensuring that the load factor on the DB due to detail file handling is always less than 50%.
That should be relatively easy to implement. The configuration should be "load_factor", expressed in percentage 1..100. The server then calculates "delay = (RTT * (100-load_factor))/load_factor"
That's adaptable, and should be easy for administrators to understand and configure.
Cool. It's a little more complex than my solution, but you are correct that it should react to differing loads which my solution would not. As long as there is _some_ way to rate limit it, I will be happy :-) Cheers -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc
participants (2)
-
Alan DeKok -
Peter Nixon