annoying stop retransmissions.
Hello, I have a setup wich Is a follow: NASes ----------------FREERADIUS Auth+live acct ----------------FREERADIUS Acct only In this setup, users connect to NASes, which authenticate them against the front freeradius (Auth + live acct). The front freeradius, maintain a db of the connected users using acct tickets. Basicly it does: INSERT in table upon Start ticket reception DELETE from table upon Stop ticket reception. In turns the front freeradius proxies acct tickets to the back freeradius (Acct only), which keeps track of all sessions doing: INSERT in table upon Start ticket reception UPDATE table upon Stop ticket reception. This work as epected for most of my NASes. Unfortunately, i have some NASes that are behind a satelite link, which is a very unreliable link with regular packets loss. UDP retramission of packet make the systems work even with that kind of link, but I have one scenario that create errors: When a stop ticket is transmitted once and reaches correctly the freradius servers (nas -> front -> back), Session record is deleted from the "live acct" table, packet is then proxied to the 2nd freeradius and session in Acct table is marked as stoped (acctstoptime=something). If the front freeradius acks the Stop packet and that Ack is lost on the link, the NAS retransmit the STOP. Same thing occur,: - front radius tries to delete the sessions using its acct_stop_query, wich result in no line being modified and so tries to run its acct_stop_query_alt (which basicly does the same thind: delete). alt query also modify no lines but no error is logged. retransmitted packet is then proxied to the back server, wich in turns tries ti run its acct_stop_query (tries to update a session with no acctstoptime). That query fails as the previous Stop ticket for that session already updated the recod. It then tries to run the acct_stop_query_alt, which is designed to try to insert a new session record based on the content of the stop ticket (this is done to deal with the case where start ticket is lost and only stop ticket is received, i guess). In my case this last query fails because of some unicity constraint in the oracle database (to prevent one session from being recorded multiple times), and an error is logged in freeradius. I can't really afford to get rid of the unique constraint, and I think changing the acct_stop_query_alt for a non insert statement would not be a good idea either. Does anybody have an idea on how to deal with that (minor) problem so I have no more regular error messages. I was maybe thinking of not proxying to the back server, packets retransmitted du to ACK loss, but I can't really find out how to do that... Thanks for reading that long post (I hope it's understandable enough). -- <http://www.horoa.net> Alexandre Chapellon Ingénierie des systèmes open sources et réseaux. Follow me on twitter: @alxgomz <http://www.twitter.com/alxgomz>
Alexandre Chapellon wrote:
This work as epected for most of my NASes. Unfortunately, i have some NASes that are behind a satelite link, which is a very unreliable link with regular packets loss. UDP retramission of packet make the systems work even with that kind of link, but I have one scenario that create errors:
This is common in RADIUS. Accounting is... awkward, to put it politely.
When a stop ticket is transmitted once and reaches correctly the freradius servers (nas -> front -> back), Session record is deleted from the "live acct" table, packet is then proxied to the 2nd freeradius and session in Acct table is marked as stoped (acctstoptime=something). If the front freeradius acks the Stop packet and that Ack is lost on the link, the NAS retransmit the STOP.
As it should. It's the responsibility of the RADIUS server to deal with retransmissions from the NAS.
Same thing occur,: - front radius tries to delete the sessions using its acct_stop_query, wich result in no line being modified and so tries to run its acct_stop_query_alt (which basicly does the same thind: delete).
It really should delete the record ONLY if it exists. Or, UPDATE the record to say "session stopped". After a suitable delay (10-20 min), the "stopped" sessions can safely be deleted.
alt query also modify no lines but no error is logged. retransmitted packet is then proxied to the back server, wich in turns tries ti run its acct_stop_query (tries to update a session with no acctstoptime). That query fails as the previous Stop ticket for that session already updated the recod. It then tries to run the acct_stop_query_alt, which is designed to try to insert a new session record based on the content of the stop ticket (this is done to deal with the case where start ticket is lost and only stop ticket is received, i guess). In my case this last query fails because of some unicity constraint in the oracle database (to prevent one session from being recorded multiple times), and an error is logged in freeradius.
The solution is to fix the queries so that they deal with non-existant sessions. This is no different than a NAS sending a STOP for sessions that *never* existed.
Does anybody have an idea on how to deal with that (minor) problem so I have no more regular error messages. I was maybe thinking of not proxying to the back server, packets retransmitted du to ACK loss, but I can't really find out how to do that...
Thanks for reading that long post (I hope it's understandable enough).
It is. There is no real solution other than building a smarter system to handle accounting packets. I suggest writing a detailed state machine describing what happens for each session, and how each kind of packet is handled. Until that's done, no good solution is possible. We can take such a state machine and use it to update the handling of accounting packets for 3.0. Alan DeKok.
Le 28/11/2011 13:53, Alan DeKok a écrit :
Alexandre Chapellon wrote:
This work as epected for most of my NASes. Unfortunately, i have some NASes that are behind a satelite link, which is a very unreliable link with regular packets loss. UDP retramission of packet make the systems work even with that kind of link, but I have one scenario that create errors: This is common in RADIUS. Accounting is... awkward, to put it politely.
When a stop ticket is transmitted once and reaches correctly the freradius servers (nas -> front -> back), Session record is deleted from the "live acct" table, packet is then proxied to the 2nd freeradius and session in Acct table is marked as stoped (acctstoptime=something). If the front freeradius acks the Stop packet and that Ack is lost on the link, the NAS retransmit the STOP. As it should. It's the responsibility of the RADIUS server to deal with retransmissions from the NAS.
Same thing occur,: - front radius tries to delete the sessions using its acct_stop_query, wich result in no line being modified and so tries to run its acct_stop_query_alt (which basicly does the same thind: delete). It really should delete the record ONLY if it exists. Or, UPDATE the record to say "session stopped". After a suitable delay (10-20 min), the "stopped" sessions can safely be deleted.
alt query also modify no lines but no error is logged. retransmitted packet is then proxied to the back server, wich in turns tries ti run its acct_stop_query (tries to update a session with no acctstoptime). That query fails as the previous Stop ticket for that session already updated the recod. It then tries to run the acct_stop_query_alt, which is designed to try to insert a new session record based on the content of the stop ticket (this is done to deal with the case where start ticket is lost and only stop ticket is received, i guess). In my case this last query fails because of some unicity constraint in the oracle database (to prevent one session from being recorded multiple times), and an error is logged in freeradius. The solution is to fix the queries so that they deal with non-existant sessions. This is no different than a NAS sending a STOP for sessions that *never* existed.
Does anybody have an idea on how to deal with that (minor) problem so I have no more regular error messages. I was maybe thinking of not proxying to the back server, packets retransmitted du to ACK loss, but I can't really find out how to do that...
Thanks for reading that long post (I hope it's understandable enough). It is.
There is no real solution other than building a smarter system to handle accounting packets.
I suggest writing a detailed state machine describing what happens for each session, and how each kind of packet is handled. Until that's done, no good solution is possible. I don't understand what you mean by "writing a detailed state machine"... state machine?
We can take such a state machine and use it to update the handling of accounting packets for 3.0.
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
-- <http://www.horoa.net> Alexandre Chapellon Ingénierie des systèmes open sources et réseaux. Follow me on twitter: @alxgomz <http://www.twitter.com/alxgomz>
Alexandre Chapellon wrote:
I don't understand what you mean by "writing a detailed state machine"... state machine?
Write down what the server receives, and what you want to do with it. The server receives accounting on/off start/stop and alive packets. It can receive those when a session is non-existent, open, or closed. How does the server determine the "right" thing to do? Write down the logic. Then, implement it. Alan DeKok.
On Tue, Nov 29, 2011 at 3:57 PM, Alan DeKok <aland@deployingradius.com> wrote:
Alexandre Chapellon wrote:
I don't understand what you mean by "writing a detailed state machine"... state machine?
Write down what the server receives, and what you want to do with it. The server receives accounting on/off start/stop and alive packets. It can receive those when a session is non-existent, open, or closed.
How does the server determine the "right" thing to do? Write down the logic. Then, implement it.
For comparison purposes, we currently implement some config and db schema modifications to better handle accounting packets. For example: - always return ok on accounting packets that the NAS sends. Implementing it requires using detail reader/writer and some unlang blocks to catch some weird corner cases (e.g. Acct-Session-Time=0) - use unique constraint on acctuniqueid - remove all *_alt queries - split "live" accounting table (those with Acct-Status-Type <> 'Stop') and "archive" accounting table - change most insert/update queries to "INSERT .... ON DUPLICATE KEY UPDATE ...." - use "DELETE IGNORE" to delete records from "live" accounting table Took some effort, but it works. @Alexandre: some of the logic might be relevant for your situation as well. -- Fajar
Le 29/11/2011 10:20, Fajar A. Nugraha a écrit : > On Tue, Nov 29, 2011 at 3:57 PM, Alan DeKok<aland@deployingradius.com> wrote: >> Alexandre Chapellon wrote: >>> I don't understand what you mean by "writing a detailed state >>> machine"... state machine? >> Write down what the server receives, and what you want to do with it. >> The server receives accounting on/off start/stop and alive packets. >> It can receive those when a session is non-existent, open, or closed. >> >> How does the server determine the "right" thing to do? Write down the >> logic. Then, implement it. > For comparison purposes, we currently implement some config and db > schema modifications to better handle accounting packets. For example: > - always return ok on accounting packets that the NAS sends. > Implementing it requires using detail reader/writer and some unlang > blocks to catch some weird corner cases (e.g. Acct-Session-Time=0) > - use unique constraint on acctuniqueid > - remove all *_alt queries > - split "live" accounting table (those with Acct-Status-Type<> > 'Stop') and "archive" accounting table > - change most insert/update queries to "INSERT .... ON DUPLICATE KEY > UPDATE ...." > - use "DELETE IGNORE" to delete records from "live" accounting table > > Took some effort, but it works. @Alexandre: some of the logic might be > relevant for your situation as well. Yes indeed, "INSERT... ON DUPLICATE KEY UPDATE" and "DELETE IGNORE" really sound like my holy graal. Will forward this to my dba asap! Thank you! Regards @Fajar: I'm eating quite a bit of your time on different mailing lists lately :)! thank you again. -- <http://www.horoa.net> Alexandre Chapellon Ingénierie des systèmes open sources et réseaux. Follow me on twitter: @alxgomz <http://www.twitter.com/alxgomz>
Fajar A. Nugraha wrote:
For comparison purposes, we currently implement some config and db schema modifications to better handle accounting packets. For example: - always return ok on accounting packets that the NAS sends. Implementing it requires using detail reader/writer and some unlang blocks to catch some weird corner cases (e.g. Acct-Session-Time=0)
OK. What does the detail reader/writer do? Catch cases where the SQL DB is down?
- use unique constraint on acctuniqueid
The git "master" branch has some additional changes to help make a more unique accounting ID.
- remove all *_alt queries
Yeah. I've never really understood the need for them. The module failover can do the work just as well, I think.
- split "live" accounting table (those with Acct-Status-Type <> 'Stop') and "archive" accounting table
That's necessary. It can speed up Simultaneous-Use checking, too.
- change most insert/update queries to "INSERT .... ON DUPLICATE KEY UPDATE ...." - use "DELETE IGNORE" to delete records from "live" accounting table
That's MySQL specific, but useful.
Took some effort, but it works. @Alexandre: some of the logic might be relevant for your situation as well.
I'd like to see that added into the server for 3.0. Patches? Alan DeKok.
On Tue, Nov 29, 2011 at 5:33 PM, Alan DeKok <aland@deployingradius.com> wrote:
Fajar A. Nugraha wrote:
For comparison purposes, we currently implement some config and db schema modifications to better handle accounting packets. For example: - always return ok on accounting packets that the NAS sends. Implementing it requires using detail reader/writer and some unlang blocks to catch some weird corner cases (e.g. Acct-Session-Time=0)
OK. What does the detail reader/writer do? Catch cases where the SQL DB is down?
Yup. Basically sites-available/buffered-sql
- use unique constraint on acctuniqueid
The git "master" branch has some additional changes to help make a more unique accounting ID.
Great to hear that. Will try it for next implementation. I think we used UserName-acctuniqid combination (forgot the exact implementation, can't access the server right now, sorry) to help prevent collision, especially in the "archive" acct table. When only using acctuniqid, I start getting collision (on test with radperf) at around 5 million records.
- remove all *_alt queries
Yeah. I've never really understood the need for them. The module failover can do the work just as well, I think.
It may be suitable to workaround some db limitation. If the db can't do INSERT ... ON DUPLICATE KEY UPDATE, you might get away with INSERT on the main query and UPDATE on _alt. However we use mysql (which support the syntax just fine) and don't want the extra query which adds load to the db server (the _alt part), so I just remove them
- split "live" accounting table (those with Acct-Status-Type <> 'Stop') and "archive" accounting table
That's necessary. It can speed up Simultaneous-Use checking, too.
Yes. The tricky part here is that you need some kind of method to track whether a record there is truly "alive". Thus we modified the query and schema further to record AcctStatusType and hijacked AcctStopTime to record when the packet was received. A cleaner approach would probably be adding PacketTimestamp column.
- change most insert/update queries to "INSERT .... ON DUPLICATE KEY UPDATE ...." - use "DELETE IGNORE" to delete records from "live" accounting table
That's MySQL specific, but useful.
as I said, I use mysql :D
Took some effort, but it works. @Alexandre: some of the logic might be relevant for your situation as well.
I'd like to see that added into the server for 3.0.
Patches?
Will try to see which parts are upstreamable. My current implementation is too complex be used for a generic implementation in its complete form, using stored-procedures for most queries in dialup.conf. Plus there's another ugly hack to correctly transfer information from "live" to "archive" accounting table. -- Fajar
On 29/11/11 12:04, Fajar A. Nugraha wrote:
It may be suitable to workaround some db limitation. If the db can't do INSERT ... ON DUPLICATE KEY UPDATE, you might get away with INSERT on the main query and UPDATE on _alt. However we use mysql (which support the syntax just fine) and don't want the extra query which adds load to the db server (the _alt part), so I just remove them
We don't use the built-in "sql" module accounting queries; but the _alt queries are important for exactly this reason, particularly with e.g. postgres. I think using module failover for this would be cumbersome; you'd have to have quite a few SQL instances AFAICT to replicate this behaviour. Or used stored procedures (which is what we do) but that's a lot of overhead.
Le 29/11/2011 14:02, Phil Mayers a écrit :
On 29/11/11 12:04, Fajar A. Nugraha wrote:
It may be suitable to workaround some db limitation. If the db can't do INSERT ... ON DUPLICATE KEY UPDATE, you might get away with INSERT on the main query and UPDATE on _alt. However we use mysql (which support the syntax just fine) and don't want the extra query which adds load to the db server (the _alt part), so I just remove them
We don't use the built-in "sql" module accounting queries; but the _alt queries are important for exactly this reason, particularly with e.g. postgres. I think using module failover for this would be cumbersome; you'd have to have quite a few SQL instances AFAICT to replicate this behaviour. I agree, I originally found a oracle page talking about 'insert on duplicate key update' sattement and so supposed oracle supported this... Unfortunatelly I didn't noticed that this oracle page was about... mysql (now oracle property)! So Oracle is like postgres and needs the alt_ queries. A similar behaviour seems to be achieved using the MERGE statement... need to take a look at it.
regards.
Or used stored procedures (which is what we do) but that's a lot of overhead. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
-- <http://www.horoa.net> Alexandre Chapellon Ingénierie des systèmes open sources et réseaux. Follow me on twitter: @alxgomz <http://www.twitter.com/alxgomz>
participants (4)
-
Alan DeKok -
Alexandre Chapellon -
Fajar A. Nugraha -
Phil Mayers