What is the purpose of the default accounting-on/off queries?
Hello everyone, I have two questions: 1) What is the purpose of the default accounting-on and accounting-off queries bundled with freeradius? I am asking this because one of my customer's NAS is sending accounting-on packets for no reason, and Freeradius is closing the sessions with the 'NAS-Reboot' Acct-Terminate-Cause. I was thinking about disabling the accounting-on query altogether, and leaving only the accounting-off. Are there any side effects by doing this? 2) Does Freeradius use accounting-off packets to release IPs allocated in the sqlippool? Regards -- Att, *Antônio Modesto*
On 10/10/2021, at 3:16 AM, Antônio Modesto <modesto@hubsoft.com.br> wrote:
Hello everyone,
I have two questions:
1) What is the purpose of the default accounting-on and accounting-off queries bundled with freeradius? I am asking this because one of my customer's NAS is sending accounting-on packets for no reason, and Freeradius is closing the sessions with the 'NAS-Reboot' Acct-Terminate-Cause. I was thinking about disabling the accounting-on query altogether, and leaving only the accounting-off. Are there any side effects by doing this?
This means the NAS is misbehaving. Accounting-On and Accounting-Off should only be sent when the NAS is booting or shutting down - i.e. when sessions are terminated en mass. In those situations, the NAS might not send Stop messages for each session (and in many cases, such as a crash, cannot). Leaving only Accounting-Off will mean in case of a NAS crash, it’ll come back and may not be able to get customers online until the sessions expire in the RADIUS server state - in case you limit concurrent sessions, or have limited IPs in sqlippool. Accounting-On is important in these situations. Perhaps to work around the poor RADIUS implementation on the NAS, you can filter out these messages - do they have any additional attributes? Some misbehaving NASes send Accounting-On/Off for subsystems, with additional attributes to identify the subsystem. Send the NAS vendor this page: https://freeradius.org/rfc/acct_status_type_subsystem.html <https://freeradius.org/rfc/acct_status_type_subsystem.html> Can you share who the vendor is?
2) Does Freeradius use accounting-off packets to release IPs allocated in the sqlippool?
By default, yes. -- Nathan Ward
On 09/10/2021 11:24, Nathan Ward wrote:
On 10/10/2021, at 3:16 AM, Antônio Modesto <modesto@hubsoft.com.br> wrote:
Hello everyone,
I have two questions:
1) What is the purpose of the default accounting-on and accounting-off queries bundled with freeradius? I am asking this because one of my customer's NAS is sending accounting-on packets for no reason, and Freeradius is closing the sessions with the 'NAS-Reboot' Acct-Terminate-Cause. I was thinking about disabling the accounting-on query altogether, and leaving only the accounting-off. Are there any side effects by doing this? This means the NAS is misbehaving. Accounting-On and Accounting-Off should only be sent when the NAS is booting or shutting down - i.e. when sessions are terminated en mass. In those situations, the NAS might not send Stop messages for each session (and in many cases, such as a crash, cannot). Leaving only Accounting-Off will mean in case of a NAS crash, it’ll come back and may not be able to get customers online until the sessions expire in the RADIUS server state - in case you limit concurrent sessions, or have limited IPs in sqlippool. Accounting-On is important in these situations.
Perhaps to work around the poor RADIUS implementation on the NAS, you can filter out these messages - do they have any additional attributes? Some misbehaving NASes send Accounting-On/Off for subsystems, with additional attributes to identify the subsystem.
Send the NAS vendor this page: https://freeradius.org/rfc/acct_status_type_subsystem.html <https://freeradius.org/rfc/acct_status_type_subsystem.html> Can you share who the vendor is?
It is a Juniper MX5 router. Based on what you said, I think the safest choice for us is to ignore accounting-on, and leave just accounting-off. We have other mechanisms to deal with staled sessions and pool addresses. This NAS is sending accounting-on in a totally random fashion.
2) Does Freeradius use accounting-off packets to release IPs allocated in the sqlippool? By default, yes.
-- Nathan Ward
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
-- Att, *Antônio Modesto*
On 10/10/2021, at 3:51 AM, Antônio Modesto <modesto@hubsoft.com.br> wrote:
On 09/10/2021 11:24, Nathan Ward wrote:
On 10/10/2021, at 3:16 AM, Antônio Modesto <modesto@hubsoft.com.br> wrote:
Hello everyone,
I have two questions:
1) What is the purpose of the default accounting-on and accounting-off queries bundled with freeradius? I am asking this because one of my customer's NAS is sending accounting-on packets for no reason, and Freeradius is closing the sessions with the 'NAS-Reboot' Acct-Terminate-Cause. I was thinking about disabling the accounting-on query altogether, and leaving only the accounting-off. Are there any side effects by doing this? This means the NAS is misbehaving. Accounting-On and Accounting-Off should only be sent when the NAS is booting or shutting down - i.e. when sessions are terminated en mass. In those situations, the NAS might not send Stop messages for each session (and in many cases, such as a crash, cannot). Leaving only Accounting-Off will mean in case of a NAS crash, it’ll come back and may not be able to get customers online until the sessions expire in the RADIUS server state - in case you limit concurrent sessions, or have limited IPs in sqlippool. Accounting-On is important in these situations.
Perhaps to work around the poor RADIUS implementation on the NAS, you can filter out these messages - do they have any additional attributes? Some misbehaving NASes send Accounting-On/Off for subsystems, with additional attributes to identify the subsystem.
Send the NAS vendor this page: https://freeradius.org/rfc/acct_status_type_subsystem.html <https://freeradius.org/rfc/acct_status_type_subsystem.html><https://freeradius.org/rfc/acct_status_type_subsystem.html <https://freeradius.org/rfc/acct_status_type_subsystem.html>> Can you share who the vendor is?
It is a Juniper MX5 router.
Based on what you said, I think the safest choice for us is to ignore accounting-on, and leave just accounting-off. We have other mechanisms to deal with staled sessions and pool addresses. This NAS is sending accounting-on in a totally random fashion.
Ah yep. It will be sending Accounting-On when a VRF (routing-instance) gets the first customer, or has its configuration changed in certain ways (depending on the version). I have this very problem. I have been working with my SE to get an ER raised for this to get them to use Subsystem-On and Subsystem-Off. I’ll follow up and see where that’s gotten to. You can work around this as follows: 1) Create an `access profile` per VRF - Juniper recommend this anyway, so that requests for one VRF aren’t competing in a queue with requests in another VRF. This can of course have the same details as the current access profile you have, except for item 2: 2) Set the NAS-Identifier attribute uniquely for each profile: `set access profile foo radius options nas-identifier “bar”` 3) Set this profile in the VRF `set routing-instances baz access-profile foo` This will get your Accounting-On messages coming with an attribute you can filter on. If you don’t want to do that per VRF, you could create one common “inside VRF” profile, and one “global” profile, and of course have different NAS-Identifiers to filter on. You want to keep the global Accounting-On messages, and throw away the others. -- Nathan Ward
On Sat, 9 Oct 2021 at 16:03, Nathan Ward <lists+freeradius@daork.net> wrote:
On 10/10/2021, at 3:51 AM, Antônio Modesto <modesto@hubsoft.com.br> wrote: On 09/10/2021 11:24, Nathan Ward wrote:
On 10/10/2021, at 3:16 AM, Antônio Modesto <modesto@hubsoft.com.br> wrote:
Hello everyone,
I have two questions:
1) What is the purpose of the default accounting-on and accounting-off queries bundled with freeradius? I am asking this because one of my customer's NAS is sending accounting-on packets for no reason, and Freeradius is closing the sessions with the 'NAS-Reboot' Acct-Terminate-Cause. I was thinking about disabling the accounting-on query altogether, and leaving only the accounting-off. Are there any side effects by doing this? This means the NAS is misbehaving. Accounting-On and Accounting-Off should only be sent when the NAS is booting or shutting down - i.e. when sessions are terminated en mass. In those situations, the NAS might not send Stop messages for each session (and in many cases, such as a crash, cannot). Leaving only Accounting-Off will mean in case of a NAS crash, it’ll come back and may not be able to get customers online until the sessions expire in the RADIUS server state - in case you limit concurrent sessions, or have limited IPs in sqlippool. Accounting-On is important in these situations.
Perhaps to work around the poor RADIUS implementation on the NAS, you can filter out these messages - do they have any additional attributes? Some misbehaving NASes send Accounting-On/Off for subsystems, with additional attributes to identify the subsystem.
Send the NAS vendor this page: https://freeradius.org/rfc/acct_status_type_subsystem.html <https://freeradius.org/rfc/acct_status_type_subsystem.html><https://freeradius.org/rfc/acct_status_type_subsystem.html <https://freeradius.org/rfc/acct_status_type_subsystem.html>> Can you share who the vendor is?
It is a Juniper MX5 router.
Based on what you said, I think the safest choice for us is to ignore accounting-on, and leave just accounting-off. We have other mechanisms to deal with staled sessions and pool addresses. This NAS is sending accounting-on in a totally random fashion.
Ah yep. It will be sending Accounting-On when a VRF (routing-instance) gets the first customer, or has its configuration changed in certain ways (depending on the version). I have this very problem.
Another issue with the MX series that we've discovered the hard way is that they can continue to send Accounting-On requests without backoff *forever*, until the request is acknowledged. So if you are "ignoring" them you must still send an Accounting-Response. As well as the VRF issue, you can experience this in global context if the database connection is configured with a query timeout and the bulk session update does not complete in time.
On 10/10/2021, at 6:11 AM, Terry Burton <terry.burton@gmail.com> wrote:
On Sat, 9 Oct 2021 at 16:03, Nathan Ward <lists+freeradius@daork.net> wrote:
Ah yep. It will be sending Accounting-On when a VRF (routing-instance) gets the first customer, or has its configuration changed in certain ways (depending on the version). I have this very problem.
Another issue with the MX series that we've discovered the hard way is that they can continue to send Accounting-On requests without backoff *forever*, until the request is acknowledged. So if you are "ignoring" them you must still send an Accounting-Response.
That sounds what I would want to happen, so I don’t think that’s an MX issue. You want to be sure that Accounting-On/Off is being handled. But yes good point - you want to make sure you respond to any requests that you “ignore”, rather than filter them out and *really* ignore them :-)
As well as the VRF issue, you can experience this in global context if the database connection is configured with a query timeout and the bulk session update does not complete in time.
It seems very odd to me that you would have an Accounting-On/Off query taking anywhere near what a DB query timeout is. That doesn’t sound like an MX issue - that sounds like some index tuning is required, or perhaps the query is doing a large amount of work that could be handled another way? -- Nathan Ward
On Oct 9, 2021, at 10:28 PM, Nathan Ward <lists+freeradius@daork.net> wrote:
As well as the VRF issue, you can experience this in global context if the database connection is configured with a query timeout and the bulk session update does not complete in time.
It seems very odd to me that you would have an Accounting-On/Off query taking anywhere near what a DB query timeout is. That doesn’t sound like an MX issue - that sounds like some index tuning is required, or perhaps the query is doing a large amount of work that could be handled another way?
If it's a big GGSN, then updating 100K rows can take time. Doing that update in an asynchronous transaction would be better, perhaps. Alan DeKok.
On Sun, 10 Oct 2021, 12:16 Alan DeKok, <aland@deployingradius.com> wrote:
On Oct 9, 2021, at 10:28 PM, Nathan Ward <lists+freeradius@daork.net> wrote:
As well as the VRF issue, you can experience this in global context if the database connection is configured with a query timeout and the bulk session update does not complete in time.
It seems very odd to me that you would have an Accounting-On/Off query taking anywhere near what a DB query timeout is. That doesn’t sound like an MX issue - that sounds like some index tuning is required, or perhaps the query is doing a large amount of work that could be handled another way?
If it's a big GGSN, then updating 100K rows can take time. Doing that update in an asynchronous transaction would be better, perhaps.
Indeed. You can also maintain a per-NAS epoch table using Accounting On/Off. Then adjust routine queries to join against this (or just make a view in which acctstoptime = NULL is replaced with the end of the previous epoch unless the session started afterwards), allowing you to ignore sessions not belonging to the current epoch. But it's the kind of thing that only becomes with doing when needed because of the added schema/query complexity...
On 11/10/2021, at 1:02 AM, Terry Burton <terry.burton@gmail.com> wrote:
On Sun, 10 Oct 2021, 12:16 Alan DeKok, <aland@deployingradius.com <mailto:aland@deployingradius.com>> wrote:
On Oct 9, 2021, at 10:28 PM, Nathan Ward <lists+freeradius@daork.net> wrote:
As well as the VRF issue, you can experience this in global context if the database connection is configured with a query timeout and the bulk session update does not complete in time.
It seems very odd to me that you would have an Accounting-On/Off query taking anywhere near what a DB query timeout is. That doesn’t sound like an MX issue - that sounds like some index tuning is required, or perhaps the query is doing a large amount of work that could be handled another way?
If it's a big GGSN, then updating 100K rows can take time. Doing that update in an asynchronous transaction would be better, perhaps.
Indeed.
You can also maintain a per-NAS epoch table using Accounting On/Off. Then adjust routine queries to join against this (or just make a view in which acctstoptime = NULL is replaced with the end of the previous epoch unless the session started afterwards), allowing you to ignore sessions not belonging to the current epoch.
But it's the kind of thing that only becomes with doing when needed because of the added schema/query complexity…
Hmm, interesting! I did some testing on a DB just now and can clear 160k sessions in about 3 seconds, from a table of about 2.8M. My schema is a little different, but not significantly for what is relevant for what accounting-on/off is doing. There are updates going on, maybe 300/s or so (it’s a DB with a mirror of live data, I’m not poking a production DB!). Some point soon I'll synthesise some data on the shipped schemas and see how it performs and see if I can tweak them to be faster - if 100k sort of size is a problem then it may be worth putting some effort in to optimising, as that’s not very big. Maybe some alternative schema/queries for high scale, with complexity as a trade off. -- Nathan Ward
On Oct 10, 2021, at 8:20 AM, Nathan Ward <lists+freeradius@daork.net> wrote:
Hmm, interesting! I did some testing on a DB just now and can clear 160k sessions in about 3 seconds, from a table of about 2.8M. My schema is a little different, but not significantly for what is relevant for what accounting-on/off is doing. There are updates going on, maybe 300/s or so (it’s a DB with a mirror of live data, I’m not poking a production DB!).
You've probably got reasonable indexes. I've seen people remove indexes in order to do "optimization". That doesn't work as well as they would think.
Some point soon I'll synthesise some data on the shipped schemas and see how it performs and see if I can tweak them to be faster - if 100k sort of size is a problem then it may be worth putting some effort in to optimising, as that’s not very big. Maybe some alternative schema/queries for high scale, with complexity as a trade off.
The main benefit I've seen is splitting "live" data from "historical" data. The "live" data is usually small (one row per online user). The "historical" data is huge, as it's one row per user session for the last 6-12 months. The key realization here is that the "live" database gets a lot of updates and simple queries, while the "historical" database gets no updates and more complex queries. We've had good luck putting the "live" data into Redis, which can easily do 40K updates/s. Then periodically flushing that to SQL, which works much better when you give it 10K rows to update in one transaction. Alan DeKok.
On Sun, 10 Oct 2021, 13:27 Alan DeKok, <aland@deployingradius.com> wrote:
On Oct 10, 2021, at 8:20 AM, Nathan Ward <lists+freeradius@daork.net> wrote:
Hmm, interesting! I did some testing on a DB just now and can clear 160k sessions in about 3 seconds, from a table of about 2.8M. My schema is a little different, but not significantly for what is relevant for what accounting-on/off is doing. There are updates going on, maybe 300/s or so (it’s a DB with a mirror of live data, I’m not poking a production DB!).
You've probably got reasonable indexes. I've seen people remove indexes in order to do "optimization". That doesn't work as well as they would think.
I agree in general. Things can become counterintuitive when mixing many update types. With workloads with a high *concurrent* UPDATE rate (e.g. >32 accounting updates) a 3 second bulk update over the table can stall things, or take a significant time to acquire the necessary row locks. Serialising through a buffered reader can help with bulk operations, up to significantly lowered limit introduced by the serialise round trips. It depends on which indexes are selected as the basis for the row locking, and often changing the query isolation level to avoid unnecessary gap locks helps a lot (especially with avoiding deadlocks). MySQL/MariaDB has several issues in which the presence of extra indexes unrelated to the query can prevent the optimal index (in terms of partitioning the data for locking purposes) from appearing as a candidate. This can result in the selection of an index that locks more rows than necessary, e.g. a long-running, repeated bulk close query can block real-time session updates because the row level locking does not partition the data by say {nasipaddress = a.b.c.d; acctstartime < Event-Timestamp} even though such an index is available :-(
On 09/10/2021 23:28, Nathan Ward wrote:
On 10/10/2021, at 6:11 AM, Terry Burton <terry.burton@gmail.com> wrote:
On Sat, 9 Oct 2021 at 16:03, Nathan Ward <lists+freeradius@daork.net> wrote:
Ah yep. It will be sending Accounting-On when a VRF (routing-instance) gets the first customer, or has its configuration changed in certain ways (depending on the version). I have this very problem. Another issue with the MX series that we've discovered the hard way is that they can continue to send Accounting-On requests without backoff *forever*, until the request is acknowledged. So if you are "ignoring" them you must still send an Accounting-Response. That sounds what I would want to happen, so I don’t think that’s an MX issue. You want to be sure that Accounting-On/Off is being handled. But yes good point - you want to make sure you respond to any requests that you “ignore”, rather than filter them out and *really* ignore them :-)
As well as the VRF issue, you can experience this in global context if the database connection is configured with a query timeout and the bulk session update does not complete in time. It seems very odd to me that you would have an Accounting-On/Off query taking anywhere near what a DB query timeout is. That doesn’t sound like an MX issue - that sounds like some index tuning is required, or perhaps the query is doing a large amount of work that could be handled another way?
-- Nathan Ward
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Can I just edit my sites-enabled/default accounting section to look like this? if (Acct-Status-Type != Accounting-On) { -sql -sqlippool } -- Att, *Antônio Modesto*
On 12/10/2021, at 6:25 AM, Antônio Modesto <modesto@hubsoft.com.br> wrote:
On 09/10/2021 23:28, Nathan Ward wrote:
On 10/10/2021, at 6:11 AM, Terry Burton <terry.burton@gmail.com> wrote:
On Sat, 9 Oct 2021 at 16:03, Nathan Ward <lists+freeradius@daork.net> wrote:
Ah yep. It will be sending Accounting-On when a VRF (routing-instance) gets the first customer, or has its configuration changed in certain ways (depending on the version). I have this very problem. Another issue with the MX series that we've discovered the hard way is that they can continue to send Accounting-On requests without backoff *forever*, until the request is acknowledged. So if you are "ignoring" them you must still send an Accounting-Response. That sounds what I would want to happen, so I don’t think that’s an MX issue. You want to be sure that Accounting-On/Off is being handled. But yes good point - you want to make sure you respond to any requests that you “ignore”, rather than filter them out and *really* ignore them :-)
As well as the VRF issue, you can experience this in global context if the database connection is configured with a query timeout and the bulk session update does not complete in time. It seems very odd to me that you would have an Accounting-On/Off query taking anywhere near what a DB query timeout is. That doesn’t sound like an MX issue - that sounds like some index tuning is required, or perhaps the query is doing a large amount of work that could be handled another way?
-- Nathan Ward
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Can I just edit my sites-enabled/default accounting section to look like this?
if (Acct-Status-Type != Accounting-On) {
-sql
-sqlippool
}
Yeah you probably could - though I guess I think you’re better off solving the problem with the NAS (BNG) rather than working around it in RADIUS. For example, if you do the above, you now need to: - ensure that your sqlippool timers are appropriate so your IPs free up if your BNG crashes so you don’t run out of IPs - ensure that you have some out of band process to close sessions if your BNG crashes - when will the stop time be? As I say.. it technically functions, but you’re making a trade off. You’re proposing that to avoid changing the BNG which is misbehaving [1], you change the RADIUS system which is doing exactly as it’s supposed to. That’s just adding technical debt - but - your network :-) [1] Juniper would say misconfigured, but really their RADIUS code is a bit wrong -- Nathan Ward
participants (4)
-
Alan DeKok -
Antônio Modesto -
Nathan Ward -
Terry Burton