Copy-acct-to-home-server Issue
Hello Guys, i was wondering if you could give me some pointers on how to troubleshoot an issue. I am running freeradius 3.0.4 on Centos 7, and i am using copy-acct-to-home-server in order to replicate the accounting database between the 3 servers. Right now, all my NAS are only pointing to one server (Lets call it server1). So server2 and server3 do not receive authentication requests, they just receive accounting packets from server1. Now, with some NAS i also see issues where i miss accounting stop packets, and i end up with stale sessions. I am working on solving that, but in the mean time i run a script on all three servers that queries all NAS for active sessions and marks the stale sessions as Terminated. My script logs the stale sessions, so i can compare them between servers. My problem is that the stale sessions between the servers are different. Usually, server 2 and 3 have more stale sessions than server 1. This would tell me that there are accounting packets not processed or missing between server 1 and server 2 and 3. First thing i did was looking at the networking side of things. All interfaces between the servers run clean, they are all connected to the same switch. So i am trying to troubleshoot what is happeing with these packets. The questions i have are: 1) My copy acct-to-home-server for each server configuration is as follows: - log all the incoming packets to a detail file. -copy-acct-to-home-server server reads the detail file and updates the proxy-to-realm control attribute so the packet gets proxied to the other server. Acct always returns ok. If i understand this correctly, in this configuration the packets are "Sent and forgotten". So if a packet fails to be received or processed by server2 or server3, it wont be retried. The fact that all the detail files are empty makes me think that this is the way the server is working right now. Any ideas on how could i implement a simple "retry 3 times" policy? My issue is that since i proxy the packet, and and always return ok on accounting, i am not sure where to put the "IF packet failed to ACK, return fail". From what i see, it cannot be done in the proxy section, unless i do something like: realm SERVER1 { } 2) I dont see anything in the logs that would indicate there is a problem with packets being sent to the other to servers. How can i catch these issues from the logs or debug perspective? the only thing i can think of is to turn debug on for all the packets with packet-source-destination equals "server 2" and server 3" in server one, and go from there, but any suggestions would be appreciated, as that would basically be my whole radius acct traffic. I am copying here the contents of copy-acct-to-home-server, for reference (They are identical for all servers): server copy-acct-to-SERVER2 { listen { type = detail filename = /buffer_disk/SERVER2/detail_SERVER2 load_factor = 10 } preacct { update control { Proxy-To-Realm := "SERVER2" } } accounting { ok } pre-proxy { } post-proxy { } } My proxy.conf section only contains a realm section from it server, in the old configuration fashion (I probably should change it to the new one): realm "SERVER1" { nostrip authhost = 192.168.1.1:1812 accthost = 192.168.1.1:1813 secret=itssecret } As always, thank you very much for your help!. Regards
On Jul 7, 2015, at 7:32 AM, Ricardo Larrañaga <ricardo.larranaga@gmail.com> wrote:
Right now, all my NAS are only pointing to one server (Lets call it server1). So server2 and server3 do not receive authentication requests, they just receive accounting packets from server1.
That's simple.
Now, with some NAS i also see issues where i miss accounting stop packets, and i end up with stale sessions. I am working on solving that, but in the mean time i run a script on all three servers that queries all NAS for active sessions and marks the stale sessions as Terminated.
Why? Just run the script once, and have it write RADIUS packets to a detail file. Those can be replicated to every server.
My script logs the stale sessions, so i can compare them between servers. My problem is that the stale sessions between the servers are different.
Since you're doing RADIUS replication, packets can be lost. If you were doing database replication, this wouldn't happen.
Usually, server 2 and 3 have more stale sessions than server 1. This would tell me that there are accounting packets not processed or missing between server 1 and server 2 and 3.
Or, they're received in a different order, and that causes problems.
First thing i did was looking at the networking side of things. All interfaces between the servers run clean, they are all connected to the same switch.
So that should be fine.
So i am trying to troubleshoot what is happeing with these packets. The questions i have are:
1) My copy acct-to-home-server for each server configuration is as follows: - log all the incoming packets to a detail file. -copy-acct-to-home-server server reads the detail file and updates the proxy-to-realm control attribute so the packet gets proxied to the other server. Acct always returns ok.
That should work.
If i understand this correctly, in this configuration the packets are "Sent and forgotten".
No. See the debug output. The packets are retransmitted until the server receives a reply.
So if a packet fails to be received or processed by server2 or server3, it wont be retried. The fact that all the detail files are empty makes me think that this is the way the server is working right now.
i.e. all packets have received replies.
Any ideas on how could i implement a simple "retry 3 times" policy?
The server does this already.
My issue is that since i proxy the packet, and and always return ok on accounting, i am not sure where to put the "IF packet failed to ACK, return fail". From what i see, it cannot be done in the proxy section, unless i do something like: realm SERVER1 {
}
It's put into the Post-Proxy-Type Fail section.
2) I dont see anything in the logs that would indicate there is a problem with packets being sent to the other to servers. How can i catch these issues from the logs or debug perspective?
If there's an error, it will be logged to radius.log.
the only thing i can think of is to turn debug on for all the packets with packet-source-destination equals "server 2" and server 3" in server one, and go from there, but any suggestions would be appreciated, as that would basically be my whole radius acct traffic.
It's RADIUS... it's imperfect. If you want more reliable replication, use database replication. But that has it's own set of issues. Alan DeKok.
Hey Alan, Thanks again for your help. I have a couple of questions about some things you commented:
Usually, server 2 and 3 have more stale sessions than server 1. This would tell me that there are accounting packets not processed or missing between server 1 and server 2 and 3.
Or, they're received in a different order, and that causes problems.
Could you explain a little bit more what kind of problems could this cause? Is it receiving packets in a different order for that user/session, or just receiving packets in a different order from that server? I would like to see it there is a way to minimize this behaviour. I read that the server has support for radius over tcp. I don't know anything about it, but woud it be possible to proxy between servers over tcp? Would that solve the reordering issue? The second question is about the following comment:
Now, with some NAS i also see issues where i miss accounting stop packets, and i end up with stale sessions. I am working on solving that, but in the mean time i run a script on all three servers that queries all NAS for active sessions and marks the stale sessions as Terminated.
Why? Just run the script once, and have it write RADIUS packets to a detail file. Those can be replicated to every server.
This was my first idea all along, and i have the code ready. i ran into an issue though, I started using the server with the default calculation for unique accounting session: Acct-Unique-Session-Id := "%{md5:%{User-Name},%{Acct-Session-ID},%{%{NAS-IPv6-Address}:-%{NAS-IP-Address}},%{NAS-Identifier},%{NAS-Port-ID},%{NAS-Port}}" Now, the problem that i have is that some of my NAS send both NAS-port and NAS-Port-ID. Nas-port is not on the accounting database, so i cannot really add it to the attribute, and that makes the accounting session ID different. I would also prefer not to add it, as i dont use it for anything. The options that i see are: -Force all my NAS to not sent NAS-port (I am working on this, but it is not solved yet. I am not sure how to do it yet, or if it is possible) -Change the calculation so it does not include the nas-port attribute. This would be my preferred option, but in order to do this (since i already have sessions opened), i also need to update my database with the new calculation. How is that md5 calculated? Is it a plain md5 of the concatenation of the fields? -I also tried to just put the acct_unique_session_id attribute in the acct stop request, but it looks like the server overwrites it. Also, that is the proper behavior, so i prefer it that way Thank you very much for your help! Regards On Wed, Jul 8, 2015 at 9:35 AM, Alan DeKok <aland@deployingradius.com> wrote:
On Jul 7, 2015, at 7:32 AM, Ricardo Larrañaga <ricardo.larranaga@gmail.com> wrote:
Right now, all my NAS are only pointing to one server (Lets call it server1). So server2 and server3 do not receive authentication requests, they just receive accounting packets from server1.
That's simple.
Now, with some NAS i also see issues where i miss accounting stop packets, and i end up with stale sessions. I am working on solving that, but in the mean time i run a script on all three servers that queries all NAS for active sessions and marks the stale sessions as Terminated.
Why? Just run the script once, and have it write RADIUS packets to a detail file. Those can be replicated to every server.
My script logs the stale sessions, so i can compare them between servers. My problem is that the stale sessions between the servers are different.
Since you're doing RADIUS replication, packets can be lost. If you were doing database replication, this wouldn't happen.
Usually, server 2 and 3 have more stale sessions than server 1. This would tell me that there are accounting packets not processed or missing between server 1 and server 2 and 3.
Or, they're received in a different order, and that causes problems.
First thing i did was looking at the networking side of things. All interfaces between the servers run clean, they are all connected to the same switch.
So that should be fine.
So i am trying to troubleshoot what is happeing with these packets. The questions i have are:
1) My copy acct-to-home-server for each server configuration is as follows: - log all the incoming packets to a detail file. -copy-acct-to-home-server server reads the detail file and updates the proxy-to-realm control attribute so the packet gets proxied to the other server. Acct always returns ok.
That should work.
If i understand this correctly, in this configuration the packets are "Sent and forgotten".
No. See the debug output. The packets are retransmitted until the server receives a reply.
So if a packet fails to be received or processed by server2 or server3, it wont be retried. The fact that all the detail files are empty makes me think that this is the way the server is working right now.
i.e. all packets have received replies.
Any ideas on how could i implement a simple "retry 3 times" policy?
The server does this already.
My issue is that since i proxy the packet, and and always return ok on accounting, i am not sure where to put the "IF packet failed to ACK, return fail". From what i see, it cannot be done in the proxy section, unless i do something like: realm SERVER1 {
}
It's put into the Post-Proxy-Type Fail section.
2) I dont see anything in the logs that would indicate there is a problem with packets being sent to the other to servers. How can i catch these issues from the logs or debug perspective?
If there's an error, it will be logged to radius.log.
the only thing i can think of is to turn debug on for all the packets with packet-source-destination equals "server 2" and server 3" in server one, and go from there, but any suggestions would be appreciated, as that would basically be my whole radius acct traffic.
It's RADIUS... it's imperfect. If you want more reliable replication, use database replication. But that has it's own set of issues.
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On Wed, Jul 8, 2015 at 9:35 AM, Alan DeKok <aland@deployingradius.com> wrote:
On Jul 7, 2015, at 7:32 AM, Ricardo Larrañaga <ricardo.larranaga@gmail.com> wrote:
Right now, all my NAS are only pointing to one server (Lets call it server1). So server2 and server3 do not receive authentication requests, they just receive accounting packets from server1.
That's simple.
Now, with some NAS i also see issues where i miss accounting stop packets, and i end up with stale sessions. I am working on solving that, but in the mean time i run a script on all three servers that queries all NAS for active sessions and marks the stale sessions as Terminated.
Why? Just run the script once, and have it write RADIUS packets to a detail file. Those can be replicated to every server.
My script logs the stale sessions, so i can compare them between servers. My problem is that the stale sessions between the servers are different.
Since you're doing RADIUS replication, packets can be lost. If you were doing database replication, this wouldn't happen.
Usually, server 2 and 3 have more stale sessions than server 1. This would tell me that there are accounting packets not processed or missing between server 1 and server 2 and 3.
Or, they're received in a different order, and that causes problems.
First thing i did was looking at the networking side of things. All interfaces between the servers run clean, they are all connected to the same switch.
So that should be fine.
So i am trying to troubleshoot what is happeing with these packets. The questions i have are:
1) My copy acct-to-home-server for each server configuration is as follows: - log all the incoming packets to a detail file. -copy-acct-to-home-server server reads the detail file and updates the proxy-to-realm control attribute so the packet gets proxied to the other server. Acct always returns ok.
That should work.
If i understand this correctly, in this configuration the packets are "Sent and forgotten".
No. See the debug output. The packets are retransmitted until the server receives a reply.
So if a packet fails to be received or processed by server2 or server3, it wont be retried. The fact that all the detail files are empty makes me think that this is the way the server is working right now.
i.e. all packets have received replies.
Any ideas on how could i implement a simple "retry 3 times" policy?
The server does this already.
My issue is that since i proxy the packet, and and always return ok on accounting, i am not sure where to put the "IF packet failed to ACK, return fail". From what i see, it cannot be done in the proxy section, unless i do something like: realm SERVER1 {
}
It's put into the Post-Proxy-Type Fail section.
2) I dont see anything in the logs that would indicate there is a problem with packets being sent to the other to servers. How can i catch these issues from the logs or debug perspective?
If there's an error, it will be logged to radius.log.
the only thing i can think of is to turn debug on for all the packets with packet-source-destination equals "server 2" and server 3" in server one, and go from there, but any suggestions would be appreciated, as that would basically be my whole radius acct traffic.
It's RADIUS... it's imperfect. If you want more reliable replication, use database replication. But that has it's own set of issues.
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On Jul 10, 2015, at 4:06 PM, Ricardo Larrañaga <ricardo.larranaga@gmail.com> wrote:
Or, they're received in a different order, and that causes problems.
Could you explain a little bit more what kind of problems could this cause?
Account sessions go start / update / update / stop. If the packets are received in a different order, things can go wrong.
Is it receiving packets in a different order for that user/session, or just receiving packets in a different order from that server?
All of that.
I would like to see it there is a way to minimize this behaviour. I read that the server has support for radius over tcp. I don't know anything about it, but woud it be possible to proxy between servers over tcp? Would that solve the reordering issue?
Maybe. But I wouldn't bother. Making the system robust will be better in the long run.
Why? Just run the script once, and have it write RADIUS packets to a detail file. Those can be replicated to every server.
This was my first idea all along, and i have the code ready. i ran into an issue though, I started using the server with the default calculation for unique accounting session:
Acct-Unique-Session-Id := "%{md5:%{User-Name},%{Acct-Session-ID},%{%{NAS-IPv6-Address}:-%{NAS-IP-Address}},%{NAS-Identifier},%{NAS-Port-ID},%{NAS-Port}}"
Now, the problem that i have is that some of my NAS send both NAS-port and NAS-Port-ID. Nas-port is not on the accounting database, so i cannot really add it to the attribute, and that makes the accounting session ID different. I would also prefer not to add it, as i dont use it for anything.
If the Acct-Session-Id attribute is unique, you can just use that, and ignore Acct-Unique-Session-Id.
The options that i see are:
-Force all my NAS to not sent NAS-port (I am working on this, but it is not solved yet. I am not sure how to do it yet, or if it is possible)
Maybe.
-Change the calculation so it does not include the nas-port attribute. This would be my preferred option, but in order to do this (since i already have sessions opened), i also need to update my database with the new calculation. How is that md5 calculated? Is it a plain md5 of the concatenation of the fields?
Yes. See the debug output. It prints out the input to the MD5, and the output MD5 hash. Alan DeKok.
Hey Alan, thank you for your help. What do you mean by "making the system more robust"? I thought that using tcp would do that, unless you are talking about using master-master replication on the db? Since we were talking about the radius protocol being intrinsically error prone, y dont know if i understand what you mean by making the system more robust. For example, in an smp system, packets might get reordered on reception, and i wouldnt see that as lack of robustness. In my case, i have servers directly connected to switches that show clean interfaces, so i am not sure what else i could do to increase robustness. Thanks a lot. Regards On Jul 10, 2015 5:54 PM, "Alan DeKok" <aland@deployingradius.com> wrote:
On Jul 10, 2015, at 4:06 PM, Ricardo Larrañaga < ricardo.larranaga@gmail.com> wrote:
Or, they're received in a different order, and that causes problems.
Could you explain a little bit more what kind of problems could this cause?
Account sessions go start / update / update / stop. If the packets are received in a different order, things can go wrong.
Is it receiving packets in a different order for that user/session, or just receiving packets in a different order from that server?
All of that.
I would like to see it there is a way to minimize this behaviour. I read that the server has support for radius over tcp. I don't know anything about it, but woud it be possible to proxy between servers over tcp? Would that solve the reordering issue?
Maybe. But I wouldn't bother. Making the system robust will be better in the long run.
Why? Just run the script once, and have it write RADIUS packets to a detail file. Those can be replicated to every server.
This was my first idea all along, and i have the code ready. i ran into an issue though, I started using the server with the default calculation for unique accounting session:
Acct-Unique-Session-Id :=
"%{md5:%{User-Name},%{Acct-Session-ID},%{%{NAS-IPv6-Address}:-%{NAS-IP-Address}},%{NAS-Identifier},%{NAS-Port-ID},%{NAS-Port}}"
Now, the problem that i have is that some of my NAS send both NAS-port
and
NAS-Port-ID. Nas-port is not on the accounting database, so i cannot really add it to the attribute, and that makes the accounting session ID different. I would also prefer not to add it, as i dont use it for anything.
If the Acct-Session-Id attribute is unique, you can just use that, and ignore Acct-Unique-Session-Id.
The options that i see are:
-Force all my NAS to not sent NAS-port (I am working on this, but it is not solved yet. I am not sure how to do it yet, or if it is possible)
Maybe.
-Change the calculation so it does not include the nas-port attribute. This would be my preferred option, but in order to do this (since i already have sessions opened), i also need to update my database with the new calculation. How is that md5 calculated? Is it a plain md5 of the concatenation of the fields?
Yes. See the debug output. It prints out the input to the MD5, and the output MD5 hash.
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On Jul 10, 2015, at 5:09 PM, Ricardo Larrañaga <ricardo.larranaga@gmail.com> wrote:
What do you mean by "making the system more robust"? I thought that using tcp would do that, unless you are talking about using master-master replication on the db?
No. I mean make the *system* more robust. Ensure that the RADIUS servers all behave exactly the same way. That they can handle packets with / without NAS-Port. That they all work correctly all of the time.
Since we were talking about the radius protocol being intrinsically error prone, y dont know if i understand what you mean by making the system more robust.
See your comments about NAS-Port. You said that the systems behaved differently depending on what they got from the NAS, and that this caused problems. So... make the systems robust to getting different data from the NAS. It's about understanding what the systems do, and what the error cases are. Then, ensuring that the error cases are handled in a sane / consistent way, and making sure that the systems behaviour correctly, and consistently. Alan DeKok.
Hi Again Alan: I think something got confused along the way. While i do have a problem of my acct unique id being different depending on if the acct start comes with nas port of not, this behaviour is the same in all servers, and it is really not causing any issues other than the fact that i cannot really disconnect the user by sending and accounting packet. The solution for this is just changing the definition of acct-unique-id in all servers. The second problem that i have, which is unrelated with the above is the fact that even though i copy all accounting packets to the other two servers, they are mostlyin sync with server1, but not exactly. We were talking about this possibly being caused by either udp packets being lost or being sent/arriving out of order. Now, this is the part that i am trying to fix. Since there is little i can do about packets being reordered. and i do not see lost packets in my network, i was wondering if there is something else i can do to resolve this. The only thing that i can think of is trying to proxy over TCP, but i am definetely open not any suggestions. Thanks!. Regards On Jul 10, 2015 6:30 PM, "Alan DeKok" <aland@deployingradius.com> wrote:
On Jul 10, 2015, at 5:09 PM, Ricardo Larrañaga < ricardo.larranaga@gmail.com> wrote:
What do you mean by "making the system more robust"? I thought that using tcp would do that, unless you are talking about using master-master replication on the db?
No. I mean make the *system* more robust. Ensure that the RADIUS servers all behave exactly the same way. That they can handle packets with / without NAS-Port. That they all work correctly all of the time.
Since we were talking about the radius protocol being intrinsically error prone, y dont know if i understand what you mean by making the system more robust.
See your comments about NAS-Port. You said that the systems behaved differently depending on what they got from the NAS, and that this caused problems. So... make the systems robust to getting different data from the NAS.
It's about understanding what the systems do, and what the error cases are. Then, ensuring that the error cases are handled in a sane / consistent way, and making sure that the systems behaviour correctly, and consistently.
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On Jul 14, 2015, at 9:22 AM, Ricardo Larrañaga <ricardo.larranaga@gmail.com> wrote:
The second problem that i have, which is unrelated with the above is the fact that even though i copy all accounting packets to the other two servers, they are mostlyin sync with server1, but not exactly.
Do the packets get sent to all servers at the same time? Or is one server faster than the other?
We were talking about this possibly being caused by either udp packets being lost or being sent/arriving out of order.
Now, this is the part that i am trying to fix. Since there is little i can do about packets being reordered. and i do not see lost packets in my network, i was wondering if there is something else i can do to resolve this. The only thing that i can think of is trying to proxy over TCP, but i am definetely open not any suggestions.
I'm not sure there's any real fix. If you want consistent DB replication, use DB replication. The RADIUS replication is designed to send RADIUS packets, often across the internet. Alan DeKok.
participants (2)
-
Alan DeKok -
Ricardo Larrañaga