RADIUS failing to start correctly when remote DB is unavailable.
Hi, Good afternoon! I have a setup where RADIUS is set to write to multiple DBs simultaneously. I have set this as follows: 1. Copied the SQL instance in /etc/raddb/mods-enabled/sql and had it renamed and configured accordingly. 2. Called them both 3. In the pool, I have set start=0 4. Set read_client to no Although from debug these seem to be correctly set, I am still getting the message that RADIUS is trying to connect to the DB when it is unavailable. After quite a number of minutes and 3 retries, RADIUS fails to start. I have attached the full debug and included some explanation too along the way. Thanks in advance. Kind Regards, SG
On Oct 11, 2022, at 9:38 AM, Sea Gull <seagull0044@gmail.com> wrote:
Good afternoon! I have a setup where RADIUS is set to write to multiple DBs simultaneously. I have set this as follows:
1. Copied the SQL instance in /etc/raddb/mods-enabled/sql and had it renamed and configured accordingly. 2. Called them both 3. In the pool, I have set start=0 4. Set read_client to no
OK...
Although from debug these seem to be correctly set, I am still getting the message that RADIUS is trying to connect to the DB when it is unavailable.
That's how it works. The only way that FreeRADIUS knows that the DB is unavailable is by trying to connect, and failing.
After quite a number of minutes and 3 retries, RADIUS fails to start. I have attached the full debug and included some explanation too along the way.
The debug log shows that it's starting fine. And that it's trying to connect to the SQL server when FreeRADIUS receives accounting packets. Because that's what you configured FreeRADIUS to do. If you want it to dynamically choose an SQL server based on which one is up, you can put them into a redundant section: redundant { sql1 sql2 } That will choose sql1 until it's down, and then will choose sql2. But it will still try to use sql1, because FreeRADIUS has no idea that it's down. There's no magical connection between SQL and FreeRADIUS which says "SQL database is down". So the server does start, and it works as documented, and it works the only way it *can* work. I'm not sure what else you expect it to do here. My recommendation is that if FreeRADIUS is using a DB, then you should ensure that the DB is up 100% of the time. All of the fail-over, redundant, etc. checks in the server are there only to catch unusual / error cases. If your SQL server is *normally* down, then just configure FreeRADIUS to not use it. Alan DeKok.
Hi Alan, Thank you for your explanation. During this time I have been trying to solve this issue, and have seen that there are two patterns of how this can occur. The first scenario is when RADIUS is receiving an ICMP unreachable packet when trying to connect to the PostgreSQL remote DB. This works as expected, where RADIUS will mark it as down and continue starting. Here users connect instantly without any delays. Below please find logs from debug, showing the time difference from when RADIUS declares the remote DB as unreachable until it is ready to process new requests (3 seconds). 14:15:42.665 (1) sql_remote: EXPAND %{tolower:type.%{%{Acct-Status-Type}:-%{Request-Processing-Stage}}.query} 14:15:42.665 (1) sql_remote: --> type.start.query 14:15:42.665 (1) sql_remote: Using query template 'query' 14:15:42.665 rlm_sql (sql_remote): 0 of 0 connections in use. You may need to increase "spare" 14:15:42.665 rlm_sql (sql_remote): Opening additional connection (0), 1 of 32 pending slots used 14:15:42.665 rlm_sql_postgresql: Connecting using parameters: dbname=tstradius host=xxx.xxx.xxx.xxx user=tstradiususer password=xxx application_name='FreeRADIUS 3.0.25 - radiusd (sql_remote)' 14:15:42.666 rlm_sql_postgresql: Connection failed: could not translate host name "xxx.xxx.xxx.xxx" to address: Name or service not known 14:15:42.666 rlm_sql_postgresql: Socket destructor called, closing socket 14:15:42.666 rlm_sql (sql_remote): Opening connection failed (0) 14:15:42.666 (1) [sql_remote] = fail 14:15:42.666 (1) } # accounting = fail 14:15:42.666 (1) Not sending reply to client. 14:15:42.666 (1) Finished request 14:15:42.666 (1) Cleaning up request packet ID 106 with timestamp +184 14:15:42.675 Waking up in 4.8 seconds. 14:15:47.378 (0) Cleaning up request packet ID 105 with timestamp +184 14:15:47.380 Ready to process requests The second scenario is when access to the remote DB is restricted due to a network outage. This is the issue that I am experiencing, and here RADIUS is taking too long until it notices that the DB is unreachable. It seems that the timeout is either not working or taking too long. Is there a way I can change it? I have tried changing the parameters shown below (in the sql files) however, the issue didn't resolve. # The number of seconds to wait after the server tries # to open a connection, and fails. During this time, # no new connections will be opened. retry_delay = 3 # The lifetime (in seconds) of the connection lifetime = 1 # idle timeout (in seconds). A connection which is # unused for this length of time will be closed. idle_timeout = 6 Also, from the debug below it shows that in this state it takes way longer until it's ready to receive requests (2 minutes in this case). 09:43:36.193 (3) sql_remote: EXPAND %{tolower:type.%{%{Acct-Status-Type}:-%{Request-Processing-Stage}}.query} 09:43:36.194 (3) sql_remote: --> type.start.query 09:43:36.195 (3) sql_remote: Using query template 'query' 09:43:36.196 rlm_sql (sql_remote): 0 of 0 connections in use. You may need to increase "spare" 09:43:36.198 rlm_sql (sql_remote): Opening additional connection (2), 1 of 1 pending slots used 09:43:36.209 rlm_sql_postgresql: Connecting using parameters: dbname=tstradius host=xxx.xxx.xxx.xxx user=tstradiususer password=xxx application_name='FreeRADIUS 3.0.25 - radiusd (sql_remote)' 09:45:46.745 rlm_sql_postgresql: Connection failed: could not connect to server: Connection timed out Is the server running on host "xxx.xxx.xxx.xxx" (x.x.x.x) and accepting TCP/IP connections on port 5432? 09:45:46.766 rlm_sql_postgresql: Socket destructor called, closing socket 09:45:46.767 rlm_sql (sql_remote): Opening connection failed (2) 09:45:46.768 (3) [sql_remote] = fail 09:45:46.769 (3) } # accounting = fail 09:45:46.771 (3) Not sending reply to client. 09:45:46.771 (3) Finished request 09:45:46.773 (3) Cleaning up request packet ID 151 with timestamp +451 09:45:46.774 Ready to process requests What I am trying to achieve is to have the behaviour of the first scenario whenever there is a network outage. Would it be possible to achieve this, please? Thanks again. Kind Regards, Clint On Tue, Oct 11, 2022 at 3:59 PM Alan DeKok <aland@deployingradius.com> wrote:
On Oct 11, 2022, at 9:38 AM, Sea Gull <seagull0044@gmail.com> wrote:
Good afternoon! I have a setup where RADIUS is set to write to multiple DBs simultaneously. I have set this as follows:
1. Copied the SQL instance in /etc/raddb/mods-enabled/sql and had it renamed and configured accordingly. 2. Called them both 3. In the pool, I have set start=0 4. Set read_client to no
OK...
Although from debug these seem to be correctly set, I am still getting the message that RADIUS is trying to connect to the DB when it is unavailable.
That's how it works.
The only way that FreeRADIUS knows that the DB is unavailable is by trying to connect, and failing.
After quite a number of minutes and 3 retries, RADIUS fails to start. I have attached the full debug and included some explanation too along the way.
The debug log shows that it's starting fine. And that it's trying to connect to the SQL server when FreeRADIUS receives accounting packets. Because that's what you configured FreeRADIUS to do.
If you want it to dynamically choose an SQL server based on which one is up, you can put them into a redundant section:
redundant { sql1 sql2 }
That will choose sql1 until it's down, and then will choose sql2. But it will still try to use sql1, because FreeRADIUS has no idea that it's down. There's no magical connection between SQL and FreeRADIUS which says "SQL database is down".
So the server does start, and it works as documented, and it works the only way it *can* work. I'm not sure what else you expect it to do here.
My recommendation is that if FreeRADIUS is using a DB, then you should ensure that the DB is up 100% of the time. All of the fail-over, redundant, etc. checks in the server are there only to catch unusual / error cases. If your SQL server is *normally* down, then just configure FreeRADIUS to not use it.
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On Oct 31, 2022, at 10:53 AM, Sea Gull <seagull0044@gmail.com> wrote:
The second scenario is when access to the remote DB is restricted due to a network outage. This is the issue that I am experiencing, and here RADIUS is taking too long until it notices that the DB is unreachable.
If your network is broken, the solution is not to fix FreeRADIUS. The solution is to fix the network. FreeRADIUS uses the libpq libraries to do all of the underlying "connect to postgres" functionality. Those libraries are limited in what they can do.
It seems that the timeout is either not working or taking too long. Is there a way I can change it? I have tried changing the parameters shown below (in the sql files) however, the issue didn't resolve.
We pass the connect_timeout parameter to the libpq library. If that library doesn't respect it, there isn't much we can do.
What I am trying to achieve is to have the behaviour of the first scenario whenever there is a network outage. Would it be possible to achieve this, please?
Fix your network so that it doesn't go down. There really isn't any other way to do this. It makes no sense to say "I want FreeRADIUS to be reliable, but the database and/or the network is just going to randomly disappear". That's not reliable. You can't build a reliable system on an unreliable foundation. Alan DeKok.
Hi Alan, Thanks again for your time. I understand your comments, and you are right. However, in my setup the RADIUS Authentication server, and the RADIUS Accounting server are two different physical servers, and as you might well know network disruptions can happen. Also, it is impossible for us to have users unable to authenticate, because the accounting server is down as this will stop thousands of users from working. Is there a way we can have RADIUS Authentication working when the accounting servers are down or unreachable, please? Thanks again for your help. Kind regards, SG On Mon, Oct 31, 2022 at 5:07 PM Alan DeKok <aland@deployingradius.com> wrote:
On Oct 31, 2022, at 10:53 AM, Sea Gull <seagull0044@gmail.com> wrote:
The second scenario is when access to the remote DB is restricted due to a network outage. This is the issue that I am experiencing, and here RADIUS is taking too long until it notices that the DB is unreachable.
If your network is broken, the solution is not to fix FreeRADIUS. The solution is to fix the network.
FreeRADIUS uses the libpq libraries to do all of the underlying "connect to postgres" functionality. Those libraries are limited in what they can do.
It seems that the timeout is either not working or taking too long. Is there a way I can change it? I have tried changing the parameters shown below (in the sql files) however, the issue didn't resolve.
We pass the connect_timeout parameter to the libpq library. If that library doesn't respect it, there isn't much we can do.
What I am trying to achieve is to have the behaviour of the first scenario whenever there is a network outage. Would it be possible to achieve this, please?
Fix your network so that it doesn't go down.
There really isn't any other way to do this. It makes no sense to say "I want FreeRADIUS to be reliable, but the database and/or the network is just going to randomly disappear".
That's not reliable. You can't build a reliable system on an unreliable foundation.
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On Dec 19, 2022, at 6:48 AM, Sea Gull <seagull0044@gmail.com> wrote:
Thanks again for your time. I understand your comments, and you are right. However, in my setup the RADIUS Authentication server, and the RADIUS Accounting server are two different physical servers, and as you might well know network disruptions can happen. Also, it is impossible for us to have users unable to authenticate, because the accounting server is down as this will stop thousands of users from working.
Is there a way we can have RADIUS Authentication working when the accounting servers are down or unreachable, please?
So... the servers are physically different, but if one stops working, the other one stops working, too? Either the servers are magic, or you're not describing what's actually going on. If you can't describe what the current system is doing or how it works, then you can't configure it to do what you want. Alan DeKok.
Hi Alan, Yes indeed. RADIUS Authentication and Accounting servers are two different physical machines, which are connected to each other. Then, I have 2 instances of the SQL configuration, one that writes to a local database and another that writes to a remote database. The reason for this setup is that the remote database is handled by a different team and this cannot be eliminated. Now the issue is that when the remote database is for any reason unreachable (could be firewall, maintenance, etc..) RADIUS will no longer authenticate users. So what I'd like to establish is if there's a way for RADIUS to continue with the authentication process of users without having them accounted for, when the remote database is unavailable. Hope this helps, if a diagram will be more helpful please let me know and I will provide you with one. Thanks again, SG On Mon, Dec 19, 2022 at 2:44 PM Alan DeKok <aland@deployingradius.com> wrote:
On Dec 19, 2022, at 6:48 AM, Sea Gull <seagull0044@gmail.com> wrote:
Thanks again for your time. I understand your comments, and you are right. However, in my setup the RADIUS Authentication server, and the RADIUS Accounting server are two different physical servers, and as you might well know network disruptions can happen. Also, it is impossible for us to have users unable to authenticate, because the accounting server is down as this will stop thousands of users from working.
Is there a way we can have RADIUS Authentication working when the accounting servers are down or unreachable, please?
So... the servers are physically different, but if one stops working, the other one stops working, too?
Either the servers are magic, or you're not describing what's actually going on.
If you can't describe what the current system is doing or how it works, then you can't configure it to do what you want.
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On 12/01/2023 08:54, Sea Gull wrote:
Yes indeed. RADIUS Authentication and Accounting servers are two different physical machines, which are connected to each other. Then, I have 2 instances of the SQL configuration, one that writes to a local database and another that writes to a remote database. The reason for this setup is that the remote database is handled by a different team and this cannot be eliminated. Now the issue is that when the remote database is for any reason unreachable (could be firewall, maintenance, etc..) RADIUS will no longer authenticate users. So what I'd like to establish is if there's a way for RADIUS to continue with the authentication process of users without having them accounted for, when the remote database is unavailable.
There are two parts to this. To allow FreeRADIUS to start when a database is down, set the "start" value of the connection pool for that database to 0. You will also want to ensure that the "read_clients" option is set to "no", otherwise FreeRADIUS will still attempt to make a connection to the database during startup to read the clients. If you do require the clients to be loaded from that database then life gets more complex, and you have to look at dynamic clients to allow the server to start without a database connection. Secondly you need to look at your policy for authentication. If that performs any calls to the database which is potentially unavailable, you need to amend your policy to allow for failure to be acceptable e.g. if that instance of the sql module is called sql-remote: sql-remote { fail = 1 } if (fail) { ... policy to handle database failure ok } otherwise the failure of the remote database will return a fail and cause an Access-Reject to be sent. Nick
On Jan 12, 2023, at 3:54 AM, Sea Gull <seagull0044@gmail.com> wrote:
Yes indeed. RADIUS Authentication and Accounting servers are two different physical machines,
OK.
which are connected to each other.
How? I asked you to describe the system, and you don't do that.
... Now the issue is that when the remote database is for any reason unreachable (could be firewall, maintenance, etc..) RADIUS will no longer authenticate users.
Nick has told you how to help somewhat with that. But realistically, if the authentication server *needs* the remote database to authenticate users, then when it's down... no one can authenticate. This is how computers work. If the authentication server doesn't need the remote database, then why is it using the remote database?
So what I'd like to establish is if there's a way for RADIUS to continue with the authentication process of users without having them accounted for, when the remote database is unavailable.
Either you're not describing what you have, or you're using the wrong words to describe things. Reading between the lines, what I *think* you mean is: * RADIUS authentication and accounting are run on the same machine, by the same FreeRADIUS * the local database is used to authenticate users * the local database is used to store accounting data * the remote database is also used to store accounting data Then, when the remote database is down, the RADIUS server blocks trying to reconnect, and it can't make progress on anything. So authentication stops working, because all of the SQL connections are blocked trying to write accounting packets. So if you describe the system accurately and correctly, it helps a lot. The solution here is "don't have the remote database go down". If the remote database is necessary, then it should stay up. Another solution is to run a second copy of FreeRADIUS to write to the remote database. See sites-available/decoupled-accounting. The idea is that the main RADIUS server doesn't write to the remote DB. Instead it writes to a detail file. The second RADIUS server then reads from the detail file, and writes to the remote DB. That way if the remote DB is down, the detail file simply gets bigger. The main RADIUS server isn't blocked, and can continue running. But this is just a guess, because you refuse to describe what your system is doing, or how it's configured. Alan DeKok.
Hi Nick, Alan Thank you both for your feedback. The first points mentioned by Nick to set the read-clients as no, I have already implemented that earlier. I added the following: sql-remote { fail = 1 } if (fail) { ... policy to handle database failure ok } This seemed to do the trick however, only when the connection timeout is set explicitly in my sql_local and sql_remote files. Now, if either of the DBs are unavailable, it will still proceed with authorization. Accounting packets will be written to the details file, which will populate the DBs when available. Thanks and Regards, SG On Thu, Jan 12, 2023 at 3:14 PM Alan DeKok <aland@deployingradius.com> wrote:
On Jan 12, 2023, at 3:54 AM, Sea Gull <seagull0044@gmail.com> wrote:
Yes indeed. RADIUS Authentication and Accounting servers are two different physical machines,
OK.
which are connected to each other.
How?
I asked you to describe the system, and you don't do that.
... Now the issue is that when the remote database is for any reason unreachable (could be firewall, maintenance, etc..) RADIUS will no longer authenticate users.
Nick has told you how to help somewhat with that.
But realistically, if the authentication server *needs* the remote database to authenticate users, then when it's down... no one can authenticate. This is how computers work.
If the authentication server doesn't need the remote database, then why is it using the remote database?
So what I'd like to establish is if there's a way for RADIUS to continue with the authentication process of users without having them accounted for, when the remote database is unavailable.
Either you're not describing what you have, or you're using the wrong words to describe things.
Reading between the lines, what I *think* you mean is:
* RADIUS authentication and accounting are run on the same machine, by the same FreeRADIUS
* the local database is used to authenticate users
* the local database is used to store accounting data
* the remote database is also used to store accounting data
Then, when the remote database is down, the RADIUS server blocks trying to reconnect, and it can't make progress on anything. So authentication stops working, because all of the SQL connections are blocked trying to write accounting packets.
So if you describe the system accurately and correctly, it helps a lot.
The solution here is "don't have the remote database go down". If the remote database is necessary, then it should stay up.
Another solution is to run a second copy of FreeRADIUS to write to the remote database. See sites-available/decoupled-accounting.
The idea is that the main RADIUS server doesn't write to the remote DB. Instead it writes to a detail file. The second RADIUS server then reads from the detail file, and writes to the remote DB.
That way if the remote DB is down, the detail file simply gets bigger. The main RADIUS server isn't blocked, and can continue running.
But this is just a guess, because you refuse to describe what your system is doing, or how it's configured.
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
participants (3)
-
Alan DeKok -
Nick Porter -
Sea Gull