Cannot increase open file descriptors above 1024
I use FreeRADIUS Version 3.0.20 as both proxy and (auth/acct) server. I have more than 80 realms with over 180 clients connected. Every one of them is redirecting to a different site file with separate sql module to a seperate database in the same db server. My problem is with this large amount of databases and tables i hit the fixed limit of 1024 open file descriptors and the radius.log file shows: mysql error: can't create unix socket (24) I double checked with: ls /proc/${freeradius_pid}/fd | wc -l it prints 1024 that's why I am sure its problem with the open files limit I tried to increase it manually but with no luck at all. Whenever I increase it the freeradius server stops working. So is there something I can do to increase the file descriptors limit ? Or am I doing something wrong?
On Nov 17, 2022, at 11:04 AM, Ibrahemoo Ghost <ibrahemooghost@gmail.com> wrote:
I use FreeRADIUS Version 3.0.20 as both proxy and (auth/acct) server. I have more than 80 realms with over 180 clients connected. Every one of them is redirecting to a different site file with separate sql module to a seperate database in the same db server.
Multiple SQL module can share a pool of connections to one database. See the documentation in mods-available/sql.
My problem is with this large amount of databases and tables i hit the fixed limit of 1024 open file descriptors and the radius.log file shows: mysql error: can't create unix socket (24)
There's just no reason to use 1024 sockets. If the SQL module all share the same pool, then the number of file descriptors will go down enormously.
I double checked with: ls /proc/${freeradius_pid}/fd | wc -l it prints 1024 that's why I am sure its problem with the open files limit
That's an OS limit. See your OS documentation for how to fix it.
I tried to increase it manually but with no luck at all. Whenever I increase it the freeradius server stops working.
I have no idea what that means.
So is there something I can do to increase the file descriptors limit ? Or am I doing something wrong?
Configure the SQL modules to share connection pools as documented. Alan DeKok.
Thanks for your reply and your hard work, really appreciated. I will try to share connection pools as you suggested> On Thu, Nov 17, 2022 at 8:17 PM Alan DeKok <aland@deployingradius.com> wrote:
On Nov 17, 2022, at 11:04 AM, Ibrahemoo Ghost <ibrahemooghost@gmail.com> wrote:
I use FreeRADIUS Version 3.0.20 as both proxy and (auth/acct) server. I have more than 80 realms with over 180 clients connected. Every one of them is redirecting to a different site file with separate sql module to a seperate database in the same db server.
Multiple SQL module can share a pool of connections to one database. See the documentation in mods-available/sql.
My problem is with this large amount of databases and tables i hit the fixed limit of 1024 open file descriptors and the radius.log file shows: mysql error: can't create unix socket (24)
There's just no reason to use 1024 sockets. If the SQL module all share the same pool, then the number of file descriptors will go down enormously.
I double checked with: ls /proc/${freeradius_pid}/fd | wc -l it prints 1024 that's why I am sure its problem with the open files limit
That's an OS limit. See your OS documentation for how to fix it.
I tried to increase it manually but with no luck at all. Whenever I increase it the freeradius server stops working.
I have no idea what that means.
So is there something I can do to increase the file descriptors limit ? Or am I doing something wrong?
Configure the SQL modules to share connection pools as documented.
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
I tried your suggestion by configuring sql modules to share connection pools. But now all sql modules use the same database which is not what i want. I am expecting this is not the normal behavior because why would i want to use the same database for different sql modules. On Thu, Nov 17, 2022 at 8:17 PM Alan DeKok <aland@deployingradius.com> wrote:
On Nov 17, 2022, at 11:04 AM, Ibrahemoo Ghost <ibrahemooghost@gmail.com> wrote:
I use FreeRADIUS Version 3.0.20 as both proxy and (auth/acct) server. I have more than 80 realms with over 180 clients connected. Every one of them is redirecting to a different site file with separate sql module to a seperate database in the same db server.
Multiple SQL module can share a pool of connections to one database. See the documentation in mods-available/sql.
My problem is with this large amount of databases and tables i hit the fixed limit of 1024 open file descriptors and the radius.log file shows: mysql error: can't create unix socket (24)
There's just no reason to use 1024 sockets. If the SQL module all share the same pool, then the number of file descriptors will go down enormously.
I double checked with: ls /proc/${freeradius_pid}/fd | wc -l it prints 1024 that's why I am sure its problem with the open files limit
That's an OS limit. See your OS documentation for how to fix it.
I tried to increase it manually but with no luck at all. Whenever I increase it the freeradius server stops working.
I have no idea what that means.
So is there something I can do to increase the file descriptors limit ? Or am I doing something wrong?
Configure the SQL modules to share connection pools as documented.
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
may be you can increase on /etc/security/limits.conf the values nofile > 1024 to solve the issue on OS Linux Best Regards Pablo Umanzor El jue, 17 nov 2022 a las 13:04, Ibrahemoo Ghost (<ibrahemooghost@gmail.com>) escribió:
I use FreeRADIUS Version 3.0.20 as both proxy and (auth/acct) server. I have more than 80 realms with over 180 clients connected. Every one of them is redirecting to a different site file with separate sql module to a seperate database in the same db server.
My problem is with this large amount of databases and tables i hit the fixed limit of 1024 open file descriptors and the radius.log file shows: mysql error: can't create unix socket (24)
I double checked with: ls /proc/${freeradius_pid}/fd | wc -l it prints 1024 that's why I am sure its problem with the open files limit
I tried to increase it manually but with no luck at all. Whenever I increase it the freeradius server stops working.
So is there something I can do to increase the file descriptors limit ? Or am I doing something wrong? - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
I tried and it does not increase by itself, I had to also increase it on /lib/systemd/system/freeradius.service. [Service] LimitNOFILE=4000 but after that the freeradius service wont start, and that's why I thought there is internal limit in freeradius for the file descriptors. On 17 Nov 2022 Thu at 9:42 PM Pablo Umanzor <info@redlibre.cl> wrote:
may be you can increase on /etc/security/limits.conf
the values
nofile > 1024
to solve the issue on OS Linux
Best Regards Pablo Umanzor
El jue, 17 nov 2022 a las 13:04, Ibrahemoo Ghost (< ibrahemooghost@gmail.com>) escribió:
I use FreeRADIUS Version 3.0.20 as both proxy and (auth/acct) server. I have more than 80 realms with over 180 clients connected. Every one of them is redirecting to a different site file with separate sql module to a seperate database in the same db server.
My problem is with this large amount of databases and tables i hit the fixed limit of 1024 open file descriptors and the radius.log file shows: mysql error: can't create unix socket (24)
I double checked with: ls /proc/${freeradius_pid}/fd | wc -l it prints 1024 that's why I am sure its problem with the open files limit
I tried to increase it manually but with no luck at all. Whenever I increase it the freeradius server stops working.
So is there something I can do to increase the file descriptors limit ? Or am I doing something wrong? - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
try /etc/security/limits.conf add freerad soft nofile 31000 freerad hard nofile 32000 i use freerad because this user launch freeradius daemon, it depends who is the owner (could be sometimes root but not recomend) you can check with this command ulimii -a if the number open files it changed also reboot the box if the previous step it doesn't work Best Regards Pablo Umanzor El jue, 17 nov 2022 a las 16:12, Ibrahemoo Ghost (<ibrahemooghost@gmail.com>) escribió:
I tried and it does not increase by itself, I had to also increase it on /lib/systemd/system/freeradius.service.
[Service] LimitNOFILE=4000
but after that the freeradius service wont start, and that's why I thought there is internal limit in freeradius for the file descriptors.
On 17 Nov 2022 Thu at 9:42 PM Pablo Umanzor <info@redlibre.cl> wrote:
may be you can increase on /etc/security/limits.conf
the values
nofile > 1024
to solve the issue on OS Linux
Best Regards Pablo Umanzor
El jue, 17 nov 2022 a las 13:04, Ibrahemoo Ghost (< ibrahemooghost@gmail.com>) escribió:
I use FreeRADIUS Version 3.0.20 as both proxy and (auth/acct) server. I have more than 80 realms with over 180 clients connected. Every one of them is redirecting to a different site file with separate sql module to a seperate database in the same db server.
My problem is with this large amount of databases and tables i hit the fixed limit of 1024 open file descriptors and the radius.log file shows: mysql error: can't create unix socket (24)
I double checked with: ls /proc/${freeradius_pid}/fd | wc -l it prints 1024 that's why I am sure its problem with the open files limit
I tried to increase it manually but with no luck at all. Whenever I increase it the freeradius server stops working.
So is there something I can do to increase the file descriptors limit ? Or am I doing something wrong? - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Thanks for your reply. I actually added freerad limits on /etc/security/limits.conf then rebooted and also tried to run freeradius as root. Then I discovered that increasing the limit for the user only does not affect the service itself, By examining the soft limit file values in /proc/$pid_of_freeradius/limits (1024) The only way to increase the limit is by also increasing it in the service file, but as I said after that the service won't start anymore. Actually now I am trying another way to solve the base problem, As Alan suggested, is to use a shared pool across sql instances to decrease the required open files altogether. Best Regards Ibrahem Ghost On Thu, Nov 17, 2022 at 10:18 PM Pablo Umanzor <info@redlibre.cl> wrote:
try
/etc/security/limits.conf
add
freerad soft nofile 31000 freerad hard nofile 32000
i use freerad because this user launch freeradius daemon, it depends who is the owner (could be sometimes root but not recomend)
you can check with this command
ulimii -a
if the number open files it changed
also reboot the box if the previous step it doesn't work
Best Regards Pablo Umanzor
El jue, 17 nov 2022 a las 16:12, Ibrahemoo Ghost (<ibrahemooghost@gmail.com>) escribió:
I tried and it does not increase by itself, I had to also increase it on /lib/systemd/system/freeradius.service.
[Service] LimitNOFILE=4000
but after that the freeradius service wont start, and that's why I thought there is internal limit in freeradius for the file descriptors.
On 17 Nov 2022 Thu at 9:42 PM Pablo Umanzor <info@redlibre.cl> wrote:
may be you can increase on /etc/security/limits.conf
the values
nofile > 1024
to solve the issue on OS Linux
Best Regards Pablo Umanzor
El jue, 17 nov 2022 a las 13:04, Ibrahemoo Ghost (< ibrahemooghost@gmail.com>) escribió:
I use FreeRADIUS Version 3.0.20 as both proxy and (auth/acct) server. I have more than 80 realms with over 180 clients connected. Every one of them is redirecting to a different site file with separate sql module to a seperate database in the same db server.
My problem is with this large amount of databases and tables i hit the fixed limit of 1024 open file descriptors and the radius.log file shows: mysql error: can't create unix socket (24)
I double checked with: ls /proc/${freeradius_pid}/fd | wc -l it prints 1024 that's why I am sure its problem with the open files limit
I tried to increase it manually but with no luck at all. Whenever I increase it the freeradius server stops working.
So is there something I can do to increase the file descriptors limit ? Or am I doing something wrong? - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On Nov 17, 2022, at 2:30 PM, Ibrahemoo Ghost <ibrahemooghost@gmail.com> wrote:
I actually added freerad limits on /etc/security/limits.conf then rebooted and also tried to run freeradius as root. Then I discovered that increasing the limit for the user only does not affect the service itself, By examining the soft limit file values in /proc/$pid_of_freeradius/limits (1024)
The only way to increase the limit is by also increasing it in the service file, but as I said after that the service won't start anymore.
What does that mean? Is there any error? Can you run it in debug mode to see what's going on? There is nothing in FreeRADIUS which says "don't start if the OS allows more than 1024 file descriptors". Alan DeKok.
Is there any error? Can you run it in debug mode to see what's going on?
I am running under ubuntu server 20, After raising the file descriptors limit, When i run freeradius as service (in multithreaded mode) signal gets aborted after 1 second with the following error in syslog: freeradius.service: Main process exited, code=killed, status=6/ABRT freeradius.service: Failed with result 'signal'. But when i try to run it under debug mode freeradius -X (single threaded), it works fine, but the problem is i can't put it under single threaded mode for too long because it is in production environment. What I've done right now to temporarily solve the issue of reaching 1024 file descriptors limit: Is to decrease sql connection pools values as following: idle_timeout = 1 retry_delay = 1 But I know this is a temporary solution and the problem will rise again when my network grows. I could not share the connection pool across sql instances because it shared a single connection to a single database, What I want is to have each sql instance connect to a different database (Under the same mysql server). Best Regards Ibrahem Ghost On Thu, Nov 17, 2022 at 11:29 PM Alan DeKok <aland@deployingradius.com> wrote:
On Nov 17, 2022, at 2:30 PM, Ibrahemoo Ghost <ibrahemooghost@gmail.com> wrote:
I actually added freerad limits on /etc/security/limits.conf then rebooted and also tried to run freeradius as root. Then I discovered that increasing the limit for the user only does not affect the service itself, By examining the soft limit file values in /proc/$pid_of_freeradius/limits (1024)
The only way to increase the limit is by also increasing it in the service file, but as I said after that the service won't start anymore.
What does that mean?
Is there any error? Can you run it in debug mode to see what's going on?
There is nothing in FreeRADIUS which says "don't start if the OS allows more than 1024 file descriptors".
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On 18/11/2022 08:45, Ibrahemoo Ghost wrote:
I could not share the connection pool across sql instances because it shared a single connection to a single database, What I want is to have each sql instance connect to a different database (Under the same mysql server).
If the databases are all on the same MySQL server, then you could simply ensure that the user you are connecting to the server as has appropriate permissions for all the databases and the amend your queries to prefix the table name with <databasename>. - MySQL allows queries to cross databases on the same host. E.g. SELECT id, username, attribute, value, op FROM db1.radcheck WHERE username = '%{SQL-User-Name}' ORDER BY id you could even make the database name a dynamic expansion SELECT id, username, attribute, value, op FROM %{control:Local-Database}.radcheck WHERE username = '%{SQL-User-Name}' ORDER BY id where Local-Database is an attribute you define in your local dictionary and populate in your policy before calling the SQL module. -- Nick Porter
On Nov 18, 2022, at 3:45 AM, Ibrahemoo Ghost <ibrahemooghost@gmail.com> wrote:
I am running under ubuntu server 20, After raising the file descriptors limit, When i run freeradius as service (in multithreaded mode) signal gets aborted after 1 second with the following error in syslog: freeradius.service: Main process exited, code=killed, status=6/ABRT freeradius.service: Failed with result 'signal'.
Hmm... it shouldn't abort.
But when i try to run it under debug mode freeradius -X (single threaded), it works fine, but the problem is i can't put it under single threaded mode for too long because it is in production environment.
You can just do: freeradius -fxxx -l stdout which runs it in debug mode (threaded). See also doc/bugs. If you're not using the latest version... upgrade. Free packages are available at http://packages.networkradius.com i.e. we're not going to debug issues which have been found && fixed years ago.
I could not share the connection pool across sql instances because it shared a single connection to a single database, What I want is to have each sql instance connect to a different database (Under the same mysql server).
As Nick suggested, you can add the DB name to the SQL query. This means you only need one SQL connection pool. But even without that, if you have ~100 different SQL modules, do they each really need 10+ connections? i.e. is your server really getting that many packets? Why not just drop the connection limit to ~3-4 connections? Unless your server is getting 1000 packets/s, there are few reasons for having 10+ SQL connections at the same time. If you're getting a low number of packets/s and still need many SQL connections, then your SQL server is slow. Fix it. It should reply to FreeRADIUS very quickly. Alan DeKok.
participants (4)
-
Alan DeKok -
Ibrahemoo Ghost -
Nick Porter -
Pablo Umanzor