rlm_ippool memory footprint
I sometimes see freeradius killed by OOM-killer at boot time. On investigating this, freeradius seems to allocate a lot of RAM to manage IP address pools (rlm_ippool). Is it possible to reduce this footprint? I have a large IP pool configured (64k addresses) using rlm_ippool. range_start = 172.28.0.1 range_stop = 172.28.255.254 netmask = 255.255.0.0 cache_size = 65535 freeradius allocates 600M with this configuration: # pgrep freeradius 4681 # ps -p 4681 --format "%cpu %mem rss" %CPU %MEM RSS 2.1 61.5 616644 Reducing cache_size in the pool config to 255 results in far less memory being allocated (92M) : # pgrep freeradius 4245 # ps -p 4245 --format "%cpu %mem rss" %CPU %MEM RSS 0.0 9.2 93148 But freeradius docs say that cache_size should be set to the number of entries in the pool. In a test setup, using a cache_size that is much less than the pool size seems to work. Other than for performance reasons, is there a reason why cache_size should be set to the number of entries in the pool? It seems like a lot of memory is otherwise needed for managing an IP address pool... My system is a minimal Ubuntu 20.04 VM with 1G RAM. freeradius is version 3.0.20. I attach output of freeradius -X. Thanks James
On 14/10/2020 21:19, James Chapman via Freeradius-Users wrote:
My system is a minimal Ubuntu 20.04 VM with 1G RAM. freeradius is version 3.0.20. I attach output of freeradius -X.
Whoops, I should have inlined the freeradius debug output for this list. Here it is:
FreeRADIUS Version 3.0.20 Copyright (C) 1999-2019 The FreeRADIUS server project and contributors There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE You may redistribute copies of FreeRADIUS under the terms of the GNU General Public License For more information about these matters, see the file named COPYRIGHT Starting - reading configuration files ... including dictionary file /usr/share/freeradius/dictionary including dictionary file /usr/share/freeradius/dictionary.dhcp including dictionary file /usr/share/freeradius/dictionary.vqp including dictionary file /etc/freeradius/3.0/dictionary including configuration file /etc/freeradius/3.0/radiusd.conf including configuration file /etc/freeradius/3.0/proxy.conf including configuration file /etc/freeradius/3.0/clients.conf including files in directory /etc/freeradius/3.0/mods-enabled/ including configuration file /etc/freeradius/3.0/mods-enabled/preprocess including configuration file /etc/freeradius/3.0/mods-enabled/always including configuration file /etc/freeradius/3.0/mods-enabled/ippool including configuration file /etc/freeradius/3.0/mods-enabled/attr_filter including configuration file /etc/freeradius/3.0/mods-enabled/chap including configuration file /etc/freeradius/3.0/mods-enabled/digest including configuration file /etc/freeradius/3.0/mods-enabled/expiration including configuration file /etc/freeradius/3.0/mods-enabled/files including configuration file /etc/freeradius/3.0/mods-enabled/echo including configuration file /etc/freeradius/3.0/mods-enabled/realm including configuration file /etc/freeradius/3.0/mods-enabled/utf8 including configuration file /etc/freeradius/3.0/mods-enabled/exec including configuration file /etc/freeradius/3.0/mods-enabled/pap including configuration file /etc/freeradius/3.0/mods-enabled/logintime including configuration file /etc/freeradius/3.0/mods-enabled/unpack including configuration file /etc/freeradius/3.0/mods-enabled/linelog including files in directory /etc/freeradius/3.0/policy.d/ including configuration file /etc/freeradius/3.0/policy.d/debug including configuration file /etc/freeradius/3.0/policy.d/dhcp including configuration file /etc/freeradius/3.0/policy.d/abfab-tr including configuration file /etc/freeradius/3.0/policy.d/operator-name including configuration file /etc/freeradius/3.0/policy.d/moonshot-targeted-ids including configuration file /etc/freeradius/3.0/policy.d/cui including configuration file /etc/freeradius/3.0/policy.d/canonicalization including configuration file /etc/freeradius/3.0/policy.d/rfc7542 including configuration file /etc/freeradius/3.0/policy.d/control including configuration file /etc/freeradius/3.0/policy.d/eap including configuration file /etc/freeradius/3.0/policy.d/accounting including configuration file /etc/freeradius/3.0/policy.d/filter including files in directory /etc/freeradius/3.0/sites-enabled/ including configuration file /etc/freeradius/3.0/sites-enabled/my-vpn main { security { user = "freerad" group = "freerad" allow_core_dumps = no } name = "freeradius" prefix = "/usr" localstatedir = "/var" logdir = "/var/log/freeradius" run_dir = "/var/run/freeradius" } main { name = "freeradius" prefix = "/usr" localstatedir = "/var" sbindir = "/usr/sbin" logdir = "/var/log/freeradius" run_dir = "/var/run/freeradius" libdir = "/usr/lib/freeradius" radacctdir = "/var/log/freeradius/radacct" hostname_lookups = no max_request_time = 30 cleanup_delay = 5 max_requests = 16384 pidfile = "/var/run/freeradius/freeradius.pid" checkrad = "/usr/sbin/checkrad" debug_level = 0 proxy_requests = yes log { stripped_names = no auth = no auth_badpass = no auth_goodpass = no colourise = yes msg_denied = "You are already logged in - access denied" } resources { } security { max_attributes = 200 reject_delay = 1.000000 status_server = yes } } radiusd: #### Loading Realms and Home Servers #### proxy server { retry_delay = 5 retry_count = 3 default_fallback = no dead_time = 120 wake_all_if_all_dead = no } home_server localhost { ipaddr = 127.0.0.1 port = 1812 type = "auth" secret = <<< secret >>> response_window = 20.000000 response_timeouts = 1 max_outstanding = 65536 zombie_period = 40 status_check = "status-server" ping_interval = 30 check_interval = 30 check_timeout = 4 num_answers_to_alive = 3 revive_interval = 120 limit { max_connections = 16 max_requests = 0 lifetime = 0 idle_timeout = 0 } coa { irt = 2 mrt = 16 mrc = 5 mrd = 30 } } home_server_pool my_auth_failover { type = fail-over home_server = localhost } realm example.com { auth_pool = my_auth_failover } realm LOCAL { } radiusd: #### Loading Clients #### client localhost { ipaddr = 127.0.0.1 require_message_authenticator = no secret = <<< secret >>> nas_type = "other" virtual_server = "my-vpn" response_window = 10.000000 proto = "udp" limit { max_connections = 16 lifetime = 0 idle_timeout = 30 } } Debugger not attached systemd watchdog is disabled # Creating Auth-Type = PAP # Creating Auth-Type = CHAP radiusd: #### Instantiating modules #### modules { # Loaded module rlm_preprocess # Loading module "preprocess" from file /etc/freeradius/3.0/mods-enabled/preprocess preprocess { huntgroups = "/etc/freeradius/3.0/mods-config/preprocess/huntgroups" hints = "/etc/freeradius/3.0/mods-config/preprocess/hints" with_ascend_hack = no ascend_channels_per_line = 23 with_ntdomain_hack = no with_specialix_jetstream_hack = no with_cisco_vsa_hack = no with_alvarion_vsa_hack = no } # Loaded module rlm_always # Loading module "reject" from file /etc/freeradius/3.0/mods-enabled/always always reject { rcode = "reject" simulcount = 0 mpp = no } # Loading module "fail" from file /etc/freeradius/3.0/mods-enabled/always always fail { rcode = "fail" simulcount = 0 mpp = no } # Loading module "ok" from file /etc/freeradius/3.0/mods-enabled/always always ok { rcode = "ok" simulcount = 0 mpp = no } # Loading module "handled" from file /etc/freeradius/3.0/mods-enabled/always always handled { rcode = "handled" simulcount = 0 mpp = no } # Loading module "invalid" from file /etc/freeradius/3.0/mods-enabled/always always invalid { rcode = "invalid" simulcount = 0 mpp = no } # Loading module "userlock" from file /etc/freeradius/3.0/mods-enabled/always always userlock { rcode = "userlock" simulcount = 0 mpp = no } # Loading module "notfound" from file /etc/freeradius/3.0/mods-enabled/always always notfound { rcode = "notfound" simulcount = 0 mpp = no } # Loading module "noop" from file /etc/freeradius/3.0/mods-enabled/always always noop { rcode = "noop" simulcount = 0 mpp = no } # Loading module "updated" from file /etc/freeradius/3.0/mods-enabled/always always updated { rcode = "updated" simulcount = 0 mpp = no } # Loaded module rlm_ippool # Loading module "main_pool" from file /etc/freeradius/3.0/mods-enabled/ippool ippool main_pool { filename = "/etc/freeradius/3.0/db.ippool" ip_index = "/etc/freeradius/3.0/db.ipindex" key = "%{NAS-IP-Address} %{NAS-Port}" range_start = 172.28.0.1 range_stop = 172.28.255.254 netmask = 255.255.0.0 cache_size = 65535 override = yes maximum_timeout = 0 } # Loaded module rlm_attr_filter # Loading module "attr_filter.post-proxy" from file /etc/freeradius/3.0/mods-enabled/attr_filter attr_filter attr_filter.post-proxy { filename = "/etc/freeradius/3.0/mods-config/attr_filter/post-proxy" key = "%{Realm}" relaxed = no } # Loading module "attr_filter.pre-proxy" from file /etc/freeradius/3.0/mods-enabled/attr_filter attr_filter attr_filter.pre-proxy { filename = "/etc/freeradius/3.0/mods-config/attr_filter/pre-proxy" key = "%{Realm}" relaxed = no } # Loading module "attr_filter.access_reject" from file /etc/freeradius/3.0/mods-enabled/attr_filter attr_filter attr_filter.access_reject { filename = "/etc/freeradius/3.0/mods-config/attr_filter/access_reject" key = "%{User-Name}" relaxed = no } # Loading module "attr_filter.access_challenge" from file /etc/freeradius/3.0/mods-enabled/attr_filter attr_filter attr_filter.access_challenge { filename = "/etc/freeradius/3.0/mods-config/attr_filter/access_challenge" key = "%{User-Name}" relaxed = no } # Loading module "attr_filter.accounting_response" from file /etc/freeradius/3.0/mods-enabled/attr_filter attr_filter attr_filter.accounting_response { filename = "/etc/freeradius/3.0/mods-config/attr_filter/accounting_response" key = "%{User-Name}" relaxed = no } # Loaded module rlm_chap # Loading module "chap" from file /etc/freeradius/3.0/mods-enabled/chap # Loaded module rlm_digest # Loading module "digest" from file /etc/freeradius/3.0/mods-enabled/digest # Loaded module rlm_expiration # Loading module "expiration" from file /etc/freeradius/3.0/mods-enabled/expiration # Loaded module rlm_files # Loading module "files" from file /etc/freeradius/3.0/mods-enabled/files files { filename = "/etc/freeradius/3.0/mods-config/files/authorize" acctusersfile = "/etc/freeradius/3.0/mods-config/files/accounting" preproxy_usersfile = "/etc/freeradius/3.0/mods-config/files/pre-proxy" } # Loaded module rlm_exec # Loading module "echo" from file /etc/freeradius/3.0/mods-enabled/echo exec echo { wait = yes program = "/bin/echo %{User-Name}" input_pairs = "request" output_pairs = "reply" shell_escape = yes } # Loaded module rlm_realm # Loading module "IPASS" from file /etc/freeradius/3.0/mods-enabled/realm realm IPASS { format = "prefix" delimiter = "/" ignore_default = no ignore_null = no } # Loading module "suffix" from file /etc/freeradius/3.0/mods-enabled/realm realm suffix { format = "suffix" delimiter = "@" ignore_default = no ignore_null = no } # Loading module "bangpath" from file /etc/freeradius/3.0/mods-enabled/realm realm bangpath { format = "prefix" delimiter = "!" ignore_default = no ignore_null = no } # Loading module "realmpercent" from file /etc/freeradius/3.0/mods-enabled/realm realm realmpercent { format = "suffix" delimiter = "%" ignore_default = no ignore_null = no } # Loading module "ntdomain" from file /etc/freeradius/3.0/mods-enabled/realm realm ntdomain { format = "prefix" delimiter = "\\" ignore_default = no ignore_null = no } # Loaded module rlm_utf8 # Loading module "utf8" from file /etc/freeradius/3.0/mods-enabled/utf8 # Loading module "exec" from file /etc/freeradius/3.0/mods-enabled/exec exec { wait = no input_pairs = "request" shell_escape = yes timeout = 10 } # Loaded module rlm_pap # Loading module "pap" from file /etc/freeradius/3.0/mods-enabled/pap pap { normalise = yes } # Loaded module rlm_logintime # Loading module "logintime" from file /etc/freeradius/3.0/mods-enabled/logintime logintime { minimum_timeout = 60 } # Loaded module rlm_unpack # Loading module "unpack" from file /etc/freeradius/3.0/mods-enabled/unpack # Loaded module rlm_linelog # Loading module "linelog" from file /etc/freeradius/3.0/mods-enabled/linelog linelog { filename = "/var/log/freeradius/linelog" escape_filenames = no syslog_severity = "info" permissions = 384 format = "This is a log message for %{User-Name}" reference = "messages.%{%{reply:Packet-Type}:-default}" } # Loading module "log_accounting" from file /etc/freeradius/3.0/mods-enabled/linelog linelog log_accounting { filename = "/var/log/freeradius/linelog-accounting" escape_filenames = no syslog_severity = "info" permissions = 384 format = "" reference = "Accounting-Request.%{%{Acct-Status-Type}:-unknown}" } instantiate { } # Instantiating module "preprocess" from file /etc/freeradius/3.0/mods-enabled/preprocess reading pairlist file /etc/freeradius/3.0/mods-config/preprocess/huntgroups reading pairlist file /etc/freeradius/3.0/mods-config/preprocess/hints # Instantiating module "reject" from file /etc/freeradius/3.0/mods-enabled/always # Instantiating module "fail" from file /etc/freeradius/3.0/mods-enabled/always # Instantiating module "ok" from file /etc/freeradius/3.0/mods-enabled/always # Instantiating module "handled" from file /etc/freeradius/3.0/mods-enabled/always # Instantiating module "invalid" from file /etc/freeradius/3.0/mods-enabled/always # Instantiating module "userlock" from file /etc/freeradius/3.0/mods-enabled/always # Instantiating module "notfound" from file /etc/freeradius/3.0/mods-enabled/always # Instantiating module "noop" from file /etc/freeradius/3.0/mods-enabled/always # Instantiating module "updated" from file /etc/freeradius/3.0/mods-enabled/always # Instantiating module "main_pool" from file /etc/freeradius/3.0/mods-enabled/ippool # Instantiating module "attr_filter.post-proxy" from file /etc/freeradius/3.0/mods-enabled/attr_filter reading pairlist file /etc/freeradius/3.0/mods-config/attr_filter/post-proxy # Instantiating module "attr_filter.pre-proxy" from file /etc/freeradius/3.0/mods-enabled/attr_filter reading pairlist file /etc/freeradius/3.0/mods-config/attr_filter/pre-proxy # Instantiating module "attr_filter.access_reject" from file /etc/freeradius/3.0/mods-enabled/attr_filter reading pairlist file /etc/freeradius/3.0/mods-config/attr_filter/access_reject # Instantiating module "attr_filter.access_challenge" from file /etc/freeradius/3.0/mods-enabled/attr_filter reading pairlist file /etc/freeradius/3.0/mods-config/attr_filter/access_challenge # Instantiating module "attr_filter.accounting_response" from file /etc/freeradius/3.0/mods-enabled/attr_filter reading pairlist file /etc/freeradius/3.0/mods-config/attr_filter/accounting_response # Instantiating module "expiration" from file /etc/freeradius/3.0/mods-enabled/expiration # Instantiating module "files" from file /etc/freeradius/3.0/mods-enabled/files reading pairlist file /etc/freeradius/3.0/mods-config/files/authorize reading pairlist file /etc/freeradius/3.0/vpnusers/all-users reading pairlist file /etc/freeradius/3.0/mods-config/files/accounting reading pairlist file /etc/freeradius/3.0/mods-config/files/pre-proxy # Instantiating module "IPASS" from file /etc/freeradius/3.0/mods-enabled/realm # Instantiating module "suffix" from file /etc/freeradius/3.0/mods-enabled/realm # Instantiating module "bangpath" from file /etc/freeradius/3.0/mods-enabled/realm # Instantiating module "realmpercent" from file /etc/freeradius/3.0/mods-enabled/realm # Instantiating module "ntdomain" from file /etc/freeradius/3.0/mods-enabled/realm # Instantiating module "pap" from file /etc/freeradius/3.0/mods-enabled/pap # Instantiating module "logintime" from file /etc/freeradius/3.0/mods-enabled/logintime # Instantiating module "linelog" from file /etc/freeradius/3.0/mods-enabled/linelog # Instantiating module "log_accounting" from file /etc/freeradius/3.0/mods-enabled/linelog } # modules radiusd: #### Loading Virtual Servers #### server { # from file /etc/freeradius/3.0/radiusd.conf } # server server my-vpn { # from file /etc/freeradius/3.0/sites-enabled/my-vpn # Loading authenticate {...} # Loading authorize {...} # Loading preacct {...} # Loading accounting {...} # Loading post-auth {...} } # server my-vpn radiusd: #### Opening IP addresses and Ports #### listen { type = "auth" ipaddr = 127.0.0.1 port = 1812 } listen { type = "acct" ipaddr = 127.0.0.1 port = 1813 } Listening on auth interface lo address 127.0.0.1 port 1812 bound to server my-vpn Listening on acct interface lo address 127.0.0.1 port 1813 bound to server my-vpn Listening on proxy address * port 44209 Ready to process requests
On Oct 14, 2020, at 4:19 PM, James Chapman via Freeradius-Users <freeradius-users@lists.freeradius.org> wrote:
I sometimes see freeradius killed by OOM-killer at boot time. On investigating this, freeradius seems to allocate a lot of RAM to manage IP address pools (rlm_ippool). Is it possible to reduce this footprint?
The module uses gdbm for it's database. So essentially all memory used is by the gdbm library.
I have a large IP pool configured (64k addresses) using rlm_ippool.
range_start = 172.28.0.1 range_stop = 172.28.255.254 netmask = 255.255.0.0 cache_size = 65535
freeradius allocates 600M with this configuration:
Hmm... that's about 10K per IP, which seems excessive.
Reducing cache_size in the pool config to 255 results in far less memory being allocated (92M) :
Yeah. Still about 10K per IP. I'm not sure why. The entries stored by the module are less than 64 bytes. And the key is 16 bytes. So 10K per entry seems _way_ excessive. But in the FreeRADIUS tradition... I can blame something else. :) The module uses gdbm, and all of the memory usage is in gdbm. Not in FreeRADIUS. You know this is true because changing the cache size directly affects the memory usage.
But freeradius docs say that cache_size should be set to the number of entries in the pool. In a test setup, using a cache_size that is much less than the pool size seems to work. Other than for performance reasons, is there a reason why cache_size should be set to the number of entries in the pool? It seems like a lot of memory is otherwise needed for managing an IP address pool...
The cache size is passed directly to gdbm. So I don't think it needs to be set to the number of IPs in the pool. it can be anything which makes sense for you. TBH, the solution here is to just use an SQL database. Something like sqlite beats gdbm in pretty much every way. In the v4 "master" branch, we've even deleted rlm_ippool. There's just not enough reasons to use a custom DB when SQL exists. If you take a look at the v3.0.x branch in GitHub, there's a bunch of new tools to manage IP ranges in SQL. That makes it much easier. Or if you want to stay with gdbm, use a small value for cache_size. The only affect is that database lookups will be a little slower. Alan DeKok.
On 14/10/2020 21:58, Alan DeKok wrote:
The cache size is passed directly to gdbm. So I don't think it needs to be set to the number of IPs in the pool. it can be anything which makes sense for you.
TBH, the solution here is to just use an SQL database. Something like sqlite beats gdbm in pretty much every way. In the v4 "master" branch, we've even deleted rlm_ippool. There's just not enough reasons to use a custom DB when SQL exists.
If you take a look at the v3.0.x branch in GitHub, there's a bunch of new tools to manage IP ranges in SQL. That makes it much easier.
Or if you want to stay with gdbm, use a small value for cache_size. The only affect is that database lookups will be a little slower.
Thanks Alan. I'll go with a smaller cache size for this particular setup. James
participants (2)
-
Alan DeKok -
James Chapman