- Freeradius-Users - lists.freeradius.org

Re: FreeRADIUS Thread Behaviour
by Arran Cudbard-Bell 17 Oct '17

17 Oct '17

> The motivation for looking into this is that I have occasions where freeradius > reports “Could not start TLS: Can't contact LDAP server” > which seems to only occur when a new thread fires up and tries to start TLS > to the LDAP server. This happens randomly for a few seconds a couple times a month. > Regardless of whether or not this indicates a problem > with my LDAP server or the connection to it, I would like to know why > I have threads continuously dying and spawning every few minutes even > though my config, as I understand it, is set so that the threads should not die. That's simply not how the rlm_ldap module and the threading code interact. It will be in v4.0.x because we'll be binding (very tightly) LDAP connections to worker threads, but it is not how the code works in v3.0.x, v2.0.x or v1.0.x. What you're seeing is threads being killed because the load on the server does not require that number of threads. i.e. The server is self-pruning threads to free up resources. If there's an uptick in load, new threads will be created, and new connections may be created to service those threads. The connection pool for LDAP is in no way tied to the threading code. The connections do not get closed because a backend thread is killed, they get closed because the LDAP module is similarly detecting that it has spare connections, and is freeing them to concentrate requests on the smallest subset possible. The reason why these events occur around the same time is because the start/mac/min_spare/max_spare parameters are shared between the thread pool and connection pool. start = ${thread[pool].start_servers} min = ${thread[pool].min_spare_servers} max = ${thread[pool].max_servers} spare = ${thread[pool].max_spare_servers} Alan's example config: start_servers = 32 max_servers = 32 min_spare_servers = 0 max_spare_servers = 32 Will indeed prevent threads and connections being pruned, but be sure to adjust your idle timeouts appropriately so you don't end up with stale connections. Regarding the OpenLDAP API. There's an issue with ldap_initialize where it does some global initialisation the first time it's called. We work around that by calling it once when the server starts up before we spawn worker threads. I don't think that's the issue here. > In multi-threaded software with a persistent connection (pool) I usually > use a lock to ensure that ldap_initialize() and bind (simple or SASL) > were successfully done before the connection can be used by another > thread. That's also the reason why I usually avoid a code path with > anon-search-without-bind because I want to provoke a connect error with > an explicit bind right from the beginning. If you're experiencing the same issue as us, you can just call ldap_initialize with a zero length string as the host argument. https://github.com/FreeRADIUS/freeradius-server/blob/v4.0.x/src/lib/ldap/li… There are multiple other issues with libldap and lazy connections. Almost all can be worked around, but it makes the calling code more complex. The major issue we currently have with libldap is even with all the magic flags set to invoke async behaviour, TLS negotiation is still blocking. -Arran Arran Cudbard-Bell <a.cudbardb(a)freeradius.org> FreeRADIUS Development Team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2

2 1

FreeRADIUS Thread Behavior
by Doug Wussler 16 Oct '17

16 Oct '17

Doug says: - The question I am trying to answer is why the threads turn over at all, and in particular so frequently. Here is my config for the settings I thought control the thread behavior: From “radiusd.conf” # THREAD POOL CONFIGURATION thread pool { start_servers = 8 max_servers = 32 min_spare_servers = 2 max_spare_servers = 8 Alan replies: i.e. if there are more than 8 spare threads, one is deleted. If there are fewer than 2 spare threads, one is created. Doug says: Have I overlooked a setting or misunderstood the way these settings work? Alan replies: Again, the documentation is VERY clear on this: # It does this by periodically checking how many servers are # waiting for a request. If there are fewer than # min_spare_servers, it creates a new spare. If there are # more than max_spare_servers, some of the spares die off. Doug responds: Yes, I agree the doc is clear. That is my point, I do not see my system behaving like the doc describes, or at least how one would expect it to behave. Further testing indicates the thread testing loop behaves like this: Check if idle threads are less than min. Always find fewer than min spare threads so spawn more idle threads. Check to see if idle threads are more than max. Always find more idle threads than max so kill oldest threads. Rest Return to beginning. One iteration of the loop happens so fast that, if you look, you almost never see the newly spawned spare threads along with the full set of threads that were just tested for idleness. Right after the new threads are spawned, older ones are terminated. Not only are threads being unintentionally retired and spawned, the LDAP and SQL connections are lost and must be restarted too. So, I’m not complaining, I just thought you would want to know. As you say, you could set: start_servers = n max_servers = n max_spare_servers = n min_spare_servers = 0 and that will put a stop to the unwanted dying and spawning of threads. This seems like an approach to be vastly preferred over the way the current non-zero spare thread implementation is working, even if it wastes some resources. However, to minimize wasted resources while ensuring peak demands are met, one first needs to determine a “good” number for n for their particular server, and, more importantly, it remains to be seen if threads that never die are subject to some sort of memory leakage or data-structure corruption between server reboots (which for us is daily at log-rotate time). Doug Doug Wussler 850.645.4201 Florida State University RK Shaw Building 644 W. Call Street Tallahassee, FL 32304

2 1

(no subject)
by Samia El Haddouti 16 Oct '17

16 Oct '17

Hi Alan, I have FreeRadius server with OpenLDAP. Passwords of users are crypted at the OpenLDAP. I want to use PEAP + EAP-MSCHAPv2 for authentication. Is this possible given that the user passwords are not in cleartext? Cheers, Sam

2 2

Freeradius 3.0.4-8.el7_3.x86_64 Update or not necessary ?
by Karim Bourenane 16 Oct '17

16 Oct '17

Hello I ask if is very necessary to update my actual version of FreeRadius ? Actually i have a version 3.0.4-8.el7_3.x86_64 on Linux. Can you confirm ? Because i found several security alert concerning this version. Regard -- Mr Karim Bourenane +33686464439

8 11

FreeRADIUS Thread Behaviour
by Doug Wussler 16 Oct '17

16 Oct '17

On a RHEL 7 system, you can use “ps –eLF | grep ^fradius” to watch the v 3.0.15 freeradius process threads turn over. In particular you can see the start time of the main process, which remains constant, and the start times of the threads, which change as the threads die off and are re-spawned. For example: UID PID PPID LWP C NLWP SZ RSS PSR STIME TTY TIME CMD <root@auth8:0 /var/log/radius>$ ps -eLF | grep ^fradius fradius 3840 1 3840 1 9 535166 383420 7 03:39 ? 00:06:05 /usr/local/sbin/radiusd fradius 3840 1 35620 1 9 535166 383420 12 10:16 ? 00:00:23 /usr/local/sbin/radiusd fradius 3840 1 35621 1 9 535166 383420 17 10:16 ? 00:00:23 /usr/local/sbin/radiusd fradius 3840 1 35622 1 9 535166 383420 6 10:16 ? 00:00:22 /usr/local/sbin/radiusd fradius 3840 1 35623 1 9 535166 383420 2 10:16 ? 00:00:22 /usr/local/sbin/radiusd fradius 3840 1 35626 1 9 535166 383420 3 10:20 ? 00:00:19 /usr/local/sbin/radiusd fradius 3840 1 35627 1 9 535166 383420 3 10:20 ? 00:00:19 /usr/local/sbin/radiusd fradius 3840 1 35628 1 9 535166 383420 12 10:22 ? 00:00:17 /usr/local/sbin/radiusd fradius 3840 1 35629 1 9 535166 383420 12 10:22 ? 00:00:17 /usr/local/sbin/radius Note that a few minutes later, two of the threads have died off and two others have fired up. <root@auth8:0 /var/log/radius>$ ps -eLF | grep ^fradius fradius 3840 1 3840 1 9 535166 385280 4 03:39 ? 00:06:07 /usr/local/sbin/radiusd fradius 3840 1 35622 1 9 535166 385280 2 10:16 ? 00:00:24 /usr/local/sbin/radiusd fradius 3840 1 35623 1 9 535166 385280 2 10:16 ? 00:00:23 /usr/local/sbin/radiusd fradius 3840 1 35626 1 9 535166 385280 12 10:20 ? 00:00:21 /usr/local/sbin/radiusd fradius 3840 1 35627 1 9 535166 385280 6 10:20 ? 00:00:21 /usr/local/sbin/radiusd fradius 3840 1 35628 1 9 535166 385280 12 10:22 ? 00:00:19 /usr/local/sbin/radiusd fradius 3840 1 35629 1 9 535166 385280 8 10:22 ? 00:00:19 /usr/local/sbin/radiusd fradius 3840 1 35701 1 9 535166 385280 6 10:42 ? 00:00:00 /usr/local/sbin/radiusd fradius 3840 1 35702 1 9 535166 385280 2 10:42 ? 00:00:00 /usr/local/sbin/radiusd When a thread dies off, it tears down its LDAP connection and when the new thread starts up it stands up a new LDAP connection. If you start these threads on an idle system you can see (using Wireshark) that they do not turn over and the LDAP connections remain stable through the use of keepalives. But on a busy system, the threads turn over and thus so do the LDAP connections. I thought I had things configured so that threads never die off. The question I am trying to answer is why the threads turn over at all, and in particular so frequently. Here is my config for the settings I thought control the thread behavior: From “radiusd.conf” # THREAD POOL CONFIGURATION thread pool { start_servers = 8 max_servers = 32 min_spare_servers = 2 max_spare_servers = 8 # max_queue_size = 65536 # '0' = 'infinity', or 'the servers never exit' max_requests_per_server = 0 # Automatically limit the number of accounting requests. auto_limit_acct = no } From my “ldap” module: pool { start = ${thread[pool].start_servers} min = ${thread[pool].min_spare_servers} max = ${thread[pool].max_servers} spare = ${thread[pool].max_spare_servers} uses = 0 retry_delay = 5 lifetime = 0 idle_timeout = 0 } options { rebind = no res_timeout = 5 srv_timelimit = 5 net_timeout = 3 # ldap_debug = 0x0028 } Have I overlooked a setting or misunderstood the way these settings work? The motivation for looking into this is that I have occasions where freeradius reports “Could not start TLS: Can't contact LDAP server” which seems to only occur when a new thread fires up and tries to start TLS to the LDAP server. This happens randomly for a few seconds a couple times a month. Regardless of whether or not this indicates a problem with my LDAP server or the connection to it, I would like to know why I have threads continuously dying and spawning every few minutes even though my config, as I understand it, is set so that the threads should not die. Doug Doug Wussler Florida State University RK Shaw Building 644 W. Call Street Tallahassee, FL 32304

3 5

DHCP relay - process replies (DHCP-Offer, DHCP-Ack...)
by Chaigneau, Nicolas 16 Oct '17

16 Oct '17

Hello, I'm trying to set up a DHCP relay with FreeRADIUS (3.0.15). I've started with the virtual server example "dhcp.relay" provided in the package. The DHCP relaying itself works fine. I would like to do stuff when receiving the response from the server, so I've tried to add this in the virtual server configuration: server dhcp.relay { (...) dhcp DHCP-Offer { linelog_dhcp } dhcp DHCP-Ack { linelog_dhcp } I see this is correctly loaded when the server is started: Thu Oct 12 10:10:55 2017 : Debug: # Loading dhcp DHCP-Offer {...} Thu Oct 12 10:10:55 2017 : Debug: group { Thu Oct 12 10:10:55 2017 : Debug: linelog_dhcp Thu Oct 12 10:10:55 2017 : Debug: } Thu Oct 12 10:10:55 2017 : Debug: # Loading dhcp DHCP-Ack {...} Thu Oct 12 10:10:55 2017 : Debug: group { Thu Oct 12 10:10:55 2017 : Debug: linelog_dhcp Thu Oct 12 10:10:55 2017 : Debug: } However, this is never called when such a message is received. (contrary to "dhcp DHCP-Discover" and "dhcp DHCP-Request" sub-sections) Am I missing something ? Thanks. Regards, Nicolas. This message contains information that may be privileged or confidential and is the property of the Capgemini Group. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message.

2 9

proxy request ok, but incomplete access-accept to my Aruba
by Ramon Escriba 13 Oct '17

13 Oct '17

Hi Experts, I'm using Freradius3 with 802.1x EAP. It works perfect with my @my-realm.com, but when I send test realms to the external proxy, it accepts the query (see below logs), but seems the Access-Accept packet does not complete the fields my Aruba needs to open the vlan, as I do in @my-realm.com with users file. How can I complete the "Access-Accept" packet I send to my Aruba? (8) MS-MPPE-Send-Key = 0xb809xxxxxxxxxxxxxxx184e8ba236944dc8 (8) MS-MPPE-Recv-Key = 0x1xxxxxxxxxxxxxxxxxxxx99248b723ffdf64 (8) EAP-Message = 0x03080004 (8) Message-Authenticator = 0xxxxb0c134c619fd65 (8) Proxy-State = 0x38 (8) # Executing section post-proxy from file /etc/raddb/sites-enabled/default (8) post-proxy { (8) eap : No pre-existing handler found (8) [eap] = noop (8) } # post-proxy = noop (8) Found Auth-Type = Accept (8) Auth-Type = Accept, accepting the user (8) Login OK: [tests(a)acme.com] (from client loopback port 0 cli 02-00-00-00-00-01) (8) # Executing section post-auth from file /etc/raddb/sites-enabled/default (8) post-auth { (8) [exec] = noop (8) remove_reply_message_if_eap remove_reply_message_if_eap { (8) if (&reply:EAP-Message && &reply:Reply-Message) (8) if (&reply:EAP-Message && &reply:Reply-Message) -> FALSE (8) else else { (8) [noop] = noop (8) } # else else = noop (8) } # remove_reply_message_if_eap remove_reply_message_if_eap = noop (8) } # post-auth = noop (8) Sending Access-Accept packet to host 127.0.0.1 port 57123, id=8, length=0 (8) MS-MPPE-Send-Key = 0xxxxxxxxxxxxxxxxxx4da8 (8) MS-MPPE-Recv-Key = 0xxxxxxxxxxxxxxxxxxfaf64 (8) EAP-Message = 0x03080004 (8) Message-Authenticator = 0xxxxxxxxxxxxxx03d20 Sending Access-Accept Id 8 from 127.0.0.1:1812 to 127.0.0.1:57123 MS-MPPE-Send-Key = 0xxxxxxxxxxxxxxx1f184e88 MS-MPPE-Recv-Key = 0xxxxxx4899a3e0fa2a20f34f EAP-Message = 0x03080004 Message-Authenticator = 0xxxxxx31f04620 (8) Finished request Waking up in 6.4 seconds Kindest Regards.

3 4

Re: Freeradius service - unexpected restarts
by Peter Hutchison 13 Oct '17

13 Oct '17

> Any ideas what cause the freeradius service to stop and restart > repeatedly like this? > > - do you run latest version of FR (or at least, latest available from your distro)? Yes - does it happen often, or only on that specified time? No it doesn't happen often, only yesterday so far. - does something happen on that time that restart freeradius (e.g. logrotate, apt-get upgrade, or something)? No, all scheduled job run outside normal daytime hours. - does FR access external things (e.g. database)? What is the state of those systems at that time? No, no databases are used. >The usual troubleshooting methods would be 'run freeradius in debug mode and read the logs', but it might not be able to help you much if the problem is due FR being restarted. It would be very hard do to 'read' such logs on a live system with the high number of connections made. Peter Hutchison MCP Senior Network Systems Specialist * 01484 473716 Networks Team University of Huddersfield | Queensgate | Huddersfield | HD1 3DH University of Huddersfield inspiring tomorrow's professionals. [http://marketing.hud.ac.uk/_HOSTED/EmailSig2014/EmailSigFooter.jpg] This transmission is confidential and may be legally privileged. If you receive it in error, please notify us immediately by e-mail and remove it from your system. If the content of this e-mail does not relate to the business of the University of Huddersfield, then we do not endorse it and will accept no liability.

2 1

Questions about ldap authentication, huntgroup and authorize file
by Jérôme BERTHIER 13 Oct '17

13 Oct '17

Hi, I'm moving a setup from freeradius 2 (CentOS 6) to freeradius 3 (CentOS 7). The server is hosting : - a backend to authenticate users on Wifi 802.1X TTLS-PAP (openldap) and PEAP-MSCHAPv2 (Active Directory) - a backend to authenticate users on network equipment for admin purpose We use several ldap instances described in the ldap module. Our openldap server does not permit to read password. We use anonymous binding to run the authentication process. So, auth-type := <ldap_instance> is setted by ldap module file in each ldap instance : set_auth_type = yes Network equipements are grouped in the file huntgroup to apply a specific policy by using filters on huntgroup and ldap group in the file users. The filter replies differents vendor specific attributes for each ldap group and finally, we reject the authentication if there is no match. For example, for devices from Juniper, we use this setup : DEFAULT Ldap-Group==ldap_group_1,Huntgroup-Name == JuniperNet Service-Type = Login, Juniper-Local-User-Name := "radius-admin" DEFAULT Ldap-Group==ldap_group_2,Huntgroup-Name == JuniperNet Service-Type = Login, Juniper-Local-User-Name := "radius-operator" DEFAULT Auth-Type := Reject,Huntgroup-Name == JuniperNet I took care of the file README.rst. I had also a look to the documentation page about ldap module : http://wiki.freeradius.org/modules/Rlm_ldap I still have few questions : 1) I notice that the variable Ldap-Group is no more set by the last ldap instance called. So, I corrected my config to describe the default instance ldap {}. It works but is it planned to be able to call another instance in the file authorize ? 2) Since the module ldap does not set auth-type := ldap anymore, I add the lines to the configuration of the site where needed : if ((ok || updated) && User-Password) { update { control:Auth-Type := ldap } } Is it correct ? 3) Is it correct to use the file authorize to apply filters like we do ? or, shall we do it in the post auth section (it could be innacurate if we need to reject the access) ? Thanks you for your help Regards, -- Jérôme BERTHIER

2 7

Freeradius service - unexpected restarts
by Peter Hutchison 13 Oct '17

13 Oct '17

On our Freeradius 2.0.12 on Ubuntu 14.04 server, we are sometimes experiencing repeated service restarts. $ grep freeradius /var/log/syslog.1 Oct 12 16:08:52 texas kernel: [1380110.477452] init: freeradius main process (1688) terminated with status 1 Oct 12 16:08:52 texas kernel: [1380110.477460] init: freeradius main process ended, respawning Oct 12 16:08:52 texas kernel: [1380110.589303] init: freeradius main process (1699) terminated with status 1 Oct 12 16:08:52 texas kernel: [1380110.589312] init: freeradius main process ended, respawning Oct 12 16:08:52 texas kernel: [1380110.698334] init: freeradius main process (1708) terminated with status 1 Oct 12 16:08:52 texas kernel: [1380110.698342] init: freeradius main process ended, respawning Oct 12 16:08:52 texas kernel: [1380110.808440] init: freeradius main process (1718) terminated with status 1 Oct 12 16:08:52 texas kernel: [1380110.808449] init: freeradius main process ended, respawning Alert from monit: Timeout Service freeradius Date: Thu, 12 Oct 2017 16:21:23 Action: unmonitor Host: texas.hud.ac.uk Description: service restarted 5 times within 5 cycles(s) - unmonitor Any ideas what cause the freeradius service to stop and restart repeatedly like this? Peter Hutchison MCP Senior Network Systems Specialist * 01484 473716 Networks Team University of Huddersfield | Queensgate | Huddersfield | HD1 3DH University of Huddersfield inspiring tomorrow's professionals. [http://marketing.hud.ac.uk/_HOSTED/EmailSig2014/EmailSigFooter.jpg] This transmission is confidential and may be legally privileged. If you receive it in error, please notify us immediately by e-mail and remove it from your system. If the content of this e-mail does not relate to the business of the University of Huddersfield, then we do not endorse it and will accept no liability.

3 2