FreeRADIUS no longer authenticating domain clients following spplication of RHEL5 package updates yesterday morning
Hi, Yesterday the following package updates were applied to our main RHEL5 FreeRADIUS server after which it was successfully rebooted: Apr 19 07:10:01 Updated: samba-client-3.0.33-3.41.el5_11.i386 Apr 19 07:10:00 Updated: 30:bind-utils-9.3.6-25.P1.el5_11.8.i386 Apr 19 07:10:00 Updated: samba-3.0.33-3.41.el5_11.i386 Apr 19 07:09:58 Updated: samba-common-3.0.33-3.41.el5_11.i386 Apr 19 07:09:58 Updated: libsmbclient-3.0.33-3.41.el5_11.i386 Apr 19 07:09:57 Installed: kernel-2.6.18-409.el5.i686 Apr 19 07:09:49 Updated: tzdata-2016c-1.el5.i386 Apr 19 07:09:47 Updated: redhat-release-5Server-5.11.0.3.i386 Apr 19 07:09:46 Updated: 30:bind-libs-9.3.6-25.P1.el5_11.8.i386 Following the reboot, the radius.log file is full of entries such as the following and no local clients are able to authenticate via this server: Wed Apr 20 16:04:26 2016 : Auth: Login incorrect: [lixc1@bathspa.ac.uk] (from client roaming1.ja.net port 0 cli B0-9F-BA-C5-EE-C0) Wed Apr 20 16:04:26 2016 : Error: rlm_eap: No EAP session matching the State variable. Wed Apr 20 16:04:20 2016 : Auth: Login incorrect: [222394@bathspa.ac.uk] (from client roaming0.ja.net port 13 cli f0-c1-f1-b0-82-01) Wed Apr 20 16:04:20 2016 : Error: rlm_eap: No EAP session matching the State variable. Wed Apr 20 16:04:14 2016 : Auth: Login incorrect: [305657@bathspa.ac.uk] (from client roaming1.ja.net port 13 cli 30-19-66-47-a3-f6) Wed Apr 20 16:04:14 2016 : Error: rlm_eap: No EAP session matching the State variable. Wed Apr 20 16:04:09 2016 : Auth: Login incorrect: [305268@bathspa.ac.uk] (from client roaming0.ja.net port 0 cli e0-c7-67-32-a1-32) Wed Apr 20 16:04:09 2016 : Error: rlm_eap: No EAP session matching the State variable. Wed Apr 20 16:04:01 2016 : Auth: Login incorrect: [222394@bathspa.ac.uk] (from client roaming0.ja.net port 13 cli f0-c1-f1-b0-82-01) Wed Apr 20 16:04:01 2016 : Error: rlm_eap: No EAP session matching the State variable. Wed Apr 20 16:04:01 2016 : Auth: Login incorrect: [305657@bathspa.ac.uk] (from client roaming1.ja.net port 13 cli 30-19-66-47-a3-f6) Wed Apr 20 16:04:01 2016 : Error: rlm_eap: No EAP session matching the State variable. Wed Apr 20 16:03:56 2016 : Auth: Login incorrect: [222394@bathspa.ac.uk] (from client roaming0.ja.net port 13 cli f0-c1-f1-b0-82-01) Wed Apr 20 16:03:56 2016 : Error: rlm_eap: No EAP session matching the State variable. Wed Apr 20 16:03:56 2016 : Auth: Login incorrect: [306757@bathspa.ac.uk] (from client roaming0.ja.net port 0 cli f0-db-f8-bf-dd-22) Wed Apr 20 16:03:56 2016 : Error: rlm_eap: No EAP session matching the State variable. Wed Apr 20 16:03:42 2016 : Auth: Login incorrect: [303331@bathspa.ac.uk] (from client roaming1.ja.net port 0 cli AC293A4C57DB) Wed Apr 20 16:03:42 2016 : Error: rlm_eap: No EAP session matching the State variable. Wed Apr 20 16:03:40 2016 : Auth: Login incorrect: [266114@bathspa.ac.uk] (from client roaming0.ja.net port 0 cli ec-1f-72-e8-95-f6) Wed Apr 20 16:03:40 2016 : Error: rlm_eap: No EAP session matching the State variable. Wed Apr 20 16:03:39 2016 : Auth: Login incorrect: [273213@bathspa.ac.uk] (from client roaming0.ja.net port 0 cli 8C3AE3979A3D) Wed Apr 20 16:03:39 2016 : Error: rlm_eap: No EAP session matching the State variable. Wed Apr 20 16:03:37 2016 : Info: Ready to process requests. Wed Apr 20 16:03:37 2016 : Info: ... adding new socket proxy address * port 36236 Wed Apr 20 16:03:37 2016 : Info: ... adding new socket proxy address * port 45657 Wed Apr 20 16:03:37 2016 : Info: ... adding new socket proxy address * port 57014 Wed Apr 20 16:03:37 2016 : Info: ... adding new socket proxy address * port 44335 Wed Apr 20 16:03:37 2016 : Info: ... adding new socket proxy address * port 49723 Wed Apr 20 16:03:37 2016 : Info: ... adding new socket proxy address * port 39639 Wed Apr 20 16:03:37 2016 : Info: ... adding new socket proxy address * port 36555 Wed Apr 20 16:03:37 2016 : Info: Loaded virtual server eduroam-inner-tunnel Wed Apr 20 16:03:37 2016 : Info: Loaded virtual server inner-tunnel Wed Apr 20 16:03:37 2016 : Info: Loaded virtual server bathspa Wed Apr 20 16:03:37 2016 : Info: Loaded virtual server eduroam Wed Apr 20 16:03:37 2016 : Auth: rlm_krb5: krb5_init ok Wed Apr 20 16:03:37 2016 : Info: Loaded virtual server <default> Oddly, the results from a: "lsof -i tcp -nP | grep winbindd" does not show any LDAP connections over port 389 to our active directory either: winbindd 4535 root 23u IPv4 16513 0t0 TCP 172.19.0.123:56635-> 172.19.0.73:445 (ESTABLISHED) We would expect to find the following: winbindd 4221 root 23u IPv4 34419318 0t0 TCP 172.19.0.124:36502->1 72.19.0.73:389 (ESTABLISHED) winbindd 4221 root 24u IPv4 8106940 0t0 TCP 172.19.0.124:50653->1 72.19.0.107:445 (ESTABLISHED) Closer inspection of the attached debug log with a search term of: warning turns up: Cleaning up request 2 ID 253 with timestamp +2 WARNING: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! WARNING: !! EAP session for state 0xdc342b61de3b32a7 did not finish! WARNING: !! Please read http://wiki.freeradius.org/Certificate_Compatibility WARNING: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Ready to process requests. To my knowledge our certificate is valid until 2018 so am not sure how to proceed, any help gratefully received. Cheers, *Jeremy HillLead Server Analyst (Windows)Bath Spa UniversityT: +44 (0)1225 875637Visit www.bathspa.ac.uk <http://www.bathspa.ac.uk/>Join us on: Facebook <http://www.facebook.com/bath.spa.university>| Twitter <https://twitter.com/#!/BathSpaUni>| YouTube <http://www.youtube.com/BathSpaUniversity>| LinkedIn <http://www.linkedin.com/company/bath-spa-university>Newton Park, Bath, BA2 9BNThink before you printDisclaimerIf you have received this message in error, please notify us and remove it from your system. Any views or opinions expressed in personal emails are solely those of the author and do not necessarily represent those of Bath Spa University. Neither Bath Spa University nor the sender accepts any responsibility for viruses and it is your responsibility to scan this email and any attachments for viruses.*
On Wed, Apr 20, 2016 at 05:11:36PM +0100, Jeremy Hill wrote:
Following the reboot, the radius.log file is full of entries such as the following and no local clients are able to authenticate via this server:
How long did you leave it running for? It's usual to see some "No EAP session matching the State variable." for a short while after starting a server if there are auths that get cut over to the new server while in-flight. There's not enough in the debug log below to tell. You start off with a half in-flight auth, then that restarts and the debug is cut off before the second one finishes.
Oddly, the results from a: "lsof -i tcp -nP | grep winbindd" does not show any LDAP connections over port 389 to our active directory either:
winbindd 4535 root 23u IPv4 16513 0t0 TCP 172.19.0.123:56635-> 172.19.0.73:445 (ESTABLISHED)
We would expect to find the following:
winbindd 4221 root 23u IPv4 34419318 0t0 TCP 172.19.0.124:36502->1 72.19.0.73:389 (ESTABLISHED) winbindd 4221 root 24u IPv4 8106940 0t0 TCP 172.19.0.124:50653->1 72.19.0.107:445 (ESTABLISHED)
Depends on where your LDAP call is made. If right at the end of the auth, and the auths aren't getting that far, then you won't see anything. So this isn't necessarily a problem in and of itself.
Cleaning up request 2 ID 253 with timestamp +2 WARNING: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! WARNING: !! EAP session for state 0xdc342b61de3b32a7 did not finish! WARNING: !! Please read http://wiki.freeradius.org/Certificate_Compatibility WARNING: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Ready to process requests.
To my knowledge our certificate is valid until 2018 so am not sure how to proceed, any help gratefully received.
Assuming you're doing PEAP/MSCHAPv2 with ntlm_auth, then what happens at the ntlm_auth stage in the debug logs? That's the bit you've changed by upgrading Samba. Might be something as simple as the winbind priv socket not being readable by FreeRADIUS any more. Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
Hi,
Yesterday the following package updates were applied to our main RHEL5 FreeRADIUS server after which it was successfully rebooted:
since that version of SAMBA is generally sane...and CentOS 5 is old (I'm surprised there are still updates....) sounds like winbind_privileged issues....you can either change group of that directory (contains the pipe file) to eg radiusd (to match group of your radiusd process) , or add radiusd user to the wbpriv group ...as , naughty naughty, installation of samba adjusts file permissions set by an admin in the post-install phase (bad RH people.....). /var/lib/samba for the winbindd_privileged directory PS move to 3.1.x on CentOS7 with SAMBA 4.3 install - you can then use native winbind connections and avoid this ntlm_auth issue alan
On Wed, Apr 20, 2016 at 06:12:28PM +0000, A.L.M.Buxey@lboro.ac.uk wrote:
/var/lib/samba for the winbindd_privileged directory
PS move to 3.1.x on CentOS7 with SAMBA 4.3 install - you can then use native winbind connections and avoid this ntlm_auth issue
FreeRADIUS will still need permission to access the winbind privileged pipe. It just talks to it directly rather than via ntlm_auth. Not sure there's any nice way to check and report if the permissions are correct, either; you need to know the location of the privileged pipe. It's possible to find out where it is[0], but I don't recall that libwbclient exposes that directly. Getting FreeRADIUS to write random things into winbind's unprivileged pipe to get the location is probably not the best idea. Maybe doing some other winbind info lookup that requires access to the priv pipe would be enough to warn what the problem is at FreeRADIUS start time. Cheers, Matthew [0] http://notes.asd.me.uk/2015/03/19/finding-the-winbind-privileged-pipe/ -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
participants (3)
-
A.L.M.Buxey＠lboro.ac.uk -
Jeremy Hill -
Matthew Newton