Invalid Authenticator... i.e. "munged" nt-key from Winbindd ...
Yes, I know this is really a Samba problem. I'm asking on this list because I really feel that a number of the users of ntlm_auth, winbindd are Radius admins. This is in regards to the "munged" nt-key bug in Winbindd. Most of the suggestions have been to simply upgrade Samba. From my reading, this all seems to go back to Samba 3.2.X'ish ? Well we are(were) running Samba 3.5.6. I figured that was relatively safe? Actually, I had noticed that the bug did still seem to exist, but would only occur after running Winbindd for a "while". I found other admins on the net reporting the same thing. We all seemed to adopt the same solution. Simply re-start Winbindd when the problem arose. This scheme worked very well for over a year. Then around 16:40 last Friday afternoon, something in our environment changed and this "bug" seemed to get tweaked all of the time. The radius servers just seemed to start to melt down. Actually, after a few hours 4 of 10 of our backend servers seemed to find a somewhat "stable" situation. In any case, I tried installing an older version of Samba 3.0.31 as there was some reference that nobody had seemed to see this problem with that version. However, that version did not do authentication at all against our win2008R2 directories. I found a bug report about that, and it basically said, "yes we know, we don't intend to fix it in 3.0.31 as that is an old version, upgrade". So, in any case, I did upgrade to the latest Samba 3.5.16 and things "seem" to be working now. After all said above, my real question is, has anybody seen anything somewhat definitive on this bug that would indicate the source of the problem has really been found and fixed ? Or, does it just seem that other changes to Winbindd have just "seemed" to make this bug go away (or hide better) ? The reason I ask, is that we use Freeradius here and we are a large R1 University with associated medical center. Our radius architecture is beginning to support not only the Campus, but the medical center as well. The plan is to really bring ALL of the medical center Wireless that requires authentication into our Freeradius architecture. Believe it or not, there are becoming more and more medical devices that are starting to have some wireless capabilities now.
From what I can tell, most of the use is to simply gather data about the device and ship it off to some master data gathering tool for analysis at a later time. However, I'm not sure, but some EKG devices in the future might start using this to actually ship the EKG results in real time to a doctor that is actually remotely located. This and other potential real time uses start to scare me a bit ??? I know that these devices should have some other backup capabilities for transmitting the data, but......
Thanks, Robert Robert Roll Computer Professional University of Utah (801) 581-7655
So, I can't speak to everything, but I can tell you that I found somewhat of an alternative to the Samba/Winbind setup that most folks run as stand-alone packages. We are an Ubuntu 12.04 LTS shop, and one of the packages we use for our Radius servers is Likewise-open5. Rather than having to configure Samba and Kerberos separately, you simply install that one package and you get all your REALM information populated on the server "automagically". From there, getting the NTLM_AUTH module to work was a breeze, and we've had none of the wonderful "Winbind" bugs since. There is something in the deb package with the version of Samba that it deploys that is somehow different. I'm not well enough equipped to figure it out myself, but I do know it works. I too had issues in my environment with Samba/Winbind and FreeRADIUS, and it ultimately drove me away from it when we went to Ubuntu. I haven't had a meltdown since. -----Original Message----- From: freeradius-users-bounces+jjulson=marketron.com@lists.freeradius.org [mailto:freeradius-users-bounces+jjulson=marketron.com@lists.freeradius.org] On Behalf Of Robert Roll Sent: Monday, July 30, 2012 9:14 AM To: FreeRadius users mailing list Subject: Invalid Authenticator... i.e. "munged" nt-key from Winbindd ... Yes, I know this is really a Samba problem. I'm asking on this list because I really feel that a number of the users of ntlm_auth, winbindd are Radius admins. This is in regards to the "munged" nt-key bug in Winbindd. Most of the suggestions have been to simply upgrade Samba. From my reading, this all seems to go back to Samba 3.2.X'ish ? Well we are(were) running Samba 3.5.6. I figured that was relatively safe? Actually, I had noticed that the bug did still seem to exist, but would only occur after running Winbindd for a "while". I found other admins on the net reporting the same thing. We all seemed to adopt the same solution. Simply re-start Winbindd when the problem arose. This scheme worked very well for over a year. Then around 16:40 last Friday afternoon, something in our environment changed and this "bug" seemed to get tweaked all of the time. The radius servers just seemed to start to melt down. Actually, after a few hours 4 of 10 of our backend servers seemed to find a somewhat "stable" situation. In any case, I tried installing an older version of Samba 3.0.31 as there was some reference that nobody had seemed to see this problem with that version. However, that version did not do authentication at all against our win2008R2 directories. I found a bug report about that, and it basically said, "yes we know, we don't intend to fix it in 3.0.31 as that is an old version, upgrade". So, in any case, I did upgrade to the latest Samba 3.5.16 and things "seem" to be working now. After all said above, my real question is, has anybody seen anything somewhat definitive on this bug that would indicate the source of the problem has really been found and fixed ? Or, does it just seem that other changes to Winbindd have just "seemed" to make this bug go away (or hide better) ? The reason I ask, is that we use Freeradius here and we are a large R1 University with associated medical center. Our radius architecture is beginning to support not only the Campus, but the medical center as well. The plan is to really bring ALL of the medical center Wireless that requires authentication into our Freeradius architecture. Believe it or not, there are becoming more and more medical devices that are starting to have some wireless capabilities now.
From what I can tell, most of the use is to simply gather data about the device and ship it off to some master data gathering tool for analysis at a later time. However, I'm not sure, but some EKG devices in the future might start using this to actually ship the EKG results in real time to a doctor that is actually remotely located. This and other potential real time uses start to scare me a bit ??? I know that these devices should have some other backup capabilities for transmitting the data, but......
Thanks, Robert Robert Roll Computer Professional University of Utah (801) 581-7655 - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html The information contained in this e-mail message may be confidential and protected from disclosure. If you are not the intended recipient, any dissemination, distribution or copying is strictly prohibited. If you think that you have received this e-mail message in error, please notify the sender immediately by replying to this message and then delete it from your system.
On 30/07/12 16:14, Robert Roll wrote:
This is in regards to the "munged" nt-key bug in Winbindd. Most of
Are you referring to this bug: https://bugzilla.samba.org/show_bug.cgi?id=6563 It looks to me like that bug has fallen into the weeds after being thought fixed. My advice would be to post on the Samba mailing list, and see if you can get someone interested.
to go back to Samba 3.2.X'ish ? Well we are(were) running Samba 3.5.6. I figured that was relatively safe? Actually, I had noticed that the bug did still seem to exist, but would only occur after running Winbindd for a "while". I found other admins on the net reporting the same thing. We all seemed to adopt the same solution. Simply re-start Winbindd when the problem arose.
This scheme worked very well for over a year. Then around 16:40 last Friday afternoon, something in our environment changed and this "bug" seemed to get tweaked all of the time. The radius servers just seemed to start to melt down. Actually, after a few hours 4 of 10 of our backend servers seemed to find a somewhat "stable" situation.
For what it's worth, we're running Samba 3.5.4 (RHEL5 package samba3x-3.5.4-0.70.el5) on Win2k8R2 DCs, and have no problems. Have you spoken to your AD admins? It seems likely some event (AD controller rebooting for patches?) triggered it. If you can figure out how to reproduce it, you can gather detailed debugging and hopefully solve the problem. Hell, if you can figure out how to reproduce it, *I* will crack out GDB and take a look.
After all said above, my real question is, has anybody seen anything somewhat definitive on this bug that would indicate the source of the problem has really been found and fixed ? Or, does it just seem that other changes to Winbindd have just "seemed" to make this bug go away (or hide better) ?
I know it's not what you want to hear, but this really *is* a Samba problem. Active Directory is, fundamentally, a closed system. You can only access it with the interfaces Microsoft makes available. Those interfaces are poorly documented, and have undesirable failure characteristics in the very best case.
However, I'm not sure, but some EKG devices in the future might start using this to actually ship the EKG results in real time to a doctor that is actually remotely located. This and other potential real time uses start to scare me a bit ??? I know that these devices should have some other backup capabilities for transmitting the data, but......
I'm sympathetic to your concerns but honestly, if you have a requirement for that level of reliability, my advice would be to abandon Active Directory for those credentials. It is relatively simple to store some credentials in a local users file or SQL database, and disable ntlm_auth for those users e.g. med-device-123 Cleartext-Password := "foo", MS-CHAP-Use-NTLM-Auth := 0 ...or equivalent in SQL. As well as being a lot more reliable, this approach has some other advantages - you don't necessarily want to use a "real" username for this kinds of embedded systems, and provisioning an AD account for them runs the risk of that account being given privileges it shouldn't have. If your local policy permits, and you can justify it, you could even do this will all users (use a password change policy DLL to capture all passwords to a database, optionally NT-hashed). But I doubt that's tenable. Alternatives include using EAP-TLS with client certs (horrible PKI mess) or EAP-TTLS/PAP and use a simpler method than ntlm_auth to check the PAP. In theory, EAP-TEAP (formerly EAP-FASTv2) with tickets on the client would solve this, but I see no realistic possibility of that appearing in client devices any time soon :o( Cheers, Phil
Yes, I do believe this is the bug in question. I did find this yesterday and noticed that while the problem may not happen 100% of the time, There are reports of it still happening. Even as late as version 3.5.10.. I am planning on adding my incident to the list... Thanks Much, Robert ________________________________________ From: freeradius-users-bounces+robert.roll=utah.edu@lists.freeradius.org [freeradius-users-bounces+robert.roll=utah.edu@lists.freeradius.org] on behalf of Phil Mayers [p.mayers@imperial.ac.uk] Sent: Monday, July 30, 2012 10:11 AM To: freeradius-users@lists.freeradius.org Subject: Re: Invalid Authenticator... i.e. "munged" nt-key from Winbindd ... On 30/07/12 16:14, Robert Roll wrote:
This is in regards to the "munged" nt-key bug in Winbindd. Most of
Are you referring to this bug: https://bugzilla.samba.org/show_bug.cgi?id=6563 It looks to me like that bug has fallen into the weeds after being thought fixed. My advice would be to post on the Samba mailing list, and see if you can get someone interested.
to go back to Samba 3.2.X'ish ? Well we are(were) running Samba 3.5.6. I figured that was relatively safe? Actually, I had noticed that the bug did still seem to exist, but would only occur after running Winbindd for a "while". I found other admins on the net reporting the same thing. We all seemed to adopt the same solution. Simply re-start Winbindd when the problem arose.
This scheme worked very well for over a year. Then around 16:40 last Friday afternoon, something in our environment changed and this "bug" seemed to get tweaked all of the time. The radius servers just seemed to start to melt down. Actually, after a few hours 4 of 10 of our backend servers seemed to find a somewhat "stable" situation.
For what it's worth, we're running Samba 3.5.4 (RHEL5 package samba3x-3.5.4-0.70.el5) on Win2k8R2 DCs, and have no problems. Have you spoken to your AD admins? It seems likely some event (AD controller rebooting for patches?) triggered it. If you can figure out how to reproduce it, you can gather detailed debugging and hopefully solve the problem. Hell, if you can figure out how to reproduce it, *I* will crack out GDB and take a look.
After all said above, my real question is, has anybody seen anything somewhat definitive on this bug that would indicate the source of the problem has really been found and fixed ? Or, does it just seem that other changes to Winbindd have just "seemed" to make this bug go away (or hide better) ?
I know it's not what you want to hear, but this really *is* a Samba problem. Active Directory is, fundamentally, a closed system. You can only access it with the interfaces Microsoft makes available. Those interfaces are poorly documented, and have undesirable failure characteristics in the very best case.
However, I'm not sure, but some EKG devices in the future might start using this to actually ship the EKG results in real time to a doctor that is actually remotely located. This and other potential real time uses start to scare me a bit ??? I know that these devices should have some other backup capabilities for transmitting the data, but......
I'm sympathetic to your concerns but honestly, if you have a requirement for that level of reliability, my advice would be to abandon Active Directory for those credentials. It is relatively simple to store some credentials in a local users file or SQL database, and disable ntlm_auth for those users e.g. med-device-123 Cleartext-Password := "foo", MS-CHAP-Use-NTLM-Auth := 0 ...or equivalent in SQL. As well as being a lot more reliable, this approach has some other advantages - you don't necessarily want to use a "real" username for this kinds of embedded systems, and provisioning an AD account for them runs the risk of that account being given privileges it shouldn't have. If your local policy permits, and you can justify it, you could even do this will all users (use a password change policy DLL to capture all passwords to a database, optionally NT-hashed). But I doubt that's tenable. Alternatives include using EAP-TLS with client certs (horrible PKI mess) or EAP-TTLS/PAP and use a simpler method than ntlm_auth to check the PAP. In theory, EAP-TEAP (formerly EAP-FASTv2) with tickets on the client would solve this, but I see no realistic possibility of that appearing in client devices any time soon :o( Cheers, Phil - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
participants (3)
-
Julson, Jim -
Phil Mayers -
Robert Roll