Hi, Just opening up for a bit of discussion on the best way to proceed with the ntlm_auth improvements I've been hacking around on. I've just been testing using libwbclient from FreeRADIUS. One RADIUS server has held up our entire wireless infrastructure for a couple of days, over all student movements between lectures, without a single complaint from the Cisco controllers about RADIUS timeouts. We normally hit problems when one RADIUS server gets to about 30 auths/second. This one has peaked at nearly 90 auths/second. This is nice - it seems execing ntlm_auth really is the problem (this is Samba 3.6.6, all running on HP DL380G6, 2-CPU Xen VMs ~1Gb RAM [mostly unused]). However - 1. It's running a version of libwbclient that has pthread support that can't be merged into Samba for a couple of reasons. It's safe for FR to use, but not for e.g. pam_winbind. So I'm working on a bigger Samba patch that hopefully can be merged. 2. Even when I get it done and it is merged into Samba, I guess it'll be a good couple of years or so before the code gets into distributions. 3. We could put a mutex around wbcAuthenticateUserEx (have tried, this works safely without a Samba patch) but that defeats the point of being able to do more than one authentication simultaneously. I've been unable to test FR3 yet. That would let me try the ntlm_auth helper option instead, which I can't easily try in FR2. I believe that this will offer similar performance improvements. It's on the list, but will take a while. When you look at the way ntlm_auth works, it connects to the winbind unpriv socket, 1 call over it to get the interface version, another call to get the priv socket location, closes the first socket, opens the priv socket, then sends the auth query. Using the ntlm_auth helper mode, and calling libwbclient directly, both do the first lot of setup once only, which means each auth goes straight to just doing the auth query. It's not surprising that doing a fork/exec, then all that setup first slows things down. So wondering what the best thing to do for FR is - noting pull request 848, which can't be merged as-is as it's unsafe with libwbclient. My thoughts are - Add ntlm_auth helper mode to 3.0.x now, which should be safe and run on anything that has ntlm_auth. And will be, IMO, nearly as fast as calling libwbclient directly. This should fix the AD auth issues for anyone with FR3. (I'm happy to provide patches as-is for Samba and FR2 for any that want, but they're not going to be merged.) Finish and submit patch to Samba, then add libwbclient mode either later on in 3.0.x or more likely to 3.1.x, due to the timescales of the Samba release. In discussions with the Samba guys, the API will change to do this anyway, which makes a configure test easy to determine if libwbclient is thread-safe. If not, then this method must be disabled. Thoughts? Cheers Matthew (who still can't believe it's possible that one small RADIUS server is handling our entire auth+acct load) -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
Add ntlm_auth helper mode to 3.0.x now, which should be safe and run on anything that has ntlm_auth. And will be, IMO, nearly as fast as calling libwbclient directly. This should fix the AD auth issues for anyone with FR3. (I'm happy to provide patches as-is for Samba and FR2 for any that want, but they're not going to be merged.)
IMO, that would be the sensible thing to do for a start.
Finish and submit patch to Samba, then add libwbclient mode either later on in 3.0.x or more likely to 3.1.x, due to the timescales of the Samba release.
Any improvement to Samba is always good. :-) Stefan Paetow Moonshot Industry & Research Liaison Coordinator t: +44 (0)1235 822 125 gpg: 0x3FCE5142 xmpp: stefanp@jabber.dev.ja.net skype: stefan.paetow.janet Lumen House, Library Avenue, Harwell Oxford, Didcot, OX11 0SG jisc.ac.uk Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800. Jisc Collections and Janet Ltd. is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under Company No. number 2881024, VAT No. GB 197 0632 86. The registered office is: Lumen House, Library Avenue, Harwell, Didcot, Oxfordshire, OX11 0SG. T 01235 822200.
On Fri, Feb 20, 2015 at 08:56:50AM +0000, Stefan Paetow wrote:
Finish and submit patch to Samba, then add libwbclient mode either later on in 3.0.x or more likely to 3.1.x, due to the timescales of the Samba release.
Any improvement to Samba is always good. :-)
In defence of Samba - the only "improvement" is to make the library thread-safe. Apart from that it works just fine. I understand there have been some improvements to the way winbind works from 4.2, but I've not looked at that. Cheers Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
Hi,
auths/second. This one has peaked at nearly 90 auths/second. This is nice - it seems execing ntlm_auth really is the problem (this is Samba 3.6.6, all running on HP DL380G6, 2-CPU Xen VMs ~1Gb RAM [mostly unused]).
oh its running ntlm_auth thats the bottleneck for sure - as the server can do many x100's more PEAP when using users file, LDAP or SQL backends.
Add ntlm_auth helper mode to 3.0.x now, which should be safe and run on anything that has ntlm_auth. And will be, IMO, nearly as fast as calling libwbclient directly. This should fix the AD auth issues for anyone with FR3. (I'm happy to provide patches as-is for Samba and FR2 for any that want, but they're not going to be merged.)
Finish and submit patch to Samba, then add libwbclient mode either later on in 3.0.x or more likely to 3.1.x, due to the timescales of the Samba release.
i thought we were going for all approaches andway - libwbclient method, ntlm_auth helper mode etc etc. I would just go for 3.0.x now anyway (I think Alan would say theres no choice, 2.x has no new features....) that MIGHT match the timescales for some distros anyway - and if there can be a run-time check for libwbclient thread-safe then it can use the feature. the other small performance tweak is to get the privileged file off disk - use tmpfs/ramdisk for the file (particularly in VMs!!) alan
On Fri, Feb 20, 2015 at 09:29:14AM +0000, A.L.M.Buxey@lboro.ac.uk wrote:
auths/second. This one has peaked at nearly 90 auths/second. This is nice - it seems execing ntlm_auth really is the problem (this is Samba 3.6.6, all running on HP DL380G6, 2-CPU Xen VMs ~1Gb RAM [mostly unused]).
oh its running ntlm_auth thats the bottleneck for sure - as the server can do many x100's more PEAP when using users file, LDAP or SQL backends.
Yeah, of course - but the question really is, is it execing ntlm_auth that is the bottleneck, or is it the handling of the auth through winbind. It seems that, in our case at present anyway, winbind can cope just fine.
Add ntlm_auth helper mode to 3.0.x now, which should be safe and run on anything that has ntlm_auth. And will be, IMO, nearly as fast as calling libwbclient directly. This should fix the AD auth issues for anyone with FR3. (I'm happy to provide patches as-is for Samba and FR2 for any that want, but they're not going to be merged.)
Finish and submit patch to Samba, then add libwbclient mode either later on in 3.0.x or more likely to 3.1.x, due to the timescales of the Samba release.
i thought we were going for all approaches andway - libwbclient method, ntlm_auth helper mode etc etc.
That was my idea and code, but Alan and Arran suggested in the PR to not have too many choices. I'm inclined to agree that eventually just authenticating via libwbclient is probably the right way to go (though some people might want the option to exec, if they're doing something esoteric or not with Samba). The question for me is the transition period, and the fact people are having auth issues now.
I would just go for 3.0.x now anyway (I think Alan would say theres no choice, 2.x has no new features....) that MIGHT match the timescales for some distros anyway - and if there can be a run-time check for libwbclient thread-safe then it can use the feature.
It's a build test - essentially whether wbcCtxAuthenticateUserEx exists in the library or not. I guess it could be done at runtime, but that gets a bit messy. Recompiling FreeRADIUS is quick and easy. Recompiling Samba takes a significant amount of time and is IMO a much bigger task. Hence using a distribution version is easier.
the other small performance tweak is to get the privileged file off disk - use tmpfs/ramdisk for the file (particularly in VMs!!)
Can't see that moving the privilege socket to tmpfs is likely to help much - it's only a socket? However, moving winbind's netsamlogon cache to tmpfs may help - we really don't care about uid mapping for this functionality, and there is no point winbind performing IO for each auth to keep a record of the uid numbers. But I've not dug into that enough yet to see if anything can or ought to be done, only that the files in /var/cache/samba are being continuously updated with Samba 3.6. Cheers! Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
On 20/02/15 01:00, Matthew Newton wrote:
Hi,
Just opening up for a bit of discussion on the best way to proceed with the ntlm_auth improvements I've been hacking around on.
I've just been testing using libwbclient from FreeRADIUS. One RADIUS server has held up our entire wireless infrastructure for a couple of days, over all student movements between lectures, without a single complaint from the Cisco controllers about RADIUS timeouts.
We normally hit problems when one RADIUS server gets to about 30 auths/second. This one has peaked at nearly 90 auths/second. This is nice - it seems execing ntlm_auth really is the problem
That makes "sense" (for some values of sense...) based on our experience. It's crazy that fork/exec of such a small binary, which is bound to be in-cache, is so slow, but I'm assuming it's actually some setup that ntlm_auth does.
Add ntlm_auth helper mode to 3.0.x now, which should be safe and run on anything that has ntlm_auth. And will be, IMO, nearly as fast as calling libwbclient directly. This should fix the AD auth
This sounds good.
issues for anyone with FR3. (I'm happy to provide patches as-is for Samba and FR2 for any that want, but they're not going to be merged.)
Finish and submit patch to Samba, then add libwbclient mode either later on in 3.0.x or more likely to 3.1.x, due to the timescales of the Samba release.
Sounds good.
On Fri, Feb 20, 2015 at 11:44:14AM +0000, Phil Mayers wrote:
On 20/02/15 01:00, Matthew Newton wrote:
We normally hit problems when one RADIUS server gets to about 30 auths/second. This one has peaked at nearly 90 auths/second. This is nice - it seems execing ntlm_auth really is the problem
That makes "sense" (for some values of sense...) based on our experience. It's crazy that fork/exec of such a small binary, which is bound to be in-cache, is so slow, but I'm assuming it's actually some setup that ntlm_auth does.
If you crank the winbind log level up to 5 or so, you can see the ntlm_auth calls - there are two setup calls (version + priv socket) before the auth. I can't imagine these, plus opening two sockets, takes that long, but it's still much more than just sending the authentication query over an already open socket.
Add ntlm_auth helper mode to 3.0.x now, which should be safe and run on anything that has ntlm_auth. And will be, IMO, nearly as fast as calling libwbclient directly. This should fix the AD auth
This sounds good. ...
Finish and submit patch to Samba, then add libwbclient mode either later on in 3.0.x or more likely to 3.1.x, due to the timescales of the Samba release.
Sounds good.
Alan and/or Arran, any thoughts on merging this? I think the main thing to sort out is probably the config syntax, unless there's some other way you'd prefer it done? FWIW, I finished the Samba patch over the weekend so they have that for review. I'm expecting I've done something not quite right, but hopefully it'll be mostly acceptable. Just not sure what timescales Samba work to and whether it will be able to hit 4.2 or have to wait for 4.3, which I guess could be a while off. Going forward, all the password-change stuff is in libwbclient as well, so it looks like ntlm_auth could be ditched entirely, but again needs the libwbclient Samba patch to be thread-safe. Cheers, Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
On Feb 23, 2015, at 6:35 PM, Matthew Newton <mcn4@leicester.ac.uk> wrote:
Alan and/or Arran, any thoughts on merging this? I think the main thing to sort out is probably the config syntax, unless there's some other way you'd prefer it done?
I’ll take a look. But I’m inclined to merge it into 3.1. I’d like to leave the changes to 3.0 as small as possible.
FWIW, I finished the Samba patch over the weekend so they have that for review. I'm expecting I've done something not quite right, but hopefully it'll be mostly acceptable. Just not sure what timescales Samba work to and whether it will be able to hit 4.2 or have to wait for 4.3, which I guess could be a while off.
Probably.
Going forward, all the password-change stuff is in libwbclient as well, so it looks like ntlm_auth could be ditched entirely, but again needs the libwbclient Samba patch to be thread-safe.
OK. Alan DeKok.
On Tue, Feb 24, 2015 at 08:14:12AM -0500, Alan DeKok wrote:
On Feb 23, 2015, at 6:35 PM, Matthew Newton <mcn4@leicester.ac.uk> wrote:
Alan and/or Arran, any thoughts on merging this? I think the main thing to sort out is probably the config syntax, unless there's some other way you'd prefer it done?
I’ll take a look. But I’m inclined to merge it into 3.1. I’d like to leave the changes to 3.0 as small as possible.
The only real thing here is that there seem to be quite a lot of sites (universities at least) that seem to be having auth trouble at the moment, so the sooner something to help can get into a released version the better. It might also help uptake of version 3 - I know there are a number of sites that are reluctant to jump from FR2 ("risk averse" is the usual disease), and this might provide a good reason. But aside from that, doesn't really matter when to me personally! Cheers, Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
On 24/02/15 13:48, Matthew Newton wrote:
On Tue, Feb 24, 2015 at 08:14:12AM -0500, Alan DeKok wrote:
On Feb 23, 2015, at 6:35 PM, Matthew Newton <mcn4@leicester.ac.uk> wrote:
Alan and/or Arran, any thoughts on merging this? I think the main thing to sort out is probably the config syntax, unless there's some other way you'd prefer it done?
I’ll take a look. But I’m inclined to merge it into 3.1. I’d like to leave the changes to 3.0 as small as possible.
The only real thing here is that there seem to be quite a lot of sites (universities at least) that seem to be having auth trouble at the moment, so the sooner something to help can get into a released version the better.
We're starting to see a trickle of auth problems again. Our huge uplift in server performance and extensive tweaking had apparently just pushed the boundary back, not solved the problem. So I can confirm we're interested in running with this code. I'll probably patch it in locally if it doesn't make it into 3.0.x
Hi,
We're starting to see a trickle of auth problems again. Our huge uplift in server performance and extensive tweaking had apparently just pushed the boundary back, not solved the problem.
we invested in EAP-TLS this year - so the PEAP crush is less of an issue (over 60% of clients are now EAP-TLS but clients configured over past years are still PEAP until they have issues and the users get their devices reconfigured). however, this performance improvement is still of interest - and there are some subtle tweaks that can be done to improve the ntlm_auth (and Cisco wireless controller) behaviour
So I can confirm we're interested in running with this code. I'll probably patch it in locally if it doesn't make it into 3.0.x
Its likely to go into 3.0.x - its 2.2.x thats not likely to be touched alan
On Feb 24, 2015, at 8:53 AM, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
So I can confirm we're interested in running with this code. I'll probably patch it in locally if it doesn't make it into 3.0.x
If it works and there’s enough demand, we can put it into 3.0.x. Arran and I have spent a fair bit of time the last 3 months hammering the daylights out of 3.0.x, to be sure it’s OK. It now builds cleanly under 3 different static analysis tools. It has a suite of regression tests. And it’s running in production in multiple environments. Alan DeKok.
On 24/02/15 15:54, Alan DeKok wrote:
Arran and I have spent a fair bit of time the last 3 months hammering the daylights out of 3.0.x, to be sure it’s OK. It now builds cleanly under 3 different static analysis tools. It has a suite of regression tests. And it’s running in production in multiple environments.
I can totally understand wanting to get 3.0.x stable. If it's disruptive, don't add it.
Hi, On Tue, Feb 24, 2015 at 03:56:46PM +0000, Phil Mayers wrote:
On 24/02/15 15:54, Alan DeKok wrote:
Arran and I have spent a fair bit of time the last 3 months hammering the daylights out of 3.0.x, to be sure it’s OK. It now builds cleanly under 3 different static analysis tools. It has a suite of regression tests. And it’s running in production in multiple environments.
I can totally understand wanting to get 3.0.x stable. If it's disruptive, don't add it.
Hopefully not - it essentially (in its current form) adds a new option to choose the ntlm/mschap auth type, and only uses the new code if that is set to the right value. Otherwise it calls ntlm_auth exactly as before. I've had about 4.6 million auths through the do_auth_wbclient code across the last week, and still going strong, so hopefully it's good enough :). The current pull request isn't quite ready to merge as-is, though the code is essentially the same. I added a new config option "auth_method" to choose the method that mschap should use for auth (rather than the current heuristic of "is ntlm_auth defined"). So the question is whether that is the best way to set the method, and whether the defaults are right. https://github.com/FreeRADIUS/freeradius-server/pull/848/files#diff-955fd7c4... I guess this also all depends on what the timescales for FR3.1 are, too! It's possible that the best way is to only the minimal config option now (which might not be the tidiest or most logical), and then completely revisit and tidy up the way mschap is configured for FR 3.1 or, more likely, FR 4. Cheers, Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
Hi,
If it works and there’s enough demand, we can put it into 3.0.x.
Arran and I have spent a fair bit of time the last 3 months hammering the daylights out of 3.0.x, to be sure it’s OK. It now builds cleanly under 3 different static analysis tools. It has a suite of regression tests. And it’s running in production in multiple environments.
since 3.0.5 I've been ready to state that 3.0.x is suitable for production (its at least as good as 2.2.6 ;-) ). I have deployed 3.0.6 onto several boxes recently ..some of which have already been upgraded to 3.0.7). shame I cant do that locally on our systems due to anti-agility systems in place :/ (we should have a project to deal with that) there are a couple of other systems I'd like to move to 3.0.7 from 2.2.x but they are rather antiquated and some of the new build requirements means that 3.0.x doesnt compile I think the issue for most will be the big reconfig required...and its understandable why some of the subtle annoying changes have been made - the server config is now much much more consistent, big thanks to Arran/Alan! :-) alan
participants (5)
-
A.L.M.Buxey＠lboro.ac.uk -
Alan DeKok -
Matthew Newton -
Phil Mayers -
Stefan Paetow