mschap via ntlm_auth over a socket
Hi, We've been hit with the same issue that a lot of other sites seem to have seen, which is that one freeradius server just doesn't seem to be able to authenticate over a certain number of users per second, somewhere around 30 or so, to our AD domain controllers. Standard ntlm_auth -> winbind -> AD stuff. We've done things like tweak "winbind max domain connections" and "winbind max clients", but can't seem to get winbind to connect to more than one DC, or seemingly parallelise anything in any way. Though reading archives it looks like we may have to use Samba 4 for that (though I still don't understand the reason for the max connections option if it can't/won't do it; I must me missing something). I've done some digging, and looking at the winbind debug logs, it seems to be taking around 3ms to do an auth, give or take. However, using sysdig to watch the ntlm_auth process, it takes around 30ms to run. That would figure with the maximum ~30 auths per second. Now I'm not entirely sure either of these times are completely accurate, but they were far enough off to get me looking. It seems ntlm_auth has a "--helper-protocol" option to enable it to start and then process requests over stdin/stdout. This should at least cut out the process exec time. So I've hacked around and updated the mschap module in a couple of ways to allow use of this. First is to add a new option to mschap, "method", which specifies the auth method to use. Currently this can be internal, or ntlm_auth, selected depending on whether ntlm_auth is defined or not. That will now still work for backwards compatibility, but if the new "method" option is set, that overrides it. The second is to add a new method, "ntlmauth_socket", which uses a connection pool to talk to a UNIX socket to send/receive auth data to ntlm_auth. To use it you then have to add ntlm_auth to inetd to create the socket. That handles spawning enough ntlm_auth processes off to meet demand as the connection pool stuff opens the sockets. I've not been able to test it in production yet; a) because I only just wrote it, and b) because our servers are still FR2. So if anyone else is running FR3 with a heavy load and willing to try, it would be good. I fear with the amount of change control (and/or paranoia) we have now, it's going to take me a while to get FR3 near the wireless controllers :( I have tried it both standalone against Samba, and against the domain. I'm getting these results. Note that the test (eapol_test) is run sequentially, so I'm artificially limiting the throughput there. All tests PEAP/EAP-MSCHAPv2 and FR threaded (not in debug mode). Standalone machine, Samba 3, not AD. 1,000 auths: internal mschap: 21.7s using ntlm_auth exec: 29.7s with socket: 22.9s so the socket looks decidedly faster, close to the internal mschap auth speed. Domain joined machine, Samba 3, 100 auths, eapol_test being run across the network from another machine: internal mschap: 6.3s using ntlm_auth exec: 8.1s with socket: 6.5s Only 100 auths in a loop this time (not to hammer the DC too much), but the socket version still manages to get 3 more auths in per second. Finally, domain joined machine, Samba 3, 100 auths, but direct mschap with radtest on the FR host rather than eapol_test remotely: internal mschap: 4.7s using ntlm_auth exec: 6.3s with socket: 4.9s So the real question (apart from whether I've completely missed something in the above results) is whether it's actually any better under real load. I've just done a pull request, but I'm sure there are things that need looking at or fixing even if the idea possibly sane. Let me know. Cheers, Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
On 2 Dec 2014, at 19:33, Matthew Newton <mcn4@LEICESTER.AC.UK> wrote:
Hi,
We've been hit with the same issue that a lot of other sites seem to have seen, which is that one freeradius server just doesn't seem to be able to authenticate over a certain number of users per second, somewhere around 30 or so, to our AD domain controllers. Standard ntlm_auth -> winbind -> AD stuff.
We've done things like tweak "winbind max domain connections" and "winbind max clients", but can't seem to get winbind to connect to more than one DC, or seemingly parallelise anything in any way. Though reading archives it looks like we may have to use Samba 4 for that (though I still don't understand the reason for the max connections option if it can't/won't do it; I must me missing something).
I've done some digging, and looking at the winbind debug logs, it seems to be taking around 3ms to do an auth, give or take. However, using sysdig to watch the ntlm_auth process, it takes around 30ms to run. That would figure with the maximum ~30 auths per second. Now I'm not entirely sure either of these times are completely accurate, but they were far enough off to get me looking.
It seems ntlm_auth has a "--helper-protocol" option to enable it to start and then process requests over stdin/stdout. This should at least cut out the process exec time. So I've hacked around and updated the mschap module in a couple of ways to allow use of this.
First is to add a new option to mschap, "method", which specifies the auth method to use. Currently this can be internal, or ntlm_auth, selected depending on whether ntlm_auth is defined or not. That will now still work for backwards compatibility, but if the new "method" option is set, that overrides it.
The second is to add a new method, "ntlmauth_socket", which uses a connection pool to talk to a UNIX socket to send/receive auth data to ntlm_auth.
To use it you then have to add ntlm_auth to inetd to create the socket. That handles spawning enough ntlm_auth processes off to meet demand as the connection pool stuff opens the sockets.
I've not been able to test it in production yet; a) because I only just wrote it, and b) because our servers are still FR2. So if anyone else is running FR3 with a heavy load and willing to try, it would be good. I fear with the amount of change control (and/or paranoia) we have now, it's going to take me a while to get FR3 near the wireless controllers :(
I have tried it both standalone against Samba, and against the domain. I'm getting these results. Note that the test (eapol_test) is run sequentially, so I'm artificially limiting the throughput there. All tests PEAP/EAP-MSCHAPv2 and FR threaded (not in debug mode).
Standalone machine, Samba 3, not AD. 1,000 auths:
internal mschap: 21.7s using ntlm_auth exec: 29.7s with socket: 22.9s
so the socket looks decidedly faster, close to the internal mschap auth speed.
Domain joined machine, Samba 3, 100 auths, eapol_test being run across the network from another machine:
internal mschap: 6.3s using ntlm_auth exec: 8.1s with socket: 6.5s
Only 100 auths in a loop this time (not to hammer the DC too much), but the socket version still manages to get 3 more auths in per second.
Finally, domain joined machine, Samba 3, 100 auths, but direct mschap with radtest on the FR host rather than eapol_test remotely:
internal mschap: 4.7s using ntlm_auth exec: 6.3s with socket: 4.9s
So the real question (apart from whether I've completely missed something in the above results) is whether it's actually any better under real load.
I've just done a pull request, but I'm sure there are things that need looking at or fixing even if the idea possibly sane. Let me know.
Very nice! Possible improvement (and I may be completely wrong here) but shouldn't it be possible to fork/exec and create a pipe with the ends mapped to stdin/stdout of the execed process? Slightly less configuration, and maybe slight performance improvement. Could just look for --helper-protocol option in the arguments to ntlm_auth and enable the connection pool automagically... In this case we could likely avoid exposing the knobs of the connection pool, as there's no benefit to tweaking anything, and just fork/exec max_servers instances of ntlm_auth? Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
On Tue, Dec 02, 2014 at 08:28:07PM -0500, Arran Cudbard-Bell wrote:
On 2 Dec 2014, at 19:33, Matthew Newton <mcn4@LEICESTER.AC.UK> wrote: I've just done a pull request, but I'm sure there are things that need looking at or fixing even if the idea possibly sane. Let me know.
Very nice! Possible improvement (and I may be completely wrong here) but shouldn't it be possible to fork/exec and create a pipe with the ends mapped to stdin/stdout of the execed process?
Slightly less configuration, and maybe slight performance improvement.
Yes, that should work. I thought about it, but the connection pool stuff seems designed to make connections, rather than spawn off processes. Is it safe for mod_conn_create (callback from fr_connection_pool_module_init) to fork/exec and return a struct with file descriptors? Then I guess _mod_conn_free would need to wait on the child. It seemed to start getting more complicated...
Could just look for --helper-protocol option in the arguments to ntlm_auth and enable the connection pool automagically...
Personally I think it's better to specify what you want to use, rather than for the server to guess - the current way is about OK as the behaviour is dependent on one config option being set, but it starts to get messy when you're doing heuristics to find the method. (Though 'method' might be better called 'backend' or 'auth_backend'; maybe it's OK as-is). The other possibility I'm thinking about next is another method that uses the wbclient library to call winbind directly, but I've not looked into that too much yet. Of course the library may not be available on all systems, so giving the option to run ntlm_auth should stay (and people may be calling a different binary rather than ntlm_auth in some situations).
In this case we could likely avoid exposing the knobs of the connection pool, as there's no benefit to tweaking anything, and just fork/exec max_servers instances of ntlm_auth?
Possible. But I'm not yet sure what the implications of this are, as it looks like there may still be throughput issues going through winbind. It might be better to limit the number of connections, rather than pushing them all to winbind and that dropping or queueing them. Squid seems to have had a similar feature for a long time, and allows the number of ntlm_auth processes to be set, but I don't know whether there is any benefit to keeping the number down or just running loads of them. Cheers, Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
On 3 Dec 2014, at 06:05, Matthew Newton <mcn4@LEICESTER.AC.UK> wrote:
On Tue, Dec 02, 2014 at 08:28:07PM -0500, Arran Cudbard-Bell wrote:
On 2 Dec 2014, at 19:33, Matthew Newton <mcn4@LEICESTER.AC.UK> wrote: I've just done a pull request, but I'm sure there are things that need looking at or fixing even if the idea possibly sane. Let me know.
Very nice! Possible improvement (and I may be completely wrong here) but shouldn't it be possible to fork/exec and create a pipe with the ends mapped to stdin/stdout of the execed process?
Slightly less configuration, and maybe slight performance improvement.
Yes, that should work. I thought about it, but the connection pool stuff seems designed to make connections, rather than spawn off processes. Is it safe for mod_conn_create (callback from fr_connection_pool_module_init) to fork/exec and return a struct with file descriptors?
Yes, it's safe.
Then I guess _mod_conn_free would need to wait on the child. It seemed to start getting more complicated...
It's not that complicated, _mod_conn_free signals the child to exit, and waits on it.
Could just look for --helper-protocol option in the arguments to ntlm_auth and enable the connection pool automagically...
Personally I think it's better to specify what you want to use, rather than for the server to guess - the current way is about OK as the behaviour is dependent on one config option being set, but it starts to get messy when you're doing heuristics to find the method. (Though 'method' might be better called 'backend' or 'auth_backend'; maybe it's OK as-is).
I guess, it's just A DRY thing. It's annoying to have to configure the argument and tweak the knob inside FreeRADIUS. I don't see any reason people would wrap ntlm_auth, or why the arguments would change. I know in some instances trying to be smart breaks things in unexpected ways, but the interface for the ntlm_auth binary is well defined. Do you know when the --helper-protocol flag was added? If it's been there for a significant period we might want to consider just adding it automatically if not present, and switching to the new method for 3.1.
The other possibility I'm thinking about next is another method that uses the wbclient library to call winbind directly, but I've not looked into that too much yet. Of course the library may not be available on all systems, so giving the option to run ntlm_auth should stay (and people may be calling a different binary rather than ntlm_auth in some situations).
That's certainly the cleanest solution.
In this case we could likely avoid exposing the knobs of the connection pool, as there's no benefit to tweaking anything, and just fork/exec max_servers instances of ntlm_auth?
Possible. But I'm not yet sure what the implications of this are, as it looks like there may still be throughput issues going through winbind. It might be better to limit the number of connections, rather than pushing them all to winbind and that dropping or queueing them.
Squid seems to have had a similar feature for a long time, and allows the number of ntlm_auth processes to be set, but I don't know whether there is any benefit to keeping the number down or just running loads of them.
Ok, fair enough, I guess keeping it as a tweakable to manage contention would be useful. Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
On 03/12/14 15:59, Arran Cudbard-Bell wrote:
I don't see any reason people would wrap ntlm_auth, or why the arguments would change.
We *do* wrap it currently, and I can think of not-ridiculous reasons to do so in some cases.
I know in some instances trying to be smart breaks things in unexpected ways, but the interface for the ntlm_auth binary is well defined.
Do you know when the --helper-protocol flag was added?
It's old. Squid used it years ago.
On 3 Dec 2014, at 11:06, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
On 03/12/14 15:59, Arran Cudbard-Bell wrote:
I don't see any reason people would wrap ntlm_auth, or why the arguments would change.
We *do* wrap it currently, and I can think of not-ridiculous reasons to do so in some cases.
Do you wrap it in such a way that it would not pass the --helper-protocol flag through? Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
On 03/12/14 16:07, Arran Cudbard-Bell wrote:
On 3 Dec 2014, at 11:06, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
On 03/12/14 15:59, Arran Cudbard-Bell wrote:
I don't see any reason people would wrap ntlm_auth, or why the arguments would change.
We *do* wrap it currently, and I can think of not-ridiculous reasons to do so in some cases.
Do you wrap it in such a way that it would not pass the --helper-protocol flag through?
No, can't think of a reason to do that.
On Wed, Dec 03, 2014 at 11:07:15AM -0500, Arran Cudbard-Bell wrote:
On 3 Dec 2014, at 11:06, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
On 03/12/14 15:59, Arran Cudbard-Bell wrote:
I don't see any reason people would wrap ntlm_auth, or why the arguments would change.
We *do* wrap it currently, and I can think of not-ridiculous reasons to do so in some cases.
Do you wrap it in such a way that it would not pass the --helper-protocol flag through?
I think Phil's current reason to wrap it (to get timings) would not be applicable with the helper-protocol flag. But I'm sure there may be other reasons. I like both options - the idea that the server will do the smart(TM) thing usually, but giving the admin the option to force an option when they want. Though you already have to alter the config to tell it to do ntlm_auth (by un-commenting and setting the cmd line), so in some way adding a new "method" is no different, and possibly a bit easier to see what's happening, e.g. current: # internal auth: # ntlm_auth = "/path/to/binary... --args" ntlm_auth_timeout = 10 # ntlm_auth: ntlm_auth = "/path/to/binary... --args" ntlm_auth_timeout = 10 new: # internal auth: # configure mschap backend here: method = internal # other options for the methods: ntlm_auth = "/path/to/binary... --args" ntlm_auth_timeout = 10 ntlmauth_socket { socket = "/run/ntlmauth.sock" } # ntlm_auth: # configure mschap backend here: method = ntlmauth_exec # other options for the methods: ntlm_auth = "/path/to/binary... --args" ntlm_auth_timeout = 10 ntlmauth_socket { socket = "/run/ntlmauth.sock" } The latter (to me) seems clearer what you're configuring it to do. Then in the future, maybe: # winbind: # configure mschap backend here: method = winbind # other options for the methods: ntlm_auth = "/path/to/binary... --args" ntlm_auth_timeout = 10 ntlmauth_socket { socket = "/run/ntlmauth.sock" } to use a samba library. Means only changing one option. After a few releases where, say, the "winbind" or "ntlmauth_socket" option is the default and no failure reports (e.g. reasons for people to want to force "ntlm_auth" by exec again), maybe ntlmauth_exec could be removed. Just thinking out loud. Maybe there's a cleaner way. Actually the first method I though of was to just split mschap into multiple modules, one for each auth type. Then realised how much was actually in the mschap module... Cheers Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
On Wed, Dec 03, 2014 at 10:59:55AM -0500, Arran Cudbard-Bell wrote:
On 3 Dec 2014, at 06:05, Matthew Newton <mcn4@LEICESTER.AC.UK> wrote:
On Tue, Dec 02, 2014 at 08:28:07PM -0500, Arran Cudbard-Bell wrote:
On 2 Dec 2014, at 19:33, Matthew Newton <mcn4@LEICESTER.AC.UK> wrote: I've just done a pull request, but I'm sure there are things that need looking at or fixing even if the idea possibly sane. Let me know.
Very nice! Possible improvement (and I may be completely wrong here) but shouldn't it be possible to fork/exec and create a pipe with the ends mapped to stdin/stdout of the execed process?
Slightly less configuration, and maybe slight performance improvement.
OK, that was simpler than I thought. I've also updated a few other bits and pushed to github. https://github.com/FreeRADIUS/freeradius-server/pull/848 Testing looks similar (though times are less; possibly due to compiling without enable-developer, or pulseaudio not eating vast amounts of CPU). 1000 iterations of eapol_test on a single box with Samba backend. internal 15.9s ntlm_auth 23.9s ntlm_auth_helper 17.0s Cheers, Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
Hi, any performance improvements for ntlm_auth are welcome...especially a c. 25% increase care to port to 2.2.x release? :-)
We've done things like tweak "winbind max domain connections" and "winbind max clients", but can't seem to get winbind to connect to more than one DC, or seemingly parallelise anything in any way.
no. binbind in 3.x is dumb... the version with samba 4 has the ability to load balance but , from what i've seen with CentOS7/RH7 releases is they havent enabled/allowed that feature so you have to build from source - but there have been MANY bug and performance fixes in 4.x for winbindd too
Though reading archives it looks like we may have to use Samba 4 for that (though I still don't understand the reason for the max connections option if it can't/won't do it; I must me missing something).
for 3.x we see that option working - you may have to check your windows server to see how mnay connections it allows - as its limited by default at that end
It seems ntlm_auth has a "--helper-protocol" option to enable it to start and then process requests over stdin/stdout. This should at least cut out the process exec time. So I've hacked around and updated the mschap module in a couple of ways to allow use of this.
another thought for performance is to not use disk for the TDB files - especially the winbind_privileged pipe.
it would be good. I fear with the amount of change control (and/or paranoia) we have now, it's going to take me a while to get FR3 near the wireless controllers :(
similar here - the jump to FR3 is a big move (we're using it for new systems for other purposes though! - hopefully this will gain us the uptime/knowledge etc etc so smooth the transition for the wireless stuff
domain. I'm getting these results. Note that the test (eapol_test) is run sequentially, so I'm artificially limiting the throughput there. All tests PEAP/EAP-MSCHAPv2 and FR threaded (not in debug mode).
tempting to run a parallel eapol_test flood :-) alan
On 03/12/14 00:33, Matthew Newton wrote:
Hi,
We've been hit with the same issue that a lot of other sites seem to have seen, which is that one freeradius server just doesn't seem to be able to authenticate over a certain number of users per second, somewhere around 30 or so, to our AD domain controllers. Standard ntlm_auth -> winbind -> AD stuff.
Ah, the AD auth spiral of doom. Very puzzling that one. Never really got to the bottom of what exactly the cause was - I think it was a mix of hardware, kernel, samba and AD problems at our site.
We've done things like tweak "winbind max domain connections" and "winbind max clients", but can't seem to get winbind to connect to more than one DC, or seemingly parallelise anything in any way.
Which version of Samba are you using, and how are you determining there's no parallelism? We're running on Samba 3.6.9 on RHEL6, and have "winbind max domain connections = 12", and with "lsof -i :445" we see many windbind processes and separate TCP connections after spikes of load. However, the parallelism is complex - in Samba 3.x it's only to one DC, not several, and you run into issues with failed auths being punted to the PDC emulator.
Though reading archives it looks like we may have to use Samba 4 for that (though I still don't understand the reason for the max connections option if it can't/won't do it; I must me missing something).
As above, works for us.
I've done some digging, and looking at the winbind debug logs, it seems to be taking around 3ms to do an auth, give or take. However, using sysdig to watch the ntlm_auth process, it takes
I've got C source for a tiny wrapper that logs process start/stop times to an append file, which I found useful for instrumentation. You might also find it useful to setup a rolling tcpdump ringbuffer capture to the DCs, and use "tshark -T fields" to dump out the msrpc header and packet time - although payload is encrypted, you can correlate request ID in request/response payload to get on-the-wire auth times for AD. We found the latter very useful.
The second is to add a new method, "ntlmauth_socket", which uses a connection pool to talk to a UNIX socket to send/receive auth data to ntlm_auth.
Handy. FWIW I think this is a better solution long-term for big/busy sites - it avoids process startup overhead completely. I have no time to test right now unfortunately, and like you ITIL has kept 3.x away from our radius servers :o(
Ah, the AD auth spiral of doom. Very puzzling that one. Never really got to the bottom of what exactly the cause was
Have you looked at setting Windows registry entry MaxConcurrentApi (see https://support.microsoft.com/kb/975363?wa=wsignin1.0)? ============================================================== Please consider the environment before printing this email. If you have received this message in error, please notify the sender and immediately delete this message and any attachment hereto and/or copy hereof, as such message contains confidential information intended solely for the individual or entity to whom it is addressed. The use or disclosure of such information to third parties is prohibited by law and may give rise to civil or criminal liability. The views presented in this message are solely those of the author(s) and do not necessarily represent the opinion of Iberdrola USA Networks, Inc. or any company of its group. Neither Iberdrola USA Networks, Inc. nor any company of its group guarantees the integrity, security or proper receipt of this message. Likewise, neither Iberdrola USA Networks, Inc. nor any company of its group accepts any liability whatsoever for any possible damages arising from, or in connection with, data interception, software viruses or manipulation by third parties. ==============================================================
On Wed, Dec 03, 2014 at 01:18:49PM +0000, Garber, Neal wrote:
Ah, the AD auth spiral of doom. Very puzzling that one. Never really got to the bottom of what exactly the cause was
Have you looked at setting Windows registry entry MaxConcurrentApi (see https://support.microsoft.com/kb/975363?wa=wsignin1.0)?
Thanks, yes - looked at that. I'm not currently convinced it's that limit we're hitting (but I'm probably reading the numbers incorrectly...) and it looks more like the ntlm_auth/winbind side. But having said that, it'll be easier to get FR3 in live than change a registry setting on a DC here. And getting any traction on FR3 is hard enough. Cheers, Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
On 03/12/14 13:18, Garber, Neal wrote:
Ah, the AD auth spiral of doom. Very puzzling that one. Never really got to the bottom of what exactly the cause was
Have you looked at setting Windows registry entry MaxConcurrentApi (see https://support.microsoft.com/kb/975363?wa=wsignin1.0)?
Yes that and a zillion other things were suggested by a zillion people the zillion times it has come up. FWIW this setting has sensible defaults on Win2012 R2, so people using such can ignore it IIRC.
Hi, On Wed, Dec 03, 2014 at 01:00:09PM +0000, Phil Mayers wrote:
On 03/12/14 00:33, Matthew Newton wrote:
We've done things like tweak "winbind max domain connections" and "winbind max clients", but can't seem to get winbind to connect to more than one DC, or seemingly parallelise anything in any way.
Which version of Samba are you using, and how are you determining there's no parallelism?
Debian wheezy - winbind 3.6.6. lsof shows 5 connections to a single DC, as you say. But tcpdump shows essentially all connections coming from a single TCP source port. Argus (network flow logging) shows the same. There's a very negligible percentage from the other source ports, even during busy periods. Also, winbind debug logs (-d4) show each request with "child daemon request 14" / "Finished processing child request 14", which seems to be a connection number. Very infrequently I may see a number other than 14, which I take to be a parallel request, but this is a) not often and b) not when the system is under heavy auth, which is when I'd expect to see it. But I may have misunderstood the "child request" number...
We're running on Samba 3.6.9 on RHEL6, and have "winbind max domain connections = 12", and with "lsof -i :445" we see many windbind processes and separate TCP connections after spikes of load.
Yes, many winbinds running (one per connection plus a parent).
I've done some digging, and looking at the winbind debug logs, it seems to be taking around 3ms to do an auth, give or take. However, using sysdig to watch the ntlm_auth process, it takes
I've got C source for a tiny wrapper that logs process start/stop times to an append file, which I found useful for instrumentation.
You might also find it useful to setup a rolling tcpdump ringbuffer capture to the DCs, and use "tshark -T fields" to dump out the msrpc header and packet time - although payload is encrypted, you can correlate request ID in request/response payload to get on-the-wire auth times for AD.
Thanks. Actually, on a quiet RADIUS server it looks like the normal request time is just over 1ms. I guess the question is if it goes up significantly for a busy server, which that would show. Running winbind in debug (-d4) produces some interesting logs, which show (with the child daemon request / Finished processing lines) the start/stop of each auth, which seems to be a fairly constant 3ms, even in busy times.
The second is to add a new method, "ntlmauth_socket", which uses a connection pool to talk to a UNIX socket to send/receive auth data to ntlm_auth.
Handy. FWIW I think this is a better solution long-term for big/busy sites - it avoids process startup overhead completely.
I have no time to test right now unfortunately, and like you ITIL has kept 3.x away from our radius servers :o(
Yeah. I might be able to use this as an excuse to get it in somehow. But really, it shouldn't be that hard. I guess progress these days is made by standing still. Cheers, Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
On 03/12/14 14:00, Matthew Newton wrote:
Debian wheezy - winbind 3.6.6.
lsof shows 5 connections to a single DC, as you say. But tcpdump shows essentially all connections coming from a single TCP source
Also, winbind debug logs (-d4) show each request with "child daemon request 14" / "Finished processing child request 14", which
Both those imply to me that winbind isn't seeing the need to use a parallel connection, almost as if the *offered* load is being limited before hitting winbind pipe dispatch, as opposed to anything else. Concurrency in winbind prior to the dispatch? Or before connecting to the pipe e.g. locking inside the .tdb files? We're on pretty quick hardware that might be hiding some of these now.
Thanks. Actually, on a quiet RADIUS server it looks like the normal request time is just over 1ms. I guess the question is if it goes up significantly for a busy server, which that would show.
For an MSCHAP auth RPC, round-trip, or ntlm_auth start-to-end? For comparison, we see: on-the-wire RTT for the RPCs * mean/median 4.1/3.9msec * stddev 2.8 * 5-95%ile range 0.5-8.3msec start-to-end ntlm_auth: * mean/median 16.4/12.2msec * stddev 21.9 * 5-95%ile range 10.0-56.3msec However, there's a very odd double-peak structure to the ntlm_auth times with a second, much smaller peak at around 50-60msec, which I don't know the cause of.
On Thu, Dec 04, 2014 at 02:38:06PM +0000, Phil Mayers wrote:
On 03/12/14 14:00, Matthew Newton wrote:
Debian wheezy - winbind 3.6.6.
lsof shows 5 connections to a single DC, as you say. But tcpdump shows essentially all connections coming from a single TCP source
Also, winbind debug logs (-d4) show each request with "child daemon request 14" / "Finished processing child request 14", which
Both those imply to me that winbind isn't seeing the need to use a parallel connection, almost as if the *offered* load is being limited before hitting winbind pipe dispatch, as opposed to anything else.
I did semi wonder about that, then got into the thought that the ntlm_auth exec/setup may be the slow bit, hence looking at the helper option. I've not dug much more into the winbind side (yet...)
Concurrency in winbind prior to the dispatch? Or before connecting to the pipe e.g. locking inside the .tdb files?
Hmm, is it really hitting disk for each auth? That wouldn't help. I might try sticking /var/lib/samba on tmpfs temporarily and see if it makes any difference.
We're on pretty quick hardware that might be hiding some of these now.
These are on fairly small Xen VMs. 2 CPU; was on 384mb RAM, but bumped it up to a gig as there was a little in swap, but they weren't actively swapping. Now there's stacks of free RAM, so I don't think there was any issue there really. The biggest thing we've done which has helped clear the mud from the water was to stop the wireless controllers jumping RADIUS server at the first sign of trouble. They were moving around every 30s or so at peak periods which makes the problem far far worse as all EAP sessions get chopped off and everything has to start again, compounding the issue.
Thanks. Actually, on a quiet RADIUS server it looks like the normal request time is just over 1ms. I guess the question is if it goes up significantly for a busy server, which that would show.
For an MSCHAP auth RPC, round-trip, or ntlm_auth start-to-end?
Sorry, tcpdump roundtrip - time from first packet to DC, to response from DC.
However, there's a very odd double-peak structure to the ntlm_auth times with a second, much smaller peak at around 50-60msec, which I don't know the cause of.
Login failures, maybe? Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
participants (5)
-
A.L.M.Buxey＠lboro.ac.uk -
Arran Cudbard-Bell -
Garber, Neal -
Matthew Newton -
Phil Mayers