On Thu, Jan 22, 2015 at 05:18:16PM -0500, John Douglass wrote:
On 01/22/2015 04:42 PM, Matthew Newton wrote:
Welcome to the club. to kill off connections idle more the X (default 60s) time. When you hit the max number of DC connections winbind stops being able to authenticate and just crashes and burns. Restarting winbind when it gets near its threshold seems to help.
Ouch. Thankfully we've not seen that one...
Run more RADIUS servers, split the load from the controllers across them. The WLCs will run out of RADIUS IDs with that number of auths. Cisco "issue". We have been working very closely with Cisco and have a pre-alpha 8.x controller release we are testing that directly addresses this issue, but does not completely fix it. We have seen a definite decrease on the issues between controllers and radius servers but the back end seems to be the issue now (the Radius -> AD),
That's sounding very promising.
But we are in WAYYYY far better shape than last year at this time.
:)
I've been working on patches to FR and samba to get FR to call winbind directly rather than have to exec ntlm_auth. It shaves a lot of time off not doing an exec, but the patches aren't merged yet.
Here at Georgia Tech we would absolutely be willing to patch, test, and compile any possible performance fixes between FR and winbind/samba. I have the knowledge, mandate, and testing infrastructure. We even have performance graphs on packets, radius logs, etc so we can verify performance and add/remove load if it breaks things.
Patches here: FreeRADIUS 2 libwbclient patch: https://github.com/mcnewton/freeradius-server/tree/v2-winbind FreeRADIUS 3 libwbclient / ntlm_auth socket patch: https://github.com/mcnewton/freeradius-server/commits/patch-mschap-ntlm-sock... Samba 3.6 patch (needs a tiny bit extra backported from Samba 4): https://gist.github.com/mcnewton/b9d4e9f43020243dcacf Samba 4 patch: https://lists.samba.org/archive/samba-technical/2015-January/104947.html The Samba patches are not required if using FreeRADIUS 3 with the "ntlm_auth_helper" method - only with FR2 patch or FR3 "winbind" method. Note this removes the need to exec ntlm_auth on each authentication, which in testing speeds up that part 2x. It doesn't address any issues within winbind - but feedback on whether it helps or not would be very much appreciated!
The patch for FR2 is simple. The patch for Samba (3 or 4) is required because the libwbclient library is not currently thread-safe. Putting a mutex around the auth call rather defeats the point...
Alternatively, there's a second patch for FR3 that uses ntlm_auth in socket mode. This saves the exec time and doesn't need patching Samba, but won't backport to FR2.
I highly recommend moving from the 3.x Samba to the 4.x samba. I'm testing the enterprisesamba.com 4.1.12 includes a fix for the winbind request timeout:
That's what I suspected. Cheers, Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>