FreeRadius 2.1.12 with winbind - performance issues

[Matthew Newton]

On Thu, Jan 22, 2015 at 05:18:16PM -0500, John Douglass wrote:
> On 01/22/2015 04:42 PM, Matthew Newton wrote:
> > Welcome to the club.
> to kill off connections idle more the X (default 60s) time. When you hit
> the max number of DC connections winbind stops being able to
> authenticate and just crashes and burns. Restarting winbind when it gets
> near its threshold seems to help.

Ouch. Thankfully we've not seen that one...

> > Run more RADIUS servers, split the load from the controllers
> > across them. The WLCs will run out of RADIUS IDs with that number
> > of auths. Cisco "issue".
> We have been working very closely with Cisco and have a pre-alpha 8.x
> controller release we are testing that directly addresses this issue,
> but does not completely fix it. We have seen a definite decrease on the
> issues between controllers and radius servers but the back end seems to
> be the issue now (the Radius -> AD),

That's sounding very promising.

> But we are in WAYYYY far better shape than last year at this time.

:)

> > I've been working on patches to FR and samba to get FR to call
> > winbind directly rather than have to exec ntlm_auth. It shaves a
> > lot of time off not doing an exec, but the patches aren't merged
> > yet.
>
> Here at Georgia Tech we would absolutely be willing to patch, test, and
> compile any possible performance fixes between FR and winbind/samba. I
> have the knowledge, mandate, and testing infrastructure. We even have
> performance graphs on packets, radius logs, etc so we can verify
> performance and add/remove load if it breaks things.

Patches here:

FreeRADIUS 2 libwbclient patch:
  https://github.com/mcnewton/freeradius-server/tree/v2-winbind

FreeRADIUS 3 libwbclient / ntlm_auth socket patch:
  https://github.com/mcnewton/freeradius-server/commits/patch-mschap-ntlm-socket

Samba 3.6 patch (needs a tiny bit extra backported from Samba 4):
  https://gist.github.com/mcnewton/b9d4e9f43020243dcacf

Samba 4 patch:
  https://lists.samba.org/archive/samba-technical/2015-January/104947.html

The Samba patches are not required if using FreeRADIUS 3 with the
"ntlm_auth_helper" method - only with FR2 patch or FR3 "winbind"
method.

Note this removes the need to exec ntlm_auth on each
authentication, which in testing speeds up that part 2x. It
doesn't address any issues within winbind - but feedback on
whether it helps or not would be very much appreciated!


> > The patch for FR2 is simple. The patch for Samba (3 or 4) is
> > required because the libwbclient library is not currently
> > thread-safe. Putting a mutex around the auth call rather defeats
> > the point...
> >
> > Alternatively, there's a second patch for FR3 that uses ntlm_auth
> > in socket mode. This saves the exec time and doesn't need patching
> > Samba, but won't backport to FR2.

> I highly recommend moving from the 3.x Samba to the 4.x samba. I'm
> testing the enterprisesamba.com 4.1.12 includes a fix for the winbind
> request timeout:

That's what I suspected.

Cheers,

Matthew


-- 
Matthew Newton, Ph.D. <mcn4 at le.ac.uk>

Systems Specialist, Infrastructure Services,
I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom

For IT help contact helpdesk extn. 2253, <ithelp at le.ac.uk>