Setting up a Proxy Server as a load balancer to bypass winbind/samba bottleneck

[John Douglass]

Walter,

Georgia Tech was having similar issues and has been working with Cisco
on some controller code improvements. Previously their implementation
was overrunning the 255 limit on a single channel from the controller to
the radius server. Check your controller code and there may be some
updates you need to apply regarding the multiple source ports (I believe
the default is now to automatically turn them on).

Cisco has since added 8 different source ports to the controller code
(this is in one of the most recently released versions) and we are
testing an additional fix now since that code still had a few issues
under load.

We utilize 3 wireless ONLY active directory servers in VMs and as
someone else pointed out, you really MUST be using a more modern version
of Samba in order to increase the connections from winbind to AD with
configurations in your smb.conf such as:

# ------------ winbind configuration
-----------------------------------------

   winbind max clients = 16192
   winbind max domain connections = 256
   winbind request timeout = 5
   winbind reconnect delay = 2

NOW this doesn't allow for round robin attachment to AD but for us,
combining the upgraded controller code, keeping up to date on freeradius
versions, and using a more up to date version of samba (we are using
Version 4.2.0 compiled on some and the now for subscription fee
enterprisesamba.org packages Version 4.2.4-SerNet-RedHat-18.el6
) and that allowed for the above configuration improvements, we have
seen a markedly improve authentication experience for our users.

We generally run about between 29 and 60 connections to AD. Our AD
server had the modifications in max-concurrent-api connections as
described by this document we found:

     https://support.microsoft.com/en-us/kb/2688798

In regards to Freeradius, version 3 has the native ntlm_auth calls now
(we initially tested some beta code from the original engineer on the
2.x version) and THAT markedly improved performance.

In short:
1) Update your controller code, look for the multi source-port queues
2) Update your SAMBA and increase the winbind options to suit your needs
3) Update your Freeradius version and configure it for the integrated
winbind stuff (see the mods-available/mschap file for documentation)
4) Increase your maxConcurrentApi settings on your AD server (this might
help as well)

- John Douglass, Georgia Institute of Technology.

On 10/16/2015 11:53 AM, Walter Reynolds wrote:
> We are running to the same problems with winbind/samba bottlenecks as many
> other universities.  While one of our servers does quite well we are
> starting to get hung modules at peak hitting as many as 150 authentications
> per second while averaging over 70/sec for several minutes.
>
> Since wireless controllers are getting bigger and with Cisco we are limited
> to sending radius auth request to one server at a time we need another way
> to balance things out.  I thought setting up a freeradius proxy that
> pointed to two separate servers might be a way to go about this.  Problem
> is I thought I read somewhere in the freeradius lists that that would
> create other problems, but now I can not find that.  I also am not sure I
> followed this right, but in looking at proxy home_server_pool it looks like
> my options for load balancing will still send all auth request from my one
> large controller to a single server as the source IP will all be the same
> under client-balance.
>
> Thanks.
> ------------------------
> Walter Reynolds
> Principal Systems Security Development Engineer
> Information and Technology Services
> University of Michigan
> (734) 615-9438
> -
> List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html