Setting up a Proxy Server as a load balancer to bypass winbind/samba bottleneck
We are running to the same problems with winbind/samba bottlenecks as many other universities. While one of our servers does quite well we are starting to get hung modules at peak hitting as many as 150 authentications per second while averaging over 70/sec for several minutes. Since wireless controllers are getting bigger and with Cisco we are limited to sending radius auth request to one server at a time we need another way to balance things out. I thought setting up a freeradius proxy that pointed to two separate servers might be a way to go about this. Problem is I thought I read somewhere in the freeradius lists that that would create other problems, but now I can not find that. I also am not sure I followed this right, but in looking at proxy home_server_pool it looks like my options for load balancing will still send all auth request from my one large controller to a single server as the source IP will all be the same under client-balance. Thanks. ------------------------ Walter Reynolds Principal Systems Security Development Engineer Information and Technology Services University of Michigan (734) 615-9438
On 16/10/15 16:53, Walter Reynolds wrote:
my options for load balancing will still send all auth request from my one large controller to a single server as the source IP will all be the same under client-balance.
There's a bunch of reasons this might not help which I don't have time to go into right now (or it might) but this isn't one of them. See "keyed-balance". You can choose to loadbalance the request on any value in the packet you like. Obviously pick to to be consistent between EAP packets - User-Name or Calling-Station-Id or some combination of them.
Upgrade to 3.0.10 on a box with samba 4.1.x or higher. Then you can use the native winbind libraries in a connection pool. No need for other servers. Big speed improvements and server will talk to all AD auth servers and not stick on one. Alternatively look at better, faster and more scalable EAP method eg EAP-TLS alan
On Oct 16, 2015, at 1:50 PM, Alan Buxey <A.L.M.Buxey@lboro.ac.uk> wrote:
Upgrade to 3.0.10 on a box with samba 4.1.x or higher.
Nah it's Samba 4.2.1. Unfortunately no distros currently package it. Boo. https://wiki.samba.org/index.php/Samba_4.2_Features_added/changed (search for newt) -Arran
Hi,
On Oct 16, 2015, at 1:50 PM, Alan Buxey <A.L.M.Buxey@lboro.ac.uk> wrote:
Upgrade to 3.0.10 on a box with samba 4.1.x or higher.
Nah it's Samba 4.2.1. Unfortunately no distros currently package it. Boo.
sorry yes, SAMBA 4.2.1 or later... (its up to 4.2.4 now) - and yes, not packaged even though the current STABLE is 4.3.0 ! alan
On Fri, Oct 16, 2015 at 06:28:05PM +0000, A.L.M.Buxey@lboro.ac.uk wrote:
Nah it's Samba 4.2.1. Unfortunately no distros currently package it. Boo.
sorry yes, SAMBA 4.2.1 or later... (its up to 4.2.4 now) - and yes, not packaged even though the current STABLE is 4.3.0 !
Distros are far more up to date with versions of Samba than versions of FreeRADIUS :( Samba 4.3.x only came out in the last few months, which explains why no distros have it. Debian has 4.3.0 in experimental, for example. I'm in the process of trying to upgrade our RADIUS servers to v3 (at last). As I actually want to use the code I spent ~3 months writing, I'll probably just install Samba from source into /opt for now. It's really not hard. For the record, I'm looking at building in a proxy layer of RADIUS servers using keyed-balance and Calling-Station-Id. At least, that's the current idea... Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
Walter, Georgia Tech was having similar issues and has been working with Cisco on some controller code improvements. Previously their implementation was overrunning the 255 limit on a single channel from the controller to the radius server. Check your controller code and there may be some updates you need to apply regarding the multiple source ports (I believe the default is now to automatically turn them on). Cisco has since added 8 different source ports to the controller code (this is in one of the most recently released versions) and we are testing an additional fix now since that code still had a few issues under load. We utilize 3 wireless ONLY active directory servers in VMs and as someone else pointed out, you really MUST be using a more modern version of Samba in order to increase the connections from winbind to AD with configurations in your smb.conf such as: # ------------ winbind configuration ----------------------------------------- winbind max clients = 16192 winbind max domain connections = 256 winbind request timeout = 5 winbind reconnect delay = 2 NOW this doesn't allow for round robin attachment to AD but for us, combining the upgraded controller code, keeping up to date on freeradius versions, and using a more up to date version of samba (we are using Version 4.2.0 compiled on some and the now for subscription fee enterprisesamba.org packages Version 4.2.4-SerNet-RedHat-18.el6 ) and that allowed for the above configuration improvements, we have seen a markedly improve authentication experience for our users. We generally run about between 29 and 60 connections to AD. Our AD server had the modifications in max-concurrent-api connections as described by this document we found: https://support.microsoft.com/en-us/kb/2688798 In regards to Freeradius, version 3 has the native ntlm_auth calls now (we initially tested some beta code from the original engineer on the 2.x version) and THAT markedly improved performance. In short: 1) Update your controller code, look for the multi source-port queues 2) Update your SAMBA and increase the winbind options to suit your needs 3) Update your Freeradius version and configure it for the integrated winbind stuff (see the mods-available/mschap file for documentation) 4) Increase your maxConcurrentApi settings on your AD server (this might help as well) - John Douglass, Georgia Institute of Technology. On 10/16/2015 11:53 AM, Walter Reynolds wrote:
We are running to the same problems with winbind/samba bottlenecks as many other universities. While one of our servers does quite well we are starting to get hung modules at peak hitting as many as 150 authentications per second while averaging over 70/sec for several minutes.
Since wireless controllers are getting bigger and with Cisco we are limited to sending radius auth request to one server at a time we need another way to balance things out. I thought setting up a freeradius proxy that pointed to two separate servers might be a way to go about this. Problem is I thought I read somewhere in the freeradius lists that that would create other problems, but now I can not find that. I also am not sure I followed this right, but in looking at proxy home_server_pool it looks like my options for load balancing will still send all auth request from my one large controller to a single server as the source IP will all be the same under client-balance.
Thanks. ------------------------ Walter Reynolds Principal Systems Security Development Engineer Information and Technology Services University of Michigan (734) 615-9438 - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
participants (7)
-
A.L.M.Buxey@lboro.ac.uk -
Alan Buxey -
Arran Cudbard-Bell -
John Douglass -
Matthew Newton -
Phil Mayers -
Walter Reynolds