recommendations for max_servers
Hello, While troubleshooting a system I came upon a case of 'hung children' and duplicate requests. I would usually ascribe this to a database issue but in this case the database is mostly unused and properly indexed. Accounting is not used, so that's one less thing to consider. On the other hand the max_servers setting had been set as high as 192 by someone with good intentions. Tuning it down to 64 seemed to significantly reduce the load on the system and the number of hung children was reduced by a factor of about 100. While there remains an issue with some (intermittent) slow ntlm_auth to take care of, I wondered how others tune the value of max_servers other than by trial and error. Most of the time the default of 32 has been enough for me. Higher is not necessarily better in my experience since at least in this case it seems to have led to the main thread working harder when under load (with most of the work done in the "system" space). This is a system running 2.2.5 on RHEL 6.4 in VmWare. It's got 24Gb of RAM and 16 cores so it should still be pretty capable. Does anyone have an algorithm, rule of thumb or other ballpark way of estimating the "ideal" maximum number of threads? Regards, -- Louis Munro lmunro@inverse.ca :: www.inverse.ca +1.514.447.4918 x125 :: +1 (866) 353-6153 x125 Inverse inc. :: Leaders behind SOGo (www.sogo.nu) and PacketFence (www.packetfence.org)
Have 64 here. There are historical reasons eg vmps where 48 requests come as one big bang. But we found the sweet spot was 64 (there have also been historical values that worked or not eg back in the 2.0.x days mysql connections didn't like to be above 10). Still working on 3.x values eg spare and min values alan -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Louis Munro wrote:
While troubleshooting a system I came upon a case of 'hung children' and duplicate requests.
So... something is blocking the server. Find out what that is, and fix it.
On the other hand the max_servers setting had been set as high as 192 by someone with good intentions. Tuning it down to 64 seemed to significantly reduce the load on the system and the number of hung children was reduced by a factor of about 100.
Adding more threads results in more contention, and more problems. The child threads get hung because they're blocked. Adding more child threads means more blocking. Playing games with max_servers won't help. The *only* solution is to find the root cause of the problem, and then fix it.
While there remains an issue with some (intermittent) slow ntlm_auth to take care of, I wondered how others tune the value of max_servers other than by trial and error. Most of the time the default of 32 has been enough for me. Higher is not necessarily better in my experience since at least in this case it seems to have led to the main thread working harder when under load (with most of the work done in the "system" space).
If the threads aren't blocked, you can set max_servers to 10000, and it won't cause a problem.
This is a system running 2.2.5 on RHEL 6.4 in VmWare. It's got 24Gb of RAM and 16 cores so it should still be pretty capable.
Except for the fact that it spends most of it's time in kernel space. So the RADIUS process isn't really doing much.
Does anyone have an algorithm, rule of thumb or other ballpark way of estimating the "ideal" maximum number of threads?
Fix the blocking issue. Then, set max_servers to some huge number. Alan DeKok.
Loius, Be aware, there are some major design flaws within the Cisco WLC controller software (Georgia Tech is working with Cisco to work through them) regarding the number of requests a controller can field. https://tools.cisco.com/bugsearch/bug/CSCuj88508 The flaws in the controller software cause an "overrun" of radiusIDs if you have too many authentications/second which will manifest as "duplicate" and "discards" in the logs. No amount of tweaking on the radius side will fix this. You can however, improve performance to try and improve the client experience. When we are talking about AD, Phil Mayers had some great suggestions on improving ntlm_auth performance. Here were his recommendations: 1. Upgraded the radius servers. Old spec: 3Gb RAM, 2x P4-based Xeon 1 core @ 3.2GHz, RHEL5 New spec: 16Gb RAM, 1x Xeon E5-2620 6 core @ 2GHz, RHEL6 2. Upgraded Samba - went from RHEL5 samba3x-3.5.4 to RHEL6 samba-3.6.9 3. Set "winbind max domain connections = 12" in smb.conf (restart winbind) (we at GT actually have so many authentications, we set to 128 as we reached our limit during peak times) 4. Forced our smb.conf to talk to specific AD controllers which are physical, not VMWare (most our DCs are VMWare) 5. Spent a *lot* of time debugging and tracking the Samba->DC RPC round-trip times and hassling our AD people to keep these stable; not sure what they did, if anything. 6. Increased radiusd.conf setting to "max_requests = 16384" 7. Worked really, really hard on getting the Cisco APs, AP radios and controllers to STOP CRASHING; their software quality has been abysmal, and this was a contributing factor - APs or controllers would crash under load, and this would trigger a burst of auths, which would trigger the problem. As Alan said before, there are lots of moving parts where issues can happen. If you improve server performance within the pieces (AD/database/winbind/etc), that's a start. If you are in a large scale Cisco deployment, depending on how many APs and users, you may find yourself having issues regardless. It's a hard problem to advise on, but adding additional radius servers and optimizing ours for performance has helped us immensely. - JohnD On 09/19/2014 02:58 PM, Louis Munro wrote:
Hello,
While troubleshooting a system I came upon a case of 'hung children' and duplicate requests. I would usually ascribe this to a database issue but in this case the database is mostly unused and properly indexed. Accounting is not used, so that's one less thing to consider.
On the other hand the max_servers setting had been set as high as 192 by someone with good intentions. Tuning it down to 64 seemed to significantly reduce the load on the system and the number of hung children was reduced by a factor of about 100.
While there remains an issue with some (intermittent) slow ntlm_auth to take care of, I wondered how others tune the value of max_servers other than by trial and error. Most of the time the default of 32 has been enough for me. Higher is not necessarily better in my experience since at least in this case it seems to have led to the main thread working harder when under load (with most of the work done in the "system" space).
This is a system running 2.2.5 on RHEL 6.4 in VmWare. It's got 24Gb of RAM and 16 cores so it should still be pretty capable.
Does anyone have an algorithm, rule of thumb or other ballpark way of estimating the "ideal" maximum number of threads?
Regards, -- Louis Munro lmunro@inverse.ca <mailto:lmunro@inverse.ca> :: www.inverse.ca <http://www.inverse.ca> +1.514.447.4918 x125 :: +1 (866) 353-6153 x125 Inverse inc. :: Leaders behind SOGo (www.sogo.nu <http://www.sogo.nu>) and PacketFence (www.packetfence.org <http://www.packetfence.org>)
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Good luck.We worked through a list. Some stuff came out of it but not all. I did highlight the problem years ago to them when I noted that the nas-port-ID was the same for all requests. So....a minor update to their code meant that the accounting now has its own id. Yeah. 255 more slots. What they really need, at least, is at least a seperate id per AP on the controller. I just hope that they don't introduce any 'solutions' that felt on their propriety ISE platform! :( We spent far too much of our time last year tweaking server/OS/radius/scripts just to get around this load issue. But hey. I guess that is one of the big disadvantages when you have 500-1000 APs all going through one single controller :/ Alan -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
John Douglass wrote:
The flaws in the controller software cause an "overrun" of radiusIDs if you have too many authentications/second which will manifest as "duplicate" and "discards" in the logs. No amount of tweaking on the radius side will fix this. You can however, improve performance to try and improve the client experience.
That should sometimes be tolerable... if the RADIUS server is fast enough. But when you tie FreeRADIUS to Active Directory, performance drops by a factor of 100 or more. I've done 40K authentications per second with a simple FreeRADIUS configuration, on commodity hardware. Using AD... is a lot slower. Personally, I advise people to avoid Active Directory if at all possible. It's just not set up for enterprise authentication. Alan DeKok.
We're migrating to EAP-TLS this year. Hopefully less than 50% of our auths will be PEAP via AD by December alan -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Thank you all for your replies. I detail some of the changes we have made below. On 2014-09-23, at 13:31 , John Douglass <john.douglass@oit.gatech.edu> wrote:
When we are talking about AD, Phil Mayers had some great suggestions on improving ntlm_auth performance. Here were his recommendations:
1. Upgraded the radius servers. Old spec: 3Gb RAM, 2x P4-based Xeon 1 core @ 3.2GHz, RHEL5 New spec: 16Gb RAM, 1x Xeon E5-2620 6 core @ 2GHz, RHEL6
It's running on a VMWare host in an ESX cluster at the moment. We moved all other VMs off the host. It's provisioned with 24 Gb of RAM and 12 cores.
2. Upgraded Samba - went from RHEL5 samba3x-3.5.4 to RHEL6 samba-3.6.9
Upgraded yesterday to the latest available from redhat. That would be 3.6.9-169 at the time.
3. Set "winbind max domain connections = 12" in smb.conf (restart winbind) (we at GT actually have so many authentications, we set to 128 as we reached our limit during peak times)
We've had it running at 64 for a while. We had to tune the AD for it to accept this many connections. We based our DC settings on the advice of this article: http://support.microsoft.com/kb/2688798
4. Forced our smb.conf to talk to specific AD controllers which are physical, not VMWare (most our DCs are VMWare)
Can you explain how you forced it to choose those DCs? I can't seem to get winbind to send requests to a specific DC. It's got a mind of its own.
5. Spent a *lot* of time debugging and tracking the Samba->DC RPC round-trip times and hassling our AD people to keep these stable; not sure what they did, if anything.
I actually wrote a wrapper (in C) around ntlm_auth to log the times between calling ntlm_auth and it returning a value. This is where I found values that vary wildly between 7ms and <= 3000ms (because FR has ntlm_auth_timeout = 3). We later patched winbind to log the time between sending the requests to the DC and getting a reply. Those timings are actually consistently fast now and yet the problem persists in FR.
6. Increased radiusd.conf setting to "max_requests = 16384"
I set it to 20000 long ago.
7. Worked really, really hard on getting the Cisco APs, AP radios and controllers to STOP CRASHING; their software quality has been abysmal, and this was a contributing factor - APs or controllers would crash under load, and this would trigger a burst of auths, which would trigger the problem.
This part is out of my hands, but I will certainly pass you advice along...
As Alan said before, there are lots of moving parts where issues can happen. If you improve server performance within the pieces (AD/database/winbind/etc), that's a start.
It's pretty clear to me it's not the database. I log slow queries, check processlist obsessively and it's mostly unused.
If you are in a large scale Cisco deployment, depending on how many APs and users, you may find yourself having issues regardless. It's a hard problem to advise on, but adding additional radius servers and optimizing ours for performance has helped us immensely.
If anything, this will make me learn more about network programming. I have taken some stack traces using gdb when the system is under load as well as stracing the process. I can provide those if anyone is interested. I see most threads just doing a sem_wait while Thread 1 is doing all the work. This would be easier of course if I had consistently bad performance. As it is, things only fall apart when a significant load is reached. There, I just got another flurry of these while replying: Info: Child PID 26929 (/usr/bin/ntlm_auth) is taking too much time: forcing failure and killing child. Regards, -- Louis Munro lmunro@inverse.ca :: www.inverse.ca +1.514.447.4918 x125 :: +1 (866) 353-6153 x125 Inverse inc. :: Leaders behind SOGo (www.sogo.nu) and PacketFence (www.packetfence.org)
Louis Munro wrote:
I actually wrote a wrapper (in C) around ntlm_auth to log the times between calling ntlm_auth and it returning a value. This is where I found values that vary wildly between 7ms and <= 3000ms
That's a problem. If the user lookup is taking 3 seconds, of COURSE FreeRADIUS will fall over. Try attacking a 10,000 pound weight to your car, and see how fast it goes. You wouldn't blame the car for not being able to move, right?
I see most threads just doing a sem_wait while Thread 1 is doing all the work.
Most of the threads will be waiting for ntlm_auth to return. Thread 1 will be handling all of the network IO, and doing real work.
This would be easier of course if I had consistently bad performance. As it is, things only fall apart when a significant load is reached.
That's ALWAYS what happens when a system is overloaded. The system falls over. That's what "overload" means.
There, I just got another flurry of these while replying: Info: Child PID 26929 (/usr/bin/ntlm_auth) is taking too much time: forcing failure and killing child.
Yes. Your ntlm_auth process is taking too long, and is destroying FreeRADIUS. I can't be any clearer about that. No amount of poking FreeRADIUS will make ntlm_auth run faster. And making ntlm_auth run faster is the ONLY solution to the problem. I've said it before, and I'll say it again. Using Active Directory is a terrible decision for almost everyone. It's slow, awkward, unstable, etc. FreeRADIUS can do 50K+ authentications per second from the "users" file. So there is no way the problem is caused by FreeRADIUS. Alan DeKok.
All, As Alan said re: AD authentication:
Yes. Your ntlm_auth process is taking too long, and is destroying FreeRADIUS. I can't be any clearer about that.
No amount of poking FreeRADIUS will make ntlm_auth run faster. And making ntlm_auth run faster is the ONLY solution to the problem.
I've said it before, and I'll say it again. Using Active Directory is a terrible decision for almost everyone. It's slow, awkward, unstable, etc.
FreeRADIUS can do 50K+ authentications per second from the "users" file. So there is no way the problem is caused by FreeRADIUS.
I agree, EAP-PEAP-MSCHAPv2 isn't great. But for 100% of our WPA-Enterprise clients, they all support EAP-PEAP-MSCHAPv2 natively. With central authentication (passwords) for 30k users, having an additional password or an additional password store is not ideal. We do not want to have to go to digital certificates just yet as there is a good bit of support overhead and management we just cannot provide at this time. If we agree that AD (EAP-PEAP-MSCHAPv2) is far from ideal, what other EAP types (outside of EAP-PEAP-TLS) do you recommend for end user authentication that is supported by native Windows, iOS, Android, and OSX clients? I imagine if there were a better option with similar properties and ties to central authentication, then we would all flock to it. - John Douglass, Sr. Systems Engineer
We do not want to have to go to digital certificates just yet as there is a good bit of support overhead and management we just cannot provide at this time.
I would say 90% of that is perceived overhead, not actual. Certificate deployment is pretty easy once you have the infrastructure. Even since XP days it's been just a few clicks to install personal certs.
If we agree that AD (EAP-PEAP-MSCHAPv2) is far from ideal, what other EAP types (outside of EAP-PEAP-TLS) do you recommend for end user authentication that is supported by native Windows, iOS, Android, and OSX clients?
There is none. You have PEAP and TLS, none of the other ubiquitously bundled ones provide the keying material required for WPA2-Enterprise. Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
Hi,
There is none. You have PEAP and TLS, none of the other ubiquitously bundled ones provide the keying material required for WPA2-Enterprise.
well, if talking about only current clients then EAP-TTLS/MSCHAPv2 - but you'd probably still use the AD :/ the others that are really nice and fast (EAP-PWD - not even any certificate to fuss around with!!, and EAP-FASTv2) just dont have much presence in clients ...and have very little support in the backend due to how they operate alan
Hi,
I imagine if there were a better option with similar properties and ties to central authentication, then we would all flock to it.
yep. we would. the best is stil EAP-TLS - as you say, thats got big setup overheads... not as nice and easy as PEAP for quick get up and going. however, there are other ways of doing this you COULD proxy the request to a farm of NPS servers - they talk native LSA (none of the nasty samba stuff) - the hit then is purely on the NPS box...but that could blow up if too many requests are in proxy-land you could investigate using SAMBA 4 locally on the FreeRADIUS box - with a local AD clone you dont have the nasty inter-process rubbish....ntlm_auth etc is all local you could investigate using SAMBA 4 tools locally - ntlm_auth has the ability (latest versions) to use all servers in the AD farm rather than just sticking to the current one its chosen (and overloading it!) alan
John Douglass wrote:
I agree, EAP-PEAP-MSCHAPv2 isn't great. But for 100% of our WPA-Enterprise clients, they all support EAP-PEAP-MSCHAPv2 natively. With central authentication (passwords) for 30k users, having an additional password or an additional password store is not ideal.
Having a password store that *works* is useful.
If we agree that AD (EAP-PEAP-MSCHAPv2) is far from ideal, what other EAP types (outside of EAP-PEAP-TLS) do you recommend for end user authentication that is supported by native Windows, iOS, Android, and OSX clients?
The issue isn't really the EAP type. It's that Active Directory is being mean to you. Most modern systems support TTLS. That will work better. But XP doesn't support it.
I imagine if there were a better option with similar properties and ties to central authentication, then we would all flock to it.
If there was a system to pull passwords out of Active Directory, you could set up a "cron" job once a day to do that. Then for authentication, do: if (ldap says password was reset today) { do ntlm_auth } else { use the cached passwords } For the few users who change their passwords, ntlm_auth is fine. For everyone else, they use a real database. It would be MUCH faster, and a LOT more stable than native Active Directory. Alan DeKok.
participants (6)
-
A.L.M.Buxey＠lboro.ac.uk -
Alan Buxey
-
Alan DeKok -
Arran Cudbard-Bell -
John Douglass -
Louis Munro