Performance Problems ntlm_auth <-> Active Directory under heavy load
Hi together, Yesterday i have compiled Freeradius 2.2.6 and configured for ntlm_auth against an Active Directory. Everything seems ok. After testing i have redirected our WLAN Controller (with thousands of Clients) to the new Radius Server and the Server was overloaded immediately. (By the way our old Microsoft NPS Server had no Problem with the Load!). So i activated the Logging for Samba an i got this: ---------------------------------------------------- [2015/03/18 16:59:50.848834, 0] ../source3/lib/util.c:896(log_stack_trace) BACKTRACE: 21 stack frames: #0 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(log_stack_trace+0x1a) [0x7f60a79bb2ca] #1 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(smb_panic_s3+0x20) [0x7f60a79bb3b0] #2 /usr/lib/x86_64-linux-gnu/libsamba-util.so.0(smb_panic+0x2f) [0x7f60abce5caf] #3 /usr/lib/x86_64-linux-gnu/libsamba-util.so.0(+0x1cecf) [0x7f60abce5ecf] #4 /lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0) [0x7f60ac1158d0] #5 /usr/lib/x86_64-linux-gnu/samba/liblibcli_netlogon3.so.0(rpccli_netlogon_sam_network_logon+0x154) [0x7f60a99c52a4] #6 /usr/sbin/winbindd(+0x48fd3) [0x7f60ac58efd3] #7 /usr/sbin/winbindd(winbindd_dual_pam_auth_crap+0x2d8) [0x7f60ac592938] #8 /usr/sbin/winbindd(+0x5fd5c) [0x7f60ac5a5d5c] #9 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x92cb) [0x7f60a5a162cb] #10 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x7797) [0x7f60a5a14797] #11 /usr/lib/x86_64-linux-gnu/libtevent.so.0(_tevent_loop_once+0x8d) [0x7f60a5a10f9d] #12 /usr/sbin/winbindd(+0x620d8) [0x7f60ac5a80d8] #13 /usr/sbin/winbindd(+0x627d5) [0x7f60ac5a87d5] #14 /usr/lib/x86_64-linux-gnu/libtevent.so.0(tevent_common_loop_immediate+0xd4) [0x7f60a5a117c4] #15 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x908e) [0x7f60a5a1608e] #16 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x7797) [0x7f60a5a14797] #17 /usr/lib/x86_64-linux-gnu/libtevent.so.0(_tevent_loop_once+0x8d) [0x7f60a5a10f9d] #18 /usr/sbin/winbindd(main+0xad4) [0x7f60ac56b024] #19 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f60a5685b45] #20 /usr/sbin/winbindd(+0x25699) [0x7f60ac56b699] [2015/03/18 16:59:50.853758, 0] ../source3/lib/util.c:797(smb_panic_s3) smb_panic(): calling panic action [/usr/share/samba/panic-action 4754] [2015/03/18 16:59:50.859870, 0] ../source3/lib/util.c:805(smb_panic_s3) smb_panic(): action returned status 0 [2015/03/18 16:59:50.860457, 0] ../source3/lib/dumpcore.c:317(dump_core) dumping core in /var/log/samba/cores/winbindd--------------------------------------------------------------------------------- At this Point its clear, that it is not a Freeradius Problem. Its a Samba Problem! I am lucky because i found a BUG Report at the Samba Site released onyl few Days ago (2015-03-10). The Problem persists for more than one Year. The BUG is: https://bugzilla.samba.org/show_bug.cgi?id=11149 The Bug Report says that you have to start Freeradius with the "-t" Option to disable Threads because Samba 4.1.X is not able to handle threads. So we have to hope that samba 4.2 will fix the Problem. Until that, i decided to clone the Radius Server twice , start with "-t" Flag and set a "Freeradius Loadbalancer" bevor the "Worker Radius Server". Now i can handle the Load ! I think more people will have the same Problem as me... So i decided to write here... System Info: ----------------------------- Debian 8 (Jessie) Samba version 4.1.17-Debian Freeradius Version 2.2.6 -----------------------------
hi, known about for ages - and we run it (FR) in threaded mode - however there are some new bits of work in 3.x code to enable NTLM auth to work better (either with connection pool method or using ntlm_auth with helper mode) alan
On Thu, Mar 19, 2015 at 09:44:41AM +0000, A.L.M.Buxey@lboro.ac.uk wrote:
known about for ages - and we run it (FR) in threaded mode - however there are some new bits of work in 3.x code to enable NTLM auth to work better (either with connection pool method or using ntlm_auth with helper mode)
Updated patch for Samba has been merged to master, the Samba guys have been great about it and it should hopefully be in Samba 4.2.1. FreeRADIUS 3 patches pretty much done, hoping to do a new pull request later today. Will do a FreeRADIUS 2 patch for those that need it, but not asking for it to be merged. I've dropped the ntlm_auth helper mode stuff, though the patches are still out there in the existing pull request. No point when the code for Samba is in and the libraries are thread-safe. Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
On Mar 19, 2015, at 7:14 , Matthew Newton <mcn4@leicester.ac.uk> wrote:
Updated patch for Samba has been merged to master, the Samba guys have been great about it and it should hopefully be in Samba 4.2.1. FreeRADIUS 3 patches pretty much done, hoping to do a new pull request later today. Will do a FreeRADIUS 2 patch for those that need it, but not asking for it to be merged.
Let me just express my interest in seeing this for FR2. Even if the feature ends up not being merged, I still commend you for the work. Regards, -- Louis Munro lmunro@inverse.ca :: www.inverse.ca +1.514.447.4918 x125 :: +1 (866) 353-6153 x125 Inverse inc. :: Leaders behind SOGo (www.sogo.nu) and PacketFence (www.packetfence.org)
On Mar 26, 2015, at 8:23 AM, Louis Munro <lmunro@inverse.ca> wrote:
Let me just express my interest in seeing this for FR2.
It’s hard. v3 has a connection pool API. v2 doesn’t. Adding the connection pool to the MSCHAP module was ~100 LoC. It would be > 1000 LoC for v2. Alan DeKok.
On 26 Mar 2015, at 09:45, Alan DeKok <aland@deployingradius.com> wrote:
On Mar 26, 2015, at 8:23 AM, Louis Munro <lmunro@inverse.ca> wrote:
Let me just express my interest in seeing this for FR2.
It’s hard. v3 has a connection pool API. v2 doesn’t. Adding the connection pool to the MSCHAP module was ~100 LoC. It would be > 1000 LoC for v2.
I am absolutely against back-porting this feature to version 2. It's an incentive for people to upgrade to v3.0.x. Which is especially useful for Universities where and upgrade might be hard to justify otherwise. Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
On Thu, Mar 26, 2015 at 10:00:45AM -0400, Arran Cudbard-Bell wrote:
On 26 Mar 2015, at 09:45, Alan DeKok <aland@deployingradius.com> wrote:
On Mar 26, 2015, at 8:23 AM, Louis Munro <lmunro@inverse.ca> wrote:
Let me just express my interest in seeing this for FR2.
It’s hard. v3 has a connection pool API. v2 doesn’t. Adding the connection pool to the MSCHAP module was ~100 LoC. It would be > 1000 LoC for v2.
I am absolutely against back-porting this feature to version 2.
It's an incentive for people to upgrade to v3.0.x. Which is especially useful for Universities where and upgrade might be hard to justify otherwise.
Agreed, v3 only it is. I will be using it myself as a reason to push for an upgrade (which is essentially down to me having to rewrite the config... sigh). If it weren't for the connection pool it would be trivial. But as Alan said, that is not trivial to do, and I'm not taking any time on working on old versions. This has taken four months already. :) If anyone really wants it in v2, then older patches that work are out there somewhere. But no support from here if you choose to go that route, I'm afraid. Cheers, Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
On Mar 26, 2015, at 10:00 , Arran Cudbard-Bell <a.cudbardb@freeradius.org> wrote:
I am absolutely against back-porting this feature to version 2.
It's an incentive for people to upgrade to v3.0.x. Which is especially useful for Universities where and upgrade might be hard to justify otherwise.
All right. Wishful thinking on my end then. I guess I need to get serious about FR3. I take it it’s now considered ready for production environments. Regards, -- Louis Munro lmunro@inverse.ca :: www.inverse.ca +1.514.447.4918 x125 :: +1 (866) 353-6153 x125 Inverse inc. :: Leaders behind SOGo (www.sogo.nu) and PacketFence (www.packetfence.org)
On Thu, Mar 19, 2015 at 09:27:12AM +0000, tom greisch wrote:
Yesterday i have compiled Freeradius 2.2.6 and configured for ntlm_auth against an Active Directory. Everything seems ok. After testing i have redirected our WLAN Controller (with thousands of Clients) to the new Radius Server and the Server was overloaded immediately.
We have thousands of clients - but that's not the issue; its the number of authentications per second that you need to watch. Current limit is more than around 30 auths/second and you'll probably hit issues.
So i activated the Logging for Samba an i got this: ---------------------------------------------------- [2015/03/18 16:59:50.848834, 0] ../source3/lib/util.c:896(log_stack_trace) BACKTRACE: 21 stack frames: #0 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(log_stack_trace+0x1a) [0x7f60a79bb2ca] ... smb_panic(): action returned status 0 [2015/03/18 16:59:50.860457, 0] ../source3/lib/dumpcore.c:317(dump_core) dumping core in /var/log/samba/cores/winbindd---------------------------------------------------------------------------------
At this Point its clear, that it is not a Freeradius Problem. Its a Samba Problem!
No, the usual case is a FreeRADIUS problem. At least, in the way people authenticate against Samba, not in the FreeRADIUS code itself. Looks to me like you may have hit a different bug in winbind. Try running ntlm_auth from the command line and see if it gives the same problem.
I am lucky because i found a BUG Report at the Samba Site released onyl few Days ago (2015-03-10). The Problem persists for more than one Year. The BUG is: https://bugzilla.samba.org/show_bug.cgi?id=11149
That is my bug report. It has nothing to do with the ntlm_auth binary.
The Bug Report says that you have to start Freeradius with the "-t" Option to disable Threads because Samba 4.1.X is not able to handle threads. So we have to hope that samba 4.2 will fix the Problem.
No, it doesn't say that, and you shouldn't do that. It will kill performance. Use ntlm_auth if you can't use the new patches (which aren't released yet anyway).
Until that, i decided to clone the Radius Server twice , start with "-t" Flag and set a "Freeradius Loadbalancer" bevor the "Worker Radius Server". Now i can handle the Load !
If you can run three separate FreeRADIUS instances in non-threaded mode, you should easily be able to run one in threaded mode with ntlm_auth. Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
On Mar 19, 2015, at 5:27 AM, tom greisch <tom.greisch@yahoo.de> wrote:
Yesterday i have compiled Freeradius 2.2.6 and configured for ntlm_auth against an Active Directory. Everything seems ok. After testing i have redirected our WLAN Controller (with thousands of Clients) to the new Radius Server and the Server was overloaded immediately. (By the way our old Microsoft NPS Server had no Problem with the Load!).
Yes, because Microsoft NPS uses secret Microsoft APIs which aren’t available to other people.
Until that, i decided to clone the Radius Server twice , start with "-t" Flag and set a "Freeradius Loadbalancer" bevor the "Worker Radius Server". Now i can handle the Load !
That’s the same as setting “max threads = 2”. Alan DeKok.
participants (6)
-
A.L.M.Buxey＠lboro.ac.uk -
Alan DeKok -
Arran Cudbard-Bell -
Louis Munro -
Matthew Newton -
tom greisch