Performance Problems ntlm_auth <-> Active Directory under heavy load

[tom greisch]

Hi together,
Yesterday i have compiled Freeradius 2.2.6 and configured for ntlm_auth against an Active Directory. Everything seems ok. After testing i have redirected our WLAN Controller (with 
thousands of Clients) to the new Radius Server and the Server was overloaded immediately. 
(By the way our old Microsoft NPS Server had no Problem with the Load!). 


So i activated the Logging for Samba an i got this:
----------------------------------------------------
[2015/03/18 16:59:50.848834,  0] ../source3/lib/util.c:896(log_stack_trace)
BACKTRACE: 21 stack frames:
#0 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(log_stack_trace+0x1a) [0x7f60a79bb2ca]
#1 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(smb_panic_s3+0x20) [0x7f60a79bb3b0]
#2 /usr/lib/x86_64-linux-gnu/libsamba-util.so.0(smb_panic+0x2f) [0x7f60abce5caf]
#3 /usr/lib/x86_64-linux-gnu/libsamba-util.so.0(+0x1cecf) [0x7f60abce5ecf]
#4 /lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0) [0x7f60ac1158d0]
#5 /usr/lib/x86_64-linux-gnu/samba/liblibcli_netlogon3.so.0(rpccli_netlogon_sam_network_logon+0x154) [0x7f60a99c52a4]
#6 /usr/sbin/winbindd(+0x48fd3) [0x7f60ac58efd3]
#7 /usr/sbin/winbindd(winbindd_dual_pam_auth_crap+0x2d8) [0x7f60ac592938]
#8 /usr/sbin/winbindd(+0x5fd5c) [0x7f60ac5a5d5c]
#9 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x92cb) [0x7f60a5a162cb]
#10 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x7797) [0x7f60a5a14797]
#11 /usr/lib/x86_64-linux-gnu/libtevent.so.0(_tevent_loop_once+0x8d) [0x7f60a5a10f9d]
#12 /usr/sbin/winbindd(+0x620d8) [0x7f60ac5a80d8]
#13 /usr/sbin/winbindd(+0x627d5) [0x7f60ac5a87d5]
#14 /usr/lib/x86_64-linux-gnu/libtevent.so.0(tevent_common_loop_immediate+0xd4) [0x7f60a5a117c4]
#15 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x908e) [0x7f60a5a1608e]
#16 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x7797) [0x7f60a5a14797]
#17 /usr/lib/x86_64-linux-gnu/libtevent.so.0(_tevent_loop_once+0x8d) [0x7f60a5a10f9d]
#18 /usr/sbin/winbindd(main+0xad4) [0x7f60ac56b024]
#19 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f60a5685b45]
#20 /usr/sbin/winbindd(+0x25699) [0x7f60ac56b699]
[2015/03/18 16:59:50.853758,  0] ../source3/lib/util.c:797(smb_panic_s3)
smb_panic(): calling panic action [/usr/share/samba/panic-action 4754]
[2015/03/18 16:59:50.859870,  0] ../source3/lib/util.c:805(smb_panic_s3)
smb_panic(): action returned status 0
[2015/03/18 16:59:50.860457,  0] ../source3/lib/dumpcore.c:317(dump_core)
dumping core in /var/log/samba/cores/winbindd---------------------------------------------------------------------------------


At this Point its clear, that it is not a Freeradius Problem. Its a Samba Problem!
I am lucky because i found a BUG Report at the Samba Site released onyl few Days ago 
(2015-03-10). The Problem persists for more than one Year.
The BUG is: https://bugzilla.samba.org/show_bug.cgi?id=11149


The Bug Report says that you have to start Freeradius with the "-t" Option to disable Threads because Samba 4.1.X is not able to handle threads. So we have to hope that samba 4.2 will fix the Problem.

Until that, i decided to clone the Radius Server twice , start with "-t" Flag and set a "Freeradius Loadbalancer" bevor the "Worker Radius Server". Now i can handle the Load !


I think more people will have the same Problem as me... So i decided to write here...



System Info:
-----------------------------
Debian 8 (Jessie)
Samba version 4.1.17-Debian
Freeradius Version 2.2.6
-----------------------------