Phil Mayers wrote:
It seems to be related to "peak movement" times - start/end of lectures and so forth - and to a massive spike in auth load as clients roam between APs. The problem is basically that above a certain load, auths suddenly aren't answered quickly enough, and the whole system goes into this spiral of doom where retransmits and reauths dominate.
The issue is that when ntlm_auth is blocked, the entire server melts down. This is the same as when an SQL DB blocks the server, too.
In our case, it was fork/exec of ntlm_auth being slow, and moving to a faster box helped a *lot*.
fork/exec shouldn't be that bad... the # of outstanding ntlm_programs is limited, but only by the total # of threads. So maybe the issues is past fork/exec, and into ntlm_auth / winbind?
In our case, we were seeing the spike on radius and ntlm_auth times, but not MSRPC, which told us it was local. A closer examination of vmstat output strongly suggested kernel load of fork/exec. A faster box with a newer Linux kernel does not have the same issues, even under significantly higher load.
Weird, but OK.
A few tricks needed here to process this data.
To get the ntlm_auth timings, at the moment you'll need a wrapper. I wrote a quick one in C which basically does clock_getime, fork/exec/wait, clock_gettime again the logs the result and returns the child exit status. I'll try to post the code later today.
Maybe the changes in 2.2.3 will help here. If the child takes more than 1 second, you're better off giving up on the request. Alan DeKok.