Instrumentation for monitoring ntlm_auths against domain controllers
Hi all, Apologies if this isn't directly on-topic, but there are a lot of people on this mailing list who are familiar with what I want to achieve, and may be able to point me in the right direction. I'm happy to field off-list replies if that's more appropriate. We're running freeradius 3.0.12, samba/winbind 4.2.10 on CentOS 7.2. We're quite cheerfully authenticating our users against AD using the winbind_username/winbind_domain parameters in the mschap module, which is a reasonably common configuration and generally works well for us (and has noticably less overhead than the traditional ntlm_auth mechanism) Yesterday, we saw a spike in authentication latency that was big enough to cause an impact to users on our wireless network, and we're trying to do some root cause analysis. We have a nagios check in place which uses ntlm_auth to do a test authentication against the domain, and that showed a corresponding spike in latency - which points the finger at AD in general, but doesn't give us useful data to pass to our windows team as it's not easy to pin down which domain controller(s) the radius servers were talking to at the time. What I'd like to do, is put some instrumentation in place that would allow our monitoring server to fire ntlm_auth's at a specified domain controller (rather than whichever one winbind happens to have connected to) so that we can monitor latency to all of them, and use the resulting graphs to pinpoint any that are under performing. I can't see an obvious way to make that happen, so if anyone has any pointers we'd really appreciate it! -Paul -- ---------------------------------------------------------------------- Paul Seward, Senior Systems Administrator, University of Bristol Paul.Seward@bristol.ac.uk +44 (0)117 39 41148 GPG Key ID: E24DA8A2 GPG Fingerprint: 7210 4E4A B5FC 7D9C 39F8 5C3C 6759 3937 E24D A8A2
On Wed, Oct 19, 2016 at 12:49:44PM +0100, Paul Seward wrote:
What I'd like to do, is put some instrumentation in place that would allow our monitoring server to fire ntlm_auth's at a specified domain controller (rather than whichever one winbind happens to have connected to) so that we can monitor latency to all of them, and use the resulting graphs to pinpoint any that are under performing.
I can't see an obvious way to make that happen, so if anyone has any pointers we'd really appreciate it!
I don't remember seeing anything like that when working on that code. You can tell winbind which DC to talk to, but that's not going to help much. The auth functions don't let you know which DC was used. Bumping up winbind logging level might give you something, but they get very verbose so you'd probably want to write to ramdisk so save IO and then scrape info from there. But on log levels 9 or 10 or so it gets pretty detailed. You might be able to find which DC is being used, and an indicative latency, by writing a small program to call wbcPingDc2 and note the time taken to respond and which DC was used. Log this once a minute might help, as I don't think winbind generally moves to a different DC without good reason (and the old versions were notoriously bad at moving to a different DC even when it needed to), so may give some good telemetry. Matthew -- Matthew Newton, Ph.D. <mcn4@leicester.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
On Wed, Oct 19, 2016 at 01:00:34PM +0100, Matthew Newton wrote:
On Wed, Oct 19, 2016 at 12:49:44PM +0100, Paul Seward wrote:
What I'd like to do, is put some instrumentation in place that would allow our monitoring server to fire ntlm_auth's at a specified domain controller (rather than whichever one winbind happens to have connected to) so that we can monitor latency to all of them, and use the resulting graphs to pinpoint any that are under performing.
I can't see an obvious way to make that happen, so if anyone has any pointers we'd really appreciate it!
You might be able to find which DC is being used, and an indicative latency, by writing a small program to call wbcPingDc2 and note the time taken to respond and which DC was used. Log this
Except here I get # ./dclatency error: unable to ping dc WBC_ERR_NOT_IMPLEMENTED # :( Look through wbclient.h. There may be another way. Matthew -- Matthew Newton, Ph.D. <mcn4@leicester.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
Thanks for the pointers, you've confirmed my suspicion that there's not an easy answer! I'll keep exploring, and if we end up writing any tools that would help we'll obviously make them more widely available. -Paul On 19 October 2016 at 13:22, Matthew Newton <mcn4@leicester.ac.uk> wrote:
On Wed, Oct 19, 2016 at 01:00:34PM +0100, Matthew Newton wrote:
On Wed, Oct 19, 2016 at 12:49:44PM +0100, Paul Seward wrote:
What I'd like to do, is put some instrumentation in place that would allow our monitoring server to fire ntlm_auth's at a specified domain controller (rather than whichever one winbind happens to have connected to) so that we can monitor latency to all of them, and use the resulting graphs to pinpoint any that are under performing.
I can't see an obvious way to make that happen, so if anyone has any pointers we'd really appreciate it!
You might be able to find which DC is being used, and an indicative latency, by writing a small program to call wbcPingDc2 and note the time taken to respond and which DC was used. Log this
Except here I get
# ./dclatency error: unable to ping dc WBC_ERR_NOT_IMPLEMENTED #
:(
Look through wbclient.h. There may be another way.
Matthew
-- Matthew Newton, Ph.D. <mcn4@leicester.ac.uk>
Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom
For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
-- ---------------------------------------------------------------------- Paul Seward, Senior Systems Administrator, University of Bristol Paul.Seward@bristol.ac.uk +44 (0)117 39 41148 GPG Key ID: E24DA8A2 GPG Fingerprint: 7210 4E4A B5FC 7D9C 39F8 5C3C 6759 3937 E24D A8A2
On Oct 19, 2016, at 7:49 AM, Paul Seward <Paul.Seward@bristol.ac.uk> wrote:
What I'd like to do, is put some instrumentation in place that would allow our monitoring server to fire ntlm_auth's at a specified domain controller (rather than whichever one winbind happens to have connected to) so that we can monitor latency to all of them, and use the resulting graphs to pinpoint any that are under performing.
That's really an issue for the Samba people. We could track latency in the server. It's not done for 3.0, but it's on the plan for 4.0. Alan DeKok.
participants (3)
-
Alan DeKok -
Matthew Newton -
Paul Seward