Re: Generating timing stats for ntlm_auth
As reported to Alan (at eduroam-uk-support), we are also seeing load problems at peak times with similar/identical error messages as those reported in this thread. We are running FreeRadius 2.2.0 on two CentOS 6 VMs, with Kerberos 1.8.2-3.el6 & Samba and Winbind 3.5.6-86.el6. Is there anything we can do here at UWE to help identify the general problem? Regards Martin. -----Original Message----- From: freeradius-users-bounces+martin.ubank=uwe.ac.uk@lists.freeradius.org [mailto:freeradius-users-bounces+martin.ubank=uwe.ac.uk@lists.freeradius.org] On Behalf Of Jonathan Gazeley Sent: 10 October 2013 23:07 To: freeradius-users@lists.freeradius.org Subject: Re: Generating timing stats for ntlm_auth Can confirm that we at Bristol (Cisco wireless, MS AD auth backend) are also seeing load problems at peak times (every hour, at lecture change-over time when approximately one billion iPhones start roaming the campus). We're also not seeing as much session resumption as we'd expect. We're also seeing the same messages as reported in this thread. Will be watching this thread with interest - happy to test patches etc. Cheers, Jonathan - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On 15/10/13 11:44, Martin Ubank wrote:
As reported to Alan (at eduroam-uk-support), we are also seeing load problems at peak times with similar/identical error messages as those reported in this thread. We are running FreeRadius 2.2.0 on two CentOS 6 VMs, with Kerberos 1.8.2-3.el6 & Samba and Winbind 3.5.6-86.el6. Is there anything we can do here at UWE to help identify the general problem? Regards
What type of wireless system are you running, and do the messages / load spikes occur at a certain time i.e. people leaving lectures? What OS version are your upstream windows DCs? I don't think there's any fault with FreeRADIUS here, so perhaps we should move this over to JANET-ROAMING?
Thanks for your reply Phil. We use Alcatel Lucent 6000 and 4750 controllers at UWE (rebadged Aruba). Our Windows DCs are OS versions 5.2 (3790) SP2 (Win 2003) and 6.1 (7601) SP1 (Win 2008 R2 Standard). We see spikes on (or just after) the hour when students leave their lectures; the largest peak is at 1pm. Hope this helps. Let me know if I can provide any further information. Martin. -----Original Message----- From: freeradius-users-bounces+martin.ubank=uwe.ac.uk@lists.freeradius.org [mailto:freeradius-users-bounces+martin.ubank=uwe.ac.uk@lists.freeradius.org] On Behalf Of Phil Mayers Sent: 15 October 2013 12:13 To: freeradius-users@lists.freeradius.org Subject: Re: Generating timing stats for ntlm_auth On 15/10/13 11:44, Martin Ubank wrote:
As reported to Alan (at eduroam-uk-support), we are also seeing load problems at peak times with similar/identical error messages as those reported in this thread. We are running FreeRadius 2.2.0 on two CentOS 6 VMs, with Kerberos 1.8.2-3.el6 & Samba and Winbind 3.5.6-86.el6. Is there anything we can do here at UWE to help identify the general problem? Regards
What type of wireless system are you running, and do the messages / load spikes occur at a certain time i.e. people leaving lectures? What OS version are your upstream windows DCs? I don't think there's any fault with FreeRADIUS here, so perhaps we should move this over to JANET-ROAMING? - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Thanks for your reply Phil. We use Alcatel Lucent 6000 and 4750 controllers at UWE (rebadged Aruba). Our Windows DCs are OS versions 5.2 (3790) SP2 (Win 2003) and 6.1 (7601) SP1 (Win 2008 R2 Standard). We see spikes on (or just after) the hour when students leave their lectures; the largest peak is at 1pm. Hope this helps. Let me know if I can provide any further information. Martin. -----Original Message----- From: freeradius-users-bounces+martin.ubank=uwe.ac.uk@lists.freeradius.org [mailto:freeradius-users-bounces+martin.ubank=uwe.ac.uk@lists.freeradius.org] On Behalf Of Phil Mayers Sent: 15 October 2013 12:13 To: freeradius-users@lists.freeradius.org Subject: Re: Generating timing stats for ntlm_auth On 15/10/13 11:44, Martin Ubank wrote:
As reported to Alan (at eduroam-uk-support), we are also seeing load problems at peak times with similar/identical error messages as those reported in this thread. We are running FreeRadius 2.2.0 on two CentOS 6 VMs, with Kerberos 1.8.2-3.el6 & Samba and Winbind 3.5.6-86.el6. Is there anything we can do here at UWE to help identify the general problem? Regards
What type of wireless system are you running, and do the messages / load spikes occur at a certain time i.e. people leaving lectures? What OS version are your upstream windows DCs? I don't think there's any fault with FreeRADIUS here, so perhaps we should move this over to JANET-ROAMING? - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Martin Ubank wrote:
As reported to Alan (at eduroam-uk-support), we are also seeing load problems at peak times with similar/identical error messages as those reported in this thread. We are running FreeRadius 2.2.0 on two CentOS 6 VMs, with Kerberos 1.8.2-3.el6 & Samba and Winbind 3.5.6-86.el6.
Hmm... it could be an ntlm_auth issue. Maybe moving to pipes as Phil suggested would help. In any case, it's not new in 2.2.1. So I think it's time to release 2.2.2.
Is there anything we can do here at UWE to help identify the general problem?
Set the exec timeout to 1s. It should take less than a second to run ntlm_auth. If it takes more than a second, something is wrong. And you're probably better off dropping the connection than waiting 30s for a timeout. Alan DeKok.
On Tue, Oct 15, 2013 at 07:54:12AM -0400, Alan DeKok wrote:
In any case, it's not new in 2.2.1. So I think it's time to release 2.2.2.
Will you include Phil's patch[0] to fix return code testing in 'if'? https://github.com/philmayers/freeradius-server/commit/51c43419 Even if it's going to change for v3, it probably shouldn't break existing configs in v2. Cheers Matthew [0] https://lists.freeradius.org/pipermail/freeradius-users/2013-October/068640.... -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
Matthew Newton <mcn4@leicester.ac.uk> wrote:
On Tue, Oct 15, 2013 at 07:54:12AM -0400, Alan DeKok wrote:
In any case, it's not new in 2.2.1. So I think it's time to release 2.2.2.
Will you include Phil's patch[0] to fix return code testing in 'if'?
https://github.com/philmayers/freeradius-server/commit/51c43419
Even if it's going to change for v3, it probably shouldn't break existing configs in v2.
Cheers
Matthew
[0] https://lists.freeradius.org/pipermail/freeradius-users/2013-October/068640....
-- Matthew Newton, Ph.D. <mcn4@le.ac.uk>
Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom
For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk> - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
For avoidance of doubt I think changing this behaviour would be a problem. In particular it would make certain styles of policy {} block next-to impossible to write. -- Sent from my phone with, please excuse brevity and typos
On 16 Oct 2013, at 09:32, Phil Mayers <p.mayers@IMPERIAL.AC.UK> wrote:
Matthew Newton <mcn4@leicester.ac.uk> wrote: On Tue, Oct 15, 2013 at 07:54:12AM -0400, Alan DeKok wrote: In any case, it's not new in 2.2.1. So I think it's time to release 2.2.2.
Will you include Phil's patch[0] to fix return code testing in 'if'?
https://github.com/philmayers/freeradius-server/commit/51c43419
Even if it's going to change for v3, it probably shouldn't break existing configs in v2.
Agreed the behaviour for 2.x.x shouldn't change, however if (<rcode>) { should of always checked the current rcode for the section, not the previous rcode returned by the module. There should be a separate paircmp function to allow the last module return code to be checked. There are legitimate use cases for both. Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS Development Team
Shrug. I would argue there should be a separate paircmp for the highest prio code, and the if() left alone, but now that you have released 3.0 in that mode I don't expect I will get my way. Also - the magic paircmp virtual attribute stuff is just a pain in the ass. They never behave quite right - Foreach-Variable is full of surprises (like being an invalid lhs for regexp unless quoted, but failing silently). So if you must do this, please make it a *short to type* and *real* control: attribute - control:RC perhaps? -- Sent from my phone with, please excuse brevity and typos
Phil Mayers wrote:
Shrug. I would argue there should be a separate paircmp for the highest prio code, and the if() left alone, but now that you have released 3.0 in that mode I don't expect I will get my way.
I'll have to think about it before changing the behavior. The issue is that the old code had a series of bugs which were increasingly hard to fix. The last one was essentially impossible, as the code had devolved into nonsense. The re-write makes the code work the way it's documented to work. There were a few bugs, but they were easy to fix.
Also - the magic paircmp virtual attribute stuff is just a pain in the ass. They never behave quite right - Foreach-Variable is full of surprises (like being an invalid lhs for regexp unless quoted, but failing silently).
That can, and should be fixed.
So if you must do this, please make it a *short to type* and *real* control: attribute - control:RC perhaps?
I'll take a look. Alan DeKok.
On Wed, Oct 16, 2013 at 10:42:57AM -0400, Alan DeKok wrote:
So if you must do this, please make it a *short to type* and *real* control: attribute - control:RC perhaps?
As a suggestion, it might be better to introduce new unambiguous syntax for both cases, and then set the existing syntax to the same as v2 (for least surprises) and deprecate it? "if (noop) {..}" has always confused me as to what status it's actually checking... How about if (rc:module == noop) { ... } to check the latest module return code, and if (rc:group == reject) { ... } to check the current code in the group? Alternatives could be "return:module" or "rc:section" or even "rc:last" and "rc:return" for the last module that set it, and the current value that will be returned from the section. Using control attributes might be better as Phil said to avoid the virtual attribute stuff - just control:RC-Module or control:RC-Group would make it unabmiguous. Cheers, Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
Matthew Newton wrote:=
As a suggestion, it might be better to introduce new unambiguous syntax for both cases, and then set the existing syntax to the same as v2 (for least surprises) and deprecate it? "if (noop) {..}" has always confused me as to what status it's actually checking...
Yeah. New syntax is always better.
How about
if (rc:module == noop) { ... }
to check the latest module return code, and
if (rc:group == reject) { ... }
"rc" is a little unclear.
to check the current code in the group? Alternatives could be "return:module" or "rc:section" or even "rc:last" and "rc:return" for the last module that set it, and the current value that will be returned from the section.
Maybe.
Using control attributes might be better as Phil said to avoid the virtual attribute stuff - just control:RC-Module or control:RC-Group would make it unabmiguous.
I just want to avoid updating / creating a new attribute for every module call. It's expensive and annoying. Alan DeKok.
On 16 Oct 2013, at 17:05, Alan DeKok <aland@deployingradius.com> wrote:
Matthew Newton wrote:=
As a suggestion, it might be better to introduce new unambiguous syntax for both cases, and then set the existing syntax to the same as v2 (for least surprises) and deprecate it? "if (noop) {..}" has always confused me as to what status it's actually checking...
Yeah. New syntax is always better.
How about
if (rc:module == noop) { ... }
to check the latest module return code, and
if (rc:group == reject) { ... }
"rc" is a little unclear.
It's also not a list.
to check the current code in the group? Alternatives could be "return:module" or "rc:section" or even "rc:last" and "rc:return" for the last module that set it, and the current value that will be returned from the section.
Maybe.
I strongly disagree with this. We should not invent new syntax just for this case. It should be a paircmp or an xlat.
Using control attributes might be better as Phil said to avoid the virtual attribute stuff - just control:RC-Module or control:RC-Group would make it unabmiguous.
I just want to avoid updating / creating a new attribute for every module call. It's expensive and annoying.
Agreed. Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS Development Team
On Wed, Oct 16, 2013 at 05:19:14PM +0100, Arran Cudbard-Bell wrote:
On 16 Oct 2013, at 17:05, Alan DeKok <aland@deployingradius.com> wrote:
"rc" is a little unclear.
It's also not a list.
Yeah, whatever - it was the first thing that came to mind as an example :-) [insert random discussion about how best to do it here ->]
to check the current code in the group? Alternatives could be "return:module" or "rc:section" or even "rc:last" and "rc:return" for the last module that set it, and the current value that will be returned from the section.
Maybe.
I strongly disagree with this. We should not invent new syntax just for this case. It should be a paircmp or an xlat.
It's already a different syntax in the current system. But an xlat does sound like a cleaner way to do it. (As long as the ambiguity is gone.) Cheers, Matthew -- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
On 16 Oct 2013, at 17:38, Matthew Newton <mcn4@leicester.ac.uk> wrote:
On Wed, Oct 16, 2013 at 05:19:14PM +0100, Arran Cudbard-Bell wrote:
On 16 Oct 2013, at 17:05, Alan DeKok <aland@deployingradius.com> wrote:
"rc" is a little unclear.
It's also not a list.
Yeah, whatever - it was the first thing that came to mind as an example :-)
[insert random discussion about how best to do it here ->]
No more magic, no more magic.
to check the current code in the group? Alternatives could be "return:module" or "rc:section" or even "rc:last" and "rc:return" for the last module that set it, and the current value that will be returned from the section.
Maybe.
I strongly disagree with this. We should not invent new syntax just for this case. It should be a paircmp or an xlat.
It's already a different syntax in the current system.
Yes. I know. Apparently absolute consistency is a bad thing. Mmm.
But an xlat does sound like a cleaner way to do it. (As long as the ambiguity is gone.)
Well it's a bit icky. paircmp is more efficient in some cases, that's why I suggested both. Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS Development Team
Matthew Newton wrote:
On Tue, Oct 15, 2013 at 07:54:12AM -0400, Alan DeKok wrote:
In any case, it's not new in 2.2.1. So I think it's time to release 2.2.2.
Will you include Phil's patch[0] to fix return code testing in 'if'?
I'll have to think about it. The new code is arguably correct, in that it acts as documented. See "man unlang", and the warnings about using the return codes.
https://github.com/philmayers/freeradius-server/commit/51c43419
Even if it's going to change for v3, it probably shouldn't break existing configs in v2.
That's true. Alan DeKok.
Hi,
In any case, it's not new in 2.2.1. So I think it's time to release 2.2.2.
just got latest 2.x.x HEAD and radiusd dies with this Tue Oct 15 12:59:45 2013 : Error: ASSERT FAILED rlm_eap.c[369]: request->proxy_reply == NULL (this was the second time running it..the first time it just went away with no Error msg) alan
A.L.M.Buxey@lboro.ac.uk wrote:
just got latest 2.x.x HEAD and radiusd dies with this
Tue Oct 15 12:59:45 2013 : Error: ASSERT FAILED rlm_eap.c[369]: request->proxy_reply == NULL
Uh... the *authenticate* section has a proxy reply filled in? Just what the heck are you doing with it? That should never happen, which is why it's an assertion. It means that instead of running post-proxy, it's re-running authenticate. Or, a proxy reply is being allocated before authentication, which should never happen. Alan DeKok.
On 10/15/2013 09:10 AM, A.L.M.Buxey@lboro.ac.uk wrote:
Hi,
In any case, it's not new in 2.2.1. So I think it's time to release 2.2.2. just got latest 2.x.x HEAD and radiusd dies with this
Tue Oct 15 12:59:45 2013 : Error: ASSERT FAILED rlm_eap.c[369]: request->proxy_reply == NULL
(this was the second time running it..the first time it just went away with no Error msg)
alan - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Earlier messages I posted to the list sound similar: http://www.mail-archive.com/freeradius-users@lists.freeradius.org/msg84313.h... and http://www.mail-archive.com/freeradius-users@lists.freeradius.org/msg84453.h... But this thread is DEFINITELY what we are experiencing here at Georgia Tech. I was considering moving from using "ntlm_auth" to enabling radius on the AD server and just proxying the auth through radius (getting rid of samba/ntlm_auth altogether) and adding any attributes for my VLAN assignment in the post-auth but other threads on this list indicate there might be an issue with servers that proxy a lot (which has some forward movement to fix soon I believe). With a proxy configuration in test, this appears to work. Unsure if it will improve our issues with load? when we are seeing: Oct 12 06:54:54 newdvlanb radiusd[21299]: WARNING: Child is hung for request 9395584 in component authenticate module peap. Oct 12 06:54:54 newdvlanb radiusd[21299]: WARNING: Child is hung for request 9395597 in component authenticate module peap. Oct 12 06:54:54 newdvlanb radiusd[21299]: WARNING: Child is hung for request 9395607 in component authenticate module peap. Oct 12 06:54:57 newdvlanb radiusd[21299]: WARNING: Child is hung for request 9394889 in component authenticate module peap. Oct 12 06:54:58 newdvlanb radiusd[21299]: WARNING: Unresponsive child for request 9394903, in component authenticate module peap Oct 12 06:54:59 newdvlanb radiusd[21299]: WARNING: Child is hung for request 9394903 in component authenticate module peap. Oct 12 06:55:00 newdvlanb radiusd[21299]: WARNING: Child is hung for request 9394945 in component authenticate module peap. Oct 12 06:56:06 newdvlanb radiusd[21299]: WARNING: Module rlm_eap became unblocked for request 9397816 Periodically through the day. In case others are interested in this approach, I am including the configuration notes from our admins to enable radius services on an AD server. There are examples of proxying within "sites-available" On ad-machine.domain.edu they did the following: added "Network Policy And Access Services" role radius config in the Standard Configuration drop down select "RADIUS server for 802.1X Wireless or Wired Connections" click "Configure 802.1X" Setup "Secure Wireless Connections" added radius client rumble.snacks In the "Configure an Authentication Method" screen, selected "Microsoft Protected EAP (PEAP)" In the "Specify Users Groups" screen, added "domain users In the properties of the newly created network policy unchecked "Enable auto-remediation of client computers" Configured Accounting to Log to a txt file then took the defaults on the remaining screens. I have successfully used that as part of an auth-proxy configuration to bypass the need for ntlm_auth (binary) completely.
Thanks for your reply Alan. However, I've not been able to find the exec timeout setting. In which configuration file or module will I find it? Martin. -----Original Message----- From: freeradius-users-bounces+martin.ubank=uwe.ac.uk@lists.freeradius.org [mailto:freeradius-users-bounces+martin.ubank=uwe.ac.uk@lists.freeradius.org] On Behalf Of Alan DeKok Sent: 15 October 2013 12:54 To: FreeRadius users mailing list Subject: Re: Generating timing stats for ntlm_auth Martin Ubank wrote:
As reported to Alan (at eduroam-uk-support), we are also seeing load problems at peak times with similar/identical error messages as those reported in this thread. We are running FreeRadius 2.2.0 on two CentOS 6 VMs, with Kerberos 1.8.2-3.el6 & Samba and Winbind 3.5.6-86.el6.
Hmm... it could be an ntlm_auth issue. Maybe moving to pipes as Phil suggested would help. In any case, it's not new in 2.2.1. So I think it's time to release 2.2.2.
Is there anything we can do here at UWE to help identify the general problem?
Set the exec timeout to 1s. It should take less than a second to run ntlm_auth. If it takes more than a second, something is wrong. And you're probably better off dropping the connection than waiting 30s for a timeout. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On Tue, Oct 15, 2013 at 04:18:45PM +0100, Martin Ubank wrote:
Thanks for your reply Alan. However, I've not been able to find the exec timeout setting. In which configuration file or module will I find it?
Looks like it's hardcoded to 10s in the source (although it is probably easy to make it configurable). Unless I'm missing something, which wouldn't be the first time. Matthew
-----Original Message----- From: freeradius-users-bounces+martin.ubank=uwe.ac.uk@lists.freeradius.org [mailto:freeradius-users-bounces+martin.ubank=uwe.ac.uk@lists.freeradius.org] On Behalf Of Alan DeKok Sent: 15 October 2013 12:54 To: FreeRadius users mailing list Subject: Re: Generating timing stats for ntlm_auth
Martin Ubank wrote:
As reported to Alan (at eduroam-uk-support), we are also seeing load problems at peak times with similar/identical error messages as those reported in this thread. We are running FreeRadius 2.2.0 on two CentOS 6 VMs, with Kerberos 1.8.2-3.el6 & Samba and Winbind 3.5.6-86.el6.
Hmm... it could be an ntlm_auth issue. Maybe moving to pipes as Phil suggested would help.
In any case, it's not new in 2.2.1. So I think it's time to release 2.2.2.
Is there anything we can do here at UWE to help identify the general problem?
Set the exec timeout to 1s. It should take less than a second to run ntlm_auth. If it takes more than a second, something is wrong. And you're probably better off dropping the connection than waiting 30s for a timeout.
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
-- Matthew Newton, Ph.D. <mcn4@le.ac.uk> Systems Specialist, Infrastructure Services, I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom For IT help contact helpdesk extn. 2253, <ithelp@le.ac.uk>
Thanks for the advice Matthew. We haven't altered any source code here at UWE up to now so, before we commit to doing that, we would probably want to know any changes would definitely reduce the number of connection failures. I think we would also need to address our load balancing issues first. Having said that, is there anything we can do (configuration-wise) to assist in diagnosing where the problem lies? Martin. -----Original Message----- From: freeradius-users-bounces+martin.ubank=uwe.ac.uk@lists.freeradius.org [mailto:freeradius-users-bounces+martin.ubank=uwe.ac.uk@lists.freeradius.org] On Behalf Of Matthew Newton Sent: 15 October 2013 17:22 To: FreeRadius users mailing list Subject: Re: Generating timing stats for ntlm_auth On Tue, Oct 15, 2013 at 04:18:45PM +0100, Martin Ubank wrote:
Thanks for your reply Alan. However, I've not been able to find the exec timeout setting. In which configuration file or module will I find it?
Looks like it's hardcoded to 10s in the source (although it is probably easy to make it configurable). Unless I'm missing something, which wouldn't be the first time. Matthew
participants (8)
-
A.L.M.Buxey＠lboro.ac.uk -
Alan DeKok -
Alan DeKok -
Arran Cudbard-Bell -
John Douglass -
Martin Ubank -
Matthew Newton -
Phil Mayers