post-auth for proxied peap inner
We have a need to run the post-auth section on proxied peap/ttls inner (proxied as EAP - none of the crazy packet mangling hacks). 2.x doesn't do this, I haven't checked 3.x but assume it's unchanged? I've tried a few crazy hacks in the source but it all explodes; does anyone have any insight into what needs doing?
Phil Mayers wrote:
We have a need to run the post-auth section on proxied peap/ttls inner (proxied as EAP - none of the crazy packet mangling hacks). 2.x doesn't do this, I haven't checked 3.x but assume it's unchanged?
Actually, 2.2.1 should do this.
I've tried a few crazy hacks in the source but it all explodes; does anyone have any insight into what needs doing?
Magic. It's always magic. Please try the latest code. I think after 3 days of cursing OSX Mavericks, we've got the build working again. There are no more known issues with 2.2.2 or 3.0.1. Alan DeKok.
Hmm... I was building off 2.x.x so not sure. I don't see any code for pushing into the inner tunnel except in the eap->non-eap case - but I could be missing something. -- Sent from my phone with, please excuse brevity and typos
On 25/10/2013 18:18, Alan DeKok wrote:
Phil Mayers wrote:
We have a need to run the post-auth section on proxied peap/ttls inner (proxied as EAP - none of the crazy packet mangling hacks). 2.x doesn't do this, I haven't checked 3.x but assume it's unchanged?
Actually, 2.2.1 should do this.
I've tried a few crazy hacks in the source but it all explodes; does anyone have any insight into what needs doing?
Magic. It's always magic.
I'm having a *really* hard time understanding how this works at all; I don't get how the code in peap.c:~1126 actually causes a proxy request to be sent; ultimately it's all called via rad_authenticate, which only seems to check/process request->proxy after authorize, when rlm_eap does all it's work in authenticate. Put another way - the original PEAP request containing the PEAP inner comes into rad_authenticate via listen.c - I don't see how, once TLS is decoded and peap.c has run the fake request via the inner tunnel server, how the proxy packet gets sent and replied to. (The reason for wanting to know this is to understand where to put the processing code so that the "fake" can be pushed through post-auth correctly without breaking "proxy as non-EAP" workaround)
Phil Mayers wrote:
I'm having a *really* hard time understanding how this works at all; I don't get how the code in peap.c:~1126 actually causes a proxy request to be sent; ultimately it's all called via rad_authenticate, which only seems to check/process request->proxy after authorize, when rlm_eap does all it's work in authenticate.
Yes. That's exactly it. The PEAP module bails out of the authenticate section. The packet is proxied. When the proxy reply arrives, the PEAP is called again. See rlm_eap.c, eap_post_proxy() for more magic.
Put another way - the original PEAP request containing the PEAP inner comes into rad_authenticate via listen.c - I don't see how, once TLS is decoded and peap.c has run the fake request via the inner tunnel server, how the proxy packet gets sent and replied to.
(The reason for wanting to know this is to understand where to put the processing code so that the "fake" can be pushed through post-auth correctly without breaking "proxy as non-EAP" workaround)
That was done in rad_authenticate, IIRC. See commit 860dd99c9d Alan DeKok.
On 26/10/2013 19:39, Alan DeKok wrote:
Phil Mayers wrote:
I'm having a *really* hard time understanding how this works at all; I don't get how the code in peap.c:~1126 actually causes a proxy request to be sent; ultimately it's all called via rad_authenticate, which only seems to check/process request->proxy after authorize, when rlm_eap does all it's work in authenticate.
Yes. That's exactly it. The PEAP module bails out of the authenticate section. The packet is proxied. When the proxy reply arrives, the PEAP is called again.
Sure; it's not strictly relevant here, but what confused me is how the middle bit works - peap bails, but what causes the proxy packet to be emitted? I see how this happens if you bail in "authorize", but not "authenticate". (This is not hugely important however, it must be said)
See rlm_eap.c, eap_post_proxy() for more magic.
Put another way - the original PEAP request containing the PEAP inner comes into rad_authenticate via listen.c - I don't see how, once TLS is decoded and peap.c has run the fake request via the inner tunnel server, how the proxy packet gets sent and replied to.
(The reason for wanting to know this is to understand where to put the processing code so that the "fake" can be pushed through post-auth correctly without breaking "proxy as non-EAP" workaround)
That was done in rad_authenticate, IIRC. See commit 860dd99c9d
I'm not seeing that, or I'm doing something wrong. AFAICT rad_virtual_server only calls post-auth if request->reply->code is set, and for packets which are proxied, it is==0 because the reply hasn't been generated yet. The post-proxy/post-auth code for eap/peap ultimately ends up in eappeap_postproxy and AFAICT only calls post-auth if "fake" is found in REQUEST_DATA_EAP_MSCHAP_TUNNEL_CALLBACK, which is only set for the "proxy inner as non-eap" case? In other words, I think post-auth works for peap proxied inner only if "proxy_tunneled_request_as_eap = no", which is not the case for what I'm trying to do. The thing I'm actually trying to do is cause the inner eap-mschapv2 requests to be proxied [to another FR process] to free up the outer processing thread ASAP - this relates to the recent issues we and others have been having with peap/ntlm_auth timeouts under load spikes, which seem to be some sort of Samba concurrency issue - I've used packet captures to time the on-wire response times for the MS-RPC calls from winbind, and they're completing in a timely fashion, but something in the fork/exec/ntlm_auth/winbind-pipe sequence is going slow and blocking the request thread pool :o( Throwing hardware at the problem is the other solution, but I have no way of knowing if this will solve or hide the problem - it could come back at any time, and I really want to understand the cause (e.g. is it the load of fork/exec or does ntlm_auth/winbind have some kind of built-in concurrency issue?)
Phil Mayers wrote:
Sure; it's not strictly relevant here, but what confused me is how the middle bit works - peap bails, but what causes the proxy packet to be emitted? I see how this happens if you bail in "authorize", but not "authenticate".
The proxy handler is independent of authorize / authenticate, IIRC.
I'm not seeing that, or I'm doing something wrong. AFAICT rad_virtual_server only calls post-auth if request->reply->code is set, and for packets which are proxied, it is==0 because the reply hasn't been generated yet.
I'll have to take a look. I've been finalizing 3.0.1, and haven't looked at 2.2.x for a while.
The post-proxy/post-auth code for eap/peap ultimately ends up in eappeap_postproxy and AFAICT only calls post-auth if "fake" is found in REQUEST_DATA_EAP_MSCHAP_TUNNEL_CALLBACK, which is only set for the "proxy inner as non-eap" case?
Yes.
In other words, I think post-auth works for peap proxied inner only if "proxy_tunneled_request_as_eap = no", which is not the case for what I'm trying to do.
Hmm... if that's so, it needs to be fixed. But likely in 3.x, because that code is enormously better. Some changes for 3.1 should make this much easier to do, and remove all of the weird issues around proxying inner tunnel stuff.
The thing I'm actually trying to do is cause the inner eap-mschapv2 requests to be proxied [to another FR process] to free up the outer processing thread ASAP - this relates to the recent issues we and others have been having with peap/ntlm_auth timeouts under load spikes, which seem to be some sort of Samba concurrency issue - I've used packet captures to time the on-wire response times for the MS-RPC calls from winbind, and they're completing in a timely fashion, but something in the fork/exec/ntlm_auth/winbind-pipe sequence is going slow and blocking the request thread pool :o(
Hmm... not nice. That could be fixed another way, but I understand what you're trying to do. v2.x.x has some fixes which let you manually set the timeout for exec / ntlm_auth. If that doesn't work well, there are other fixes.
Throwing hardware at the problem is the other solution, but I have no way of knowing if this will solve or hide the problem - it could come back at any time, and I really want to understand the cause (e.g. is it the load of fork/exec or does ntlm_auth/winbind have some kind of built-in concurrency issue?)
The load of fork / exec shouldn't be a problem. Perhaps a better solution would be to run ntlm_auth in "pipe" mode. That would remove the fork/exec issue, and perhaps allow for greater concurrency. Alan DeKok.
On 28/10/13 14:31, Alan DeKok wrote:
The load of fork / exec shouldn't be a problem.
Unfortunately we ran out of time debugging this properly and just threw hardware at the problem, and it has resolved the issue. Annoyingly, it was expedient to roll out the new hardware with our existing 2.x.x config, which means I need to go back to the start of my 3.x migration :o( I plan to go back and reproduce on the old server once I have time and definitely identify whether it was an internal winbind concurrency issue, fork/exec load or something else - it's possible we were just loading the old boxes too hard, and the new hardware is a *lot* quicker.
Perhaps a better solution would be to run ntlm_auth in "pipe" mode. That would remove the fork/exec issue, and perhaps allow for greater concurrency.
There's some evidence Samba 3.6 is better in this regard too - for one thing, you can set "winbind max domain connections" and have >1 DC MSRPC pipe.
participants (2)
-
Alan DeKok -
Phil Mayers