Re: Degradation of service when authentication fails with Windows AD
If you can describe the problem you're having, in correct terminology, people might be able to make a suggestion. Be specific, about the issues, the architecture you have, what you're trying to achieve, and so on.
Sorry, I will try to explain the problem better. I have a mail server where users are validated with local accounts (UNIX) or against a Windows AD. For this reason we use Radius. Sometimes the Radius server fails and stops authentication for everybody. In that point the logs that I sent to you appear. I need to restart Radius in order to it works again.
From what you've described so far, it sounds like you are losing connectivity to one or more AD controllers, which is causing PAM to hang (waiting for a Kerberos reply) or Samba/ntlm_auth to hang (waiting for an RPC reply).
It should be obvious what the solution is - reliable connectivity to a reliable AD controller.
When we monitored the network and one of the Windows AD we could confirm that requests from Radius don't reach the AD, because they don't leave Radius. We believe that connectivity between Radius and AD is correct, they are on the same LAN and the AD continues to validate correctly even when Radius is failing.
From my point of view, for any reason, Radius receives requests that it can not manage, because of the AD, the network or whatever. These requests keep waiting and the buffer fills completly. I don't know why these requests are not removed from the queue and the buffer is cleared in order to allow new request. In this way we could avoid that request to the rest of the AD fail because of a particular AD.
On 11/02/13 11:23, Antonio Alberola wrote:
When we monitored the network and one of the Windows AD we could confirm that requests from Radius don't reach the AD, because they don't leave Radius. We believe that connectivity between Radius and AD is correct, they are on the same LAN and the AD continues to validate correctly even when Radius is failing.
Let me put this as simply as possible, but I'm getting tired of repeating myself: Do not use PAM. PAM APIs are blocking. If PAM or an underlying module hangs, the thread in FreeRADIUS will hang. If you keep receiving authentication requests, eventually the entire thread pool will be blocked. Do not use PAM. Is this clear?
From my point of view, for any reason, Radius receives requests that it can not manage, because of the AD, the network or whatever. These requests keep waiting and the buffer fills completly. I don't know why these requests are not removed from the queue and the buffer is cleared in order to allow new request. In this way we could avoid that request to the rest of the AD fail because of a particular AD.
Because FreeRADIUS doesn't work that way. It uses a thread pool (of finite size) to process requests in a blocking fashion. If the API or intermediate system it calls blocks forever, FreeRADIUS blocks forever, and if you keep receiving requests, eventually your entire thread pool will be exhausted. FreeRADIUS cannot timeout synchronous C API calls. This is impossible to do safely in POSIX. Is this clear? Only a few FreeRADIUS modules can "timeout" requests. The "exec" module (and functionality built on top of it) is one of them. Therefore, you should be using "exec" wrapping "ntlm_auth" from Samba. Unfortunately, the "exec" timeout is hard-coded in the source at 10 seconds. Therefore, depending on your load, you might still suffer thread pool exhaustion. But it will self-correct, because the requests will eventually time out. I hope I had made myself clear. Regards, Phil
Antonio Alberola wrote:
I have a mail server where users are validated with local accounts (UNIX) or against a Windows AD. For this reason we use Radius. Sometimes the Radius server fails and stops authentication for everybody. In that point the logs that I sent to you appear. I need to restart Radius in order to it works again.
The RADIUS *server*, or the entire machine? You've been vague as to what you mean. Please be precise. It's the only way we can help you.
When we monitored the network and one of the Windows AD we could confirm that requests from Radius don't reach the AD, because they don't leave Radius.
Again, the RADIUS *server* doesn't contact AD. It's another component on the same machine. Maybe Kerberos, maybe Samba. Have you tried to find out *which* component is causing the problem?
We believe that connectivity between Radius and AD is correct, they are on the same LAN and the AD continues to validate correctly even when Radius is failing.
That doesn't mean much. It's nice, but the problem could be somewhere else. i.e. I've seen people put firewalls between the RADIUS server and a database. The firewall then drops the database connections RADIUS started. So RADIUS gets blocked. But you can still ping the DB from the RADIUS machine. And new connections work fine.
From my point of view, for any reason, Radius receives requests that it can not manage, because of the AD, the network or whatever. These requests keep waiting and the buffer fills completly. I don't know why these requests are not removed from the queue and the buffer is cleared in order to allow new request.
Because FreeRADIUS doesn't implement *EVERYTHING* itself. It relies on libraries / other programs for AD connectivity. If those libraries block, the underlying APIs often don't *allow* FreeRADIUS to detect that and recover. You need to stop blaming FreeRADIUS. It's preventing you from finding out what the real problem is. Again, it's like you're trying to drive a car with no petrol in it. You're stuck looking at the gauge in front of you. You're thinking you may need to replace it. All the time we're trying to tell you PUT MORE PETROL IN THE TANK. Start paying attention to the responses on this list. It's the only way you'll get the problem solved. Alan DeKok.
participants (3)
-
Alan DeKok -
Antonio Alberola -
Phil Mayers