Upstream NAS flooding my radius
I have a setup such as: NAS > Freeradius Proxy > Freeradius Auth Periodically the NAS (different company and outside of my control) gets rebooted and when it starts up it sends thousands of simultaneous requests to the radius proxy, which in turn forwards them all to the appropriate freeradius auth server. The challenge is, the auth server can get overwhelmed and starts throwing some of these errors: Error: rlm_sql (sql): There are no DB handles to use! I'm curious if there's an elegant way on the freeradius proxy to rate limit the # of connections per second being thrown at the auth server. Ideally if the threshold is reached, requests above the threshold would be dropped (and not rejected). I've searched all the docs, the mailing list, and config files and haven't turned up a solution yet. Any help or advice is appreciated. - N
On Thu, Dec 15, 2011 at 9:11 AM, Nathan M <locu.lists@gmail.com> wrote:
I have a setup such as:
NAS > Freeradius Proxy > Freeradius Auth
Periodically the NAS (different company and outside of my control) gets rebooted and when it starts up it sends thousands of simultaneous requests to the radius proxy, which in turn forwards them all to the appropriate freeradius auth server. The challenge is, the auth server can get overwhelmed and starts throwing some of these errors:
Error: rlm_sql (sql): There are no DB handles to use!
I'm curious if there's an elegant way on the freeradius proxy to rate limit the # of connections per second being thrown at the auth server. Ideally if the threshold is reached, requests above the threshold would be dropped (and not rejected). I've searched all the docs, the mailing list, and config files and haven't turned up a solution yet.
Why would you want it dropped? The NAS would then resend the request, causing even MORE load to your server. It's better to just reject it immediately. The best options I can give right now is either: - just accept it. There's no magic setting that would automagically make your problem go away. OR, - use a fast backend. If you use files for authentication, AND accounting to detail file, AND don't activate any kind of simultaneous check, it's easy to get several thousand auth/s. OR - make your db faster. It might involve hiring a qualified DBA & sysadmin, or implementing mysql cluster, or buying something like http://www.clustrix.com/products/ -- Fajar
Hi,
Error: rlm_sql (sql): There are no DB handles to use!
improve your SQL performance - eg use InnoDB instead of myISAM , or postgresQL instead of MySQL increase number of PERL and SQL instances use another 'non-inline' method to handle the accounting - so its buffered and put into DB when daemon is free - eg use the 'buffered_sql' virtual server alan
I appreciate the replies and suggestions to upgrade the SQL infrastructure. What I'm attempting to do is to basically limit a friendly DOS attack. I think throttling the offender is a better approach than adding more hardware in this case. Maybe inside freeradius isn't the answer, and maybe a firewall rule would be better suited for the task. Although this seems like it would be a common issue, especially with lots of new wirless ISPs which have their radios rebooted (thereby causing re-auth of all connected customers upon reboot) far more frequently than traditional wireline ISPs. @Fajar - the intent in having them dropped is exactly that. I don't want the end-user trying to authenticate to fail authentication, I do want the NAS to retry. I just want to control how quickly it can retry from my end. If anyone else has experience solving the source of the problem, ideally at the proxy process level, I'm definitely open to suggestions and experience. Thanks, - N On Thu, Dec 15, 2011 at 12:58 AM, Alan Buxey <A.L.M.Buxey@lboro.ac.uk> wrote:
Hi,
Error: rlm_sql (sql): There are no DB handles to use!
improve your SQL performance - eg use InnoDB instead of myISAM , or postgresQL instead of MySQL
increase number of PERL and SQL instances
use another 'non-inline' method to handle the accounting - so its buffered and put into DB when daemon is free - eg use the 'buffered_sql' virtual server
alan - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On Fri, Dec 16, 2011 at 1:02 AM, Nathan M <locu.lists@gmail.com> wrote:
@Fajar - the intent in having them dropped is exactly that. I don't want the end-user trying to authenticate to fail authentication, I do want the NAS to retry. I just want to control how quickly it can retry from my end.
Have you actually tried that? Your "solution" may actually be worse than the original problem. In my system, when radius reponds too late, the NAS will either: - mark the radius server dead, OR - have ridiculously high load that it's unable to function correctly, due to the high number of ongoing authentication request. In either case, the user will have to reauthenticate anyway. This is usually not a problem, since most of my clients are ADSL with the modem doing automatic reauthentication as needed. That's why is better to respond as quickly as possible, without any delay. Even if the response is reject. -- Fajar
Nathan M wrote:
@Fajar - the intent in having them dropped is exactly that. I don't want the end-user trying to authenticate to fail authentication, I do want the NAS to retry. I just want to control how quickly it can retry from my end.
RADIUS doesn't work like that.
If anyone else has experience solving the source of the problem, ideally at the proxy process level, I'm definitely open to suggestions and experience.
People generally upgrade their systems to deal with peak loads. Alan DeKok.
On Fri, Dec 16, 2011 at 5:18 AM, Alan DeKok <aland@deployingradius.com> wrote:
People generally upgrade their systems to deal with peak loads.
Thanks for the input. As previously mentioned, it's not what I would really consider a peak load issue, but more of a DOS from a mis-configured or poorly managed NAS, which is entirely outside of our control; however, we have to accept packets from it and have no ability to resolve whatever their problem may be. Rather than updating the systems to accommodate 6000 packets/second from this one NAS when it sends a spew of authentication requests, I'll share the solution I'm trying out in case anyone else winds up in a similar predicament. In all tests thus far, this shaves off the wild peak load and distributes it over a short period of time as the NAS I'm working with will retry frequently if the first attempt is dropped. iptables has a recent module which seems to fit the bill as it also tracks source IP. The limit module was also pretty close; however, it limits all packets to a port ignoring the source IP. The rules below using the recent module, which on RHEL/CentOS system go into /etc/sysconfig/iptables limit inbound auth packet rates to no more than 60/second, and acct packets to 120/second from any given source IP. (ie.. if there are 2 separate sources of packets it would allow 60/s auth requests from each). These same rules would protect against other DOS based attacks to these ports as well I believe. -A INPUT -p udp -m udp --dport 1645 -m state --state NEW -m recent --rcheck --seconds 1 --hitcount 60 --name RADAUTH --rsource -j LOG -A INPUT -p udp -m udp --dport 1645 -m state --state NEW -m recent --rcheck --seconds 1 --hitcount 60 --name RADAUTH --rsource -j DROP -A INPUT -p udp -m udp --dport 1645 -m state --state NEW -m recent --set --name RADAUTH --rsource -A INPUT -m state --state NEW -m udp -p udp --dport 1645 -j ACCEPT -A INPUT -p udp -m udp --dport 1646 -m state --state NEW -m recent --rcheck --seconds 1 --hitcount 120 --name RADACCT --rsource -j LOG -A INPUT -p udp -m udp --dport 1646 -m state --state NEW -m recent --rcheck --seconds 1 --hitcount 120 --name RADACCT --rsource -j DROP -A INPUT -p udp -m udp --dport 1646 -m state --state NEW -m recent --set --name RADACCT --rsource -A INPUT -m state --state NEW -m udp -p udp --dport 1646 -j ACCEPT - N
Nathan M wrote:
Thanks for the input. As previously mentioned, it's not what I would really consider a peak load issue, but more of a DOS from a mis-configured or poorly managed NAS, which is entirely outside of our control; however, we have to accept packets from it and have no ability to resolve whatever their problem may be.
That is distinctly anti-social behavior from whoever owns the NAS.
Rather than updating the systems to accommodate 6000 packets/second from this one NAS when it sends a spew of authentication requests, I'll share the solution I'm trying out in case anyone else winds up in a similar predicament. In all tests thus far, this shaves off the wild peak load and distributes it over a short period of time as the NAS I'm working with will retry frequently if the first attempt is dropped.
Another solution is to use RADIUS. :) Set up a proxy for ONLY that NAS. Call it "A". Have it proxy ALL packets to the local proxy you're already running, "B". This configuration should be very, very, small. You can strip out 99% of the normal server configuration. In the configuration for "A", set "max_outstanding" to a low value, like 100 or 200. See raddb/proxy.conf for details. Then, in the "post-proxy type Fail" section, set "do_not_respond". This configuration limits the proxy load to no more than the upstream can handle. It also throws away packets when it receives too many. It's a bit more work than iptables, but it's cross-platform, and guaranteed to work. Alan DeKok.
On Fri, Dec 16, 2011 at 9:17 PM, Alan DeKok <aland@deployingradius.com> wrote:
That is distinctly anti-social behavior from whoever owns the NAS.
Agreed.
Another solution is to use RADIUS. :)
Set up a proxy for ONLY that NAS. Call it "A". Have it proxy ALL packets to the local proxy you're already running, "B". This configuration should be very, very, small. You can strip out 99% of the normal server configuration.
In the configuration for "A", set "max_outstanding" to a low value, like 100 or 200. See raddb/proxy.conf for details.
Then, in the "post-proxy type Fail" section, set "do_not_respond".
This configuration limits the proxy load to no more than the upstream can handle. It also throws away packets when it receives too many.
It's a bit more work than iptables, but it's cross-platform, and guaranteed to work.
Alan DeKok. -
Bingo! That's what I was looking for. Thanks Alan. - N
participants (4)
-
Alan Buxey -
Alan DeKok -
Fajar A. Nugraha -
Nathan M