FreeRadius high availability lab
Hi Users, I started a test for a large deployment and this topology was recommended, I tested it in the lab and apparently did it work, any comments from someone about the attached topology? is there something like this in production? Claudio.
On Nov 6, 2017, at 3:52 PM, Claudio Carvalho <claudio@nextdigitalservices.com.br> wrote:
I started a test for a large deployment and this topology was recommended, I tested it in the lab and apparently did it work, any comments from someone about the attached topology? is there something like this in production?
The mailing list strips attachments. Too many people were sending in JPG images of a screenshot of the debug output. Just describe the topology. Alan DeKok.
sorry, I forgot that the service does not carry images, below the link, thanks. http://www.nextdigitalservices.com.br/images/radiustopology.png Claudio. -----Mensagem original----- De: Freeradius-Users [mailto:freeradius-users-bounces+claudio=nextdigitalservices.com.br@lists.freeradius.org] Em nome de Alan DeKok Enviada em: segunda-feira, 6 de novembro de 2017 19:17 Para: FreeRadius users mailing list <freeradius-users@lists.freeradius.org> Assunto: Re: FreeRadius high availability lab On Nov 6, 2017, at 3:52 PM, Claudio Carvalho <claudio@nextdigitalservices.com.br> wrote:
I started a test for a large deployment and this topology was recommended, I tested it in the lab and apparently did it work, any comments from someone about the attached topology? is there something like this in production?
The mailing list strips attachments. Too many people were sending in JPG images of a screenshot of the debug output. Just describe the topology. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Claudio Carvalho wrote:
sorry, I forgot that the service does not carry images, below the link, thanks.
http://www.nextdigitalservices.com.br/images/radiustopology.png
To me the SQL DB seems to be your SPOF. Or maybe it's simply the picture not fully reflecting the HA setup of your DB. Ciao, Michael.
Hi, Kindly describe the topology so that we can clearly understand and have an idea. Regards, On Tue, Nov 7, 2017 at 3:03 AM, Claudio Carvalho <claudio@nextdigitalservices.com.br> wrote:
sorry, I forgot that the service does not carry images, below the link, thanks.
http://www.nextdigitalservices.com.br/images/radiustopology.png
Claudio.
-----Mensagem original----- De: Freeradius-Users [mailto:freeradius-users-bounces+claudio=nextdigitalservices.com.br@lists.freeradius.org] Em nome de Alan DeKok Enviada em: segunda-feira, 6 de novembro de 2017 19:17 Para: FreeRadius users mailing list <freeradius-users@lists.freeradius.org> Assunto: Re: FreeRadius high availability lab
On Nov 6, 2017, at 3:52 PM, Claudio Carvalho <claudio@nextdigitalservices.com.br> wrote:
I started a test for a large deployment and this topology was recommended, I tested it in the lab and apparently did it work, any comments from someone about the attached topology? is there something like this in production?
The mailing list strips attachments. Too many people were sending in JPG images of a screenshot of the debug output.
Just describe the topology.
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
what is your NGINX load balancer balancing, the RADIUS requests? if so, does it understand RADIUS? your SQL entry is a single box...just one single DB or is that an HA pair or more? for high availability, you might be better off with eg REDIS/MoNGO style DB running on the RADIUS boxes themselves (all data sharded etc). what sort of load are you talking about - why the 4 boxes etc - many questions before a random drawing showing a basic load balanced front end alan
On 8/11/2017, at 12:34 AM, Alan Buxey <alan.buxey@gmail.com> wrote:
what is your NGINX load balancer balancing, the RADIUS requests? if so, does it understand RADIUS? your SQL entry is a single box...just one single DB or is that an HA pair or more? for high availability, you might be better off with eg REDIS/MoNGO style DB running on the RADIUS boxes themselves (all data sharded etc). what sort of load are you talking about - why the 4 boxes etc - many questions before a random drawing showing a basic load balanced front end
Looks like a common, basic load balancer design done by someone who hasn’t been burned trying to do this sort of thing. Idealistic sort of thing from a lecturer maybe. First thing I’d say is don’t have both load balancers sending traffic to all the RADIUS servers. You don’t gain much resiliency there except in the load balancer - if the pool of RADIUS servers dies for some reason (it will, at some point) you lose your service. Split the backends in to two pools, if you don’t have enough in each pool, get more - don’t merge the pools. NGINX seems like a very odd choice for a load balancer, but, if you’re comfortable with it then maybe it’s ok.. Other than that, there are so, so many questions that need to be asked before you start drawing this sort of picture. In addition to the above: - What is RADIUS doing? Auth? Accounting? Both? - Can you accept lag in accounting getting to the DB? How much? - Can you accept lag in password/user changes? How much? - Can you accept accounting black spots? How long/often? - What conditions cause your NAS to use the secondary server? - Are you doing anything that requires shared state (i.e. IP address assignment, multiple login prevention, etc.)? How important is integrity here if you have to trade it for availability? These are just some that I came up with off the top of my head.. there are many, many more that would come from the answers to these questions.. Don’t try and design your solution to cater for failure modes that don’t cause problems in your specific environment by adding needless complexity, as adding complexity will cause failures of its own. -- Nathan Ward
On Nov 7, 2017, at 6:49 AM, Nathan Ward <lists+freeradius@daork.net> wrote:
First thing I’d say is don’t have both load balancers sending traffic to all the RADIUS servers. You don’t gain much resiliency there except in the load balancer - if the pool of RADIUS servers dies for some reason (it will, at some point) you lose your service. Split the backends in to two pools, if you don’t have enough in each pool, get more - don’t merge the pools.
The approach I often take is: - two data centres. Each NAS points to one DC as primary, and the other as secondary - each DC is the same - two RADIUS servers, sharing an IP via VRRP - two DBs. - each RADIUS server can write to both SQL DBs, in a fail-over configuration. That's good enough for most common uses. If people need higher performance, the front-end RADIUS servers become proxies, and they load-balance to a set of back-ends, which are all identical (RADIUS + SQL). The load-balancing is done on a hash of User-Name, for simple sharing and consistency. The main issue here is merging all of the data which is split into different SQL DBs. But that can be done off-line, i.e. *not* live, and *not* by the RADIUS server. Much of "high performance" system design ends up working around performance limitations with databases.
Other than that, there are so, so many questions that need to be asked before you start drawing this sort of picture. In addition to the above: - What is RADIUS doing? Auth? Accounting? Both? - Can you accept lag in accounting getting to the DB? How much? - Can you accept lag in password/user changes? How much? - Can you accept accounting black spots? How long/often? - What conditions cause your NAS to use the secondary server? - Are you doing anything that requires shared state (i.e. IP address assignment, multiple login prevention, etc.)? How important is integrity here if you have to trade it for availability?
i.e. What problem are you trying to solve? Knowing that will lead you to a solution.
These are just some that I came up with off the top of my head.. there are many, many more that would come from the answers to these questions.. Don’t try and design your solution to cater for failure modes that don’t cause problems in your specific environment by adding needless complexity, as adding complexity will cause failures of its own.
Very true. Alan DeKok.
It's interesting to read this, as the suggestions mirror what I've learned over the years. This is on a single university campus, so everything is geographically local. But we have two DCs, a FreeRADIUS server in each. The servers each have a floating "primary" address, which magically floats to the other box using keepalived. So if Server A fails, Server B adopts failed IP address. NAS's (in our case, switches & APs) never have to deal with failover. Makes things SO much smoother. Holy cow was this a "lightbulb moment". We have a database as the "authoritative record", but everything is Redis-cached on each FreeRADIUS server. It's a bit of a chore to keep the cache synchronized, but it means that FreeRADIUS is completely self-sufficient. We use rlm_perl to read this cache when responding to requests, and to log accounting updates back to the Redis cache. There are helper scripts (on other servers) which grab those updates and log them to the database. If there is a flurry of updates, then the Redis cache queues up for a few seconds while the database catches up. But FreeRADIUS chugs along, even if the database server completely blows up. This happened for the first time 2 weeks ago. RAID controller died. The Redis cache queued up all the requests until we got the backup database server online. Fun times. I'm curious if rlm_redis + unlang would be more performant than rlm_perl + perl script. Haven't had time to test this yet. Norman On Tue, Nov 7, 2017 at 8:54 AM, Alan DeKok <aland@deployingradius.com> wrote:
On Nov 7, 2017, at 6:49 AM, Nathan Ward <lists+freeradius@daork.net> wrote:
First thing I’d say is don’t have both load balancers sending traffic to all the RADIUS servers. You don’t gain much resiliency there except in the load balancer - if the pool of RADIUS servers dies for some reason (it will, at some point) you lose your service. Split the backends in to two pools, if you don’t have enough in each pool, get more - don’t merge the pools.
The approach I often take is:
- two data centres. Each NAS points to one DC as primary, and the other as secondary - each DC is the same - two RADIUS servers, sharing an IP via VRRP - two DBs. - each RADIUS server can write to both SQL DBs, in a fail-over configuration.
That's good enough for most common uses. If people need higher performance, the front-end RADIUS servers become proxies, and they load-balance to a set of back-ends, which are all identical (RADIUS + SQL). The load-balancing is done on a hash of User-Name, for simple sharing and consistency.
The main issue here is merging all of the data which is split into different SQL DBs. But that can be done off-line, i.e. *not* live, and *not* by the RADIUS server.
Much of "high performance" system design ends up working around performance limitations with databases.
Other than that, there are so, so many questions that need to be asked before you start drawing this sort of picture. In addition to the above: - What is RADIUS doing? Auth? Accounting? Both? - Can you accept lag in accounting getting to the DB? How much? - Can you accept lag in password/user changes? How much? - Can you accept accounting black spots? How long/often? - What conditions cause your NAS to use the secondary server? - Are you doing anything that requires shared state (i.e. IP address assignment, multiple login prevention, etc.)? How important is integrity here if you have to trade it for availability?
i.e. What problem are you trying to solve? Knowing that will lead you to a solution.
These are just some that I came up with off the top of my head.. there are many, many more that would come from the answers to these questions.. Don’t try and design your solution to cater for failure modes that don’t cause problems in your specific environment by adding needless complexity, as adding complexity will cause failures of its own.
Very true.
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On Nov 7, 2017, at 4:14 PM, Norman Elton <normelton@gmail.com> wrote:
It's interesting to read this, as the suggestions mirror what I've learned over the years.
That's experience for you.
This is on a single university campus, so everything is geographically local. But we have two DCs, a FreeRADIUS server in each. The servers each have a floating "primary" address, which magically floats to the other box using keepalived. So if Server A fails, Server B adopts failed IP address. NAS's (in our case, switches & APs) never have to deal with failover. Makes things SO much smoother. Holy cow was this a "lightbulb moment".
Yeah. The NAS implementations of fail-over are *terrible*. Even equipment costing $100K often just does primary + secondary. With fail-over when the primary goes down... and never fail-back until the system reboots. Remember: NASes are dumb. RADIUS servers are smart. Put the smarts into the RADIUS server and/or the network. Don't rely on the NASes for *anything* other than network access.
We have a database as the "authoritative record", but everything is Redis-cached on each FreeRADIUS server. It's a bit of a chore to keep the cache synchronized, but it means that FreeRADIUS is completely self-sufficient. We use rlm_perl to read this cache when responding to requests, and to log accounting updates back to the Redis cache.
That should be possible with unlang. Maybe a bit more awkward than Perl, but is should work.
There are helper scripts (on other servers) which grab those updates and log them to the database. If there is a flurry of updates, then the Redis cache queues up for a few seconds while the database catches up. But FreeRADIUS chugs along, even if the database server completely blows up. This happened for the first time 2 weeks ago. RAID controller died. The Redis cache queued up all the requests until we got the backup database server online. Fun times.
Yup. The detail file and detail reader does this, too. Arguably not as catchy as Redis, but it works. We're working on making the detail file reader faster in v4, too.
I'm curious if rlm_redis + unlang would be more performant than rlm_perl + perl script. Haven't had time to test this yet.
Send the scripts over and we'll take a look. If they're useful to you, they're probably useful to other people, too. Alan DeKo.
hi,
I'm curious if rlm_redis + unlang would be more performant than rlm_perl + perl script. Haven't had time to test this yet.
my first instinct would be to say yes, very much so (given the massive gains over using native rlm_ldap over PERL ldap code) however, it depends if the REDIS module can do the things your PERL script does....if so, then its trivial to benchmark it on a VM to get a speed comparison. alan
On Nov 6, 2017, at 4:33 PM, Claudio Carvalho <claudio@nextdigitalservices.com.br> wrote:
sorry, I forgot that the service does not carry images, below the link, thanks.
http://www.nextdigitalservices.com.br/images/radiustopology.png
I'll echo most of the other comments. For one, nginx is an odd choice. It does UDP load-balancing. It does *not* do RADIUS load balancing. This means that you need to break your RADIUS configuration in order to support nginx. i.e. the servers will see the load-balancer as the source of all packets, and the clients will see the load-balancer as the RADIUS server. This might not be what you want. If all of your NASes are the same, it should probably be fine. If you're not doing EAP, it's fine. If you are doing EAP, you want a RADIUS-aware load balancer. The SQL DB is a single point of failure. Worse, it's a single point of performance loss. FreeRADIUS can handle 20K packets/s without a problem. SQL servers typically fall over at about 1K writes/s. This means that the smarts of the system should be in FreeRADIUS. i.e. if the SQL database is down, FR writes to "detail" files, etc. Then when SQL is back up, FR reads the detail files and writes to SQL. You also need to split the SQL database, especially for accounting. If you're doing monthly billing queries, those queries will *hammer* the SQL DB, and cause problems for FR. Instead, build a master / slave system. FR writes to the master. The master replicates to the slave. Billing queries are done against the slave. NAS fail-over is also terrible. You're usually better off doing fail-over via VRPP, or something similar. But again... doing RADIUS-aware fail-over is better than trying to bolt it on externally. You also haven't said how many packets/s this system is supposed to handle. If it's 100 packets/s, this system is overkill. If it's 2000+ packets/s, your database will die long before the RADIUS servers get 10% busy. As I keep recommending on this list, you should be talking about problems, not solutions. What problem do you have? What do you need the RADIUS server to do? Once we know that, we can recommend a solution. But until then, I wouldn't recommend using the proposed design. Alan DeKok.
On Nov 6, 2017, at 9:17 PM, Alan DeKok <aland@deployingradius.com> wrote:
On Nov 6, 2017, at 3:52 PM, Claudio Carvalho <claudio@nextdigitalservices.com.br> wrote:
I started a test for a large deployment and this topology was recommended, I tested it in the lab and apparently did it work, any comments from someone about the attached topology? is there something like this in production?
The mailing list strips attachments. Too many people were sending in JPG images of a screenshot of the debug output.
Well that and those obnoxiously large signature images. The list allows PNGs and SVGs because they're smart people formats :) -Arran
participants (8)
-
Alan Buxey -
Alan DeKok -
Arran Cudbard-Bell -
Burn Zero -
Claudio Carvalho -
Michael Ströder -
Nathan Ward -
Norman Elton