Hello everyone from a long-time user of freeradius, but this is my first post on the list. While playing around with the experimental DHCP code in the latest freeradius releases, I came across a problem where the DHCP packet is not recognized, when the Message-Type is not the first among the list of options. Currently, the code uses a static offset to look for the message type in the packet. A patch is enclosed to fix the problem by iterating through the available options, but will most likely need fixing by someone who knows how to code :-) It applies to 2.0.6 and works in the lab, though. Also, I'm interested to hear about the plans for DHCP in freeradius. Personally, I'm evaluating a solution to assign static IPs based on the Agent-Circuit-Id without being tied to a specific MAC - something ISC dhcpd will not do by design. This is supposed to eventually replace PPPoE connections from CPEs. I guess freeradius could someday be able to handle this since the backend data store is really flexible. While most users will probably use the DHCP code with a leases database, in our scenario the configuration would just hand out a static IP calculated from the values in the Agent-Circuit-ID, probably by making use of scripting. Another suggestion with regards to the server design. Currently, the whole server needs to run as root to bind to port 67. That's not necessary for a RADIUS server binding only to ports above 1024. For the long run, would it be feasible to separate the DHCP server code into its own daemon, as a pure DHCP-to-RADIUS translator? Best regards, Martin Lorentz
Martin Lorentz wrote:
While playing around with the experimental DHCP code in the latest freeradius releases, I came across a problem where the DHCP packet is not recognized, when the Message-Type is not the first among the list of options.
Yes... it is usually first, but it doesn't necessarily have to be.
Currently, the code uses a static offset to look for the message type in the packet. A patch is enclosed to fix the problem by iterating through the available options, but will most likely need fixing by someone who knows how to code :-) It applies to 2.0.6 and works in the lab, though.
It should apply to recent versions.
Also, I'm interested to hear about the plans for DHCP in freeradius.
"Take over the world". I'm giving a talk at LinuxTag in a few weeks: http://www.linuxtag.org/2009/de/program/freies-vortragsprogramm/rednerliste/... Really... I'm astonished that in 2009, the most widely used DHCP server doesn't do the new-fangled "database" thing. It's bizarre.
Personally, I'm evaluating a solution to assign static IPs based on the Agent-Circuit-Id without being tied to a specific MAC - something ISC dhcpd will not do by design.
:) There's a lot more of that in ISC DHCPd. I could go on...
This is supposed to eventually replace PPPoE connections from CPEs. I guess freeradius could someday be able to handle this since the backend data store is really flexible. While most users will probably use the DHCP code with a leases database, in our scenario the configuration would just hand out a static IP calculated from the values in the Agent-Circuit-ID, probably by making use of scripting.
Exactly. A configurable DHCP server is extremely valuable.
Another suggestion with regards to the server design. Currently, the whole server needs to run as root to bind to port 67. That's not necessary for a RADIUS server binding only to ports above 1024. For the long run, would it be feasible to separate the DHCP server code into its own daemon, as a pure DHCP-to-RADIUS translator?
Not really. The longer-term goal is to gradually remove any knowledge of RADIUS from the server core. For now, you can run two servers, one for port 67, and another for port 1812. Or, in 2.1.6, it can drop "root" permissions as soon as it's running, which gives you much more confidence about security. Alan DeKok.
Martin Lorentz wrote:
While playing around with the experimental DHCP code in the latest freeradius releases, I came across a problem where the DHCP packet is not recognized, when the Message-Type is not the first among the list of options. Currently, the code uses a static offset to look for the message type in the packet. A patch is enclosed to fix the problem by iterating through the available options, but will most likely need fixing by someone who knows how to code :-) It applies to 2.0.6 and works in the lab, though.
I've added a similar patch to git head.
Also, I'm interested to hear about the plans for DHCP in freeradius. Personally, I'm evaluating a solution to assign static IPs based on the Agent-Circuit-Id without being tied to a specific MAC - something ISC dhcpd will not do by design.
I'm working on changes to the sqlippool module that will integrate it with DHCP. Once that's done, assigning IP by circuit ID is as simple as editing the schema && queries. Alan DeKok.
On Sun, 2009-06-07 at 09:50 +0200, Alan DeKok wrote:
Martin Lorentz wrote:
While playing around with the experimental DHCP code in the latest freeradius releases
Sorry, I missed this. Why do people see a need for a DHCP server in freeradius? Or can someone point me to the existing discussion? Regards, K. -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Karl Auer (kauer@biplane.com.au) +61-2-64957160 (h) http://www.biplane.com.au/~kauer/ +61-428-957160 (mob) GPG fingerprint: 07F3 1DF9 9D45 8BCD 7DD5 00CE 4A44 6A03 F43A 7DEF
Karl Auer <kauer@biplane.com.au> wrote:
On Sun, 2009-06-07 at 09:50 +0200, Alan DeKok wrote:
Martin Lorentz wrote:
While playing around with the experimental DHCP code in the latest freeradius releases
Sorry, I missed this. Why do people see a need for a DHCP server in freeradius? Or can someone point me to the existing discussion?
I thought exactly the same when I first heard this. Looking out there, you do not get really any options for a DHCP server with LDAP support (which is what I look for). I have been using DHCP with a LDAP patch that is getting harder and harder to maintain. FreeRADIUS can pretty much do the same, I get to keep my LDAP policy schema stuff (and write a unlang glue to use it) and you get proper DHCP load-balancing/failover. The ISC DHCP load-balancing works, but it seems for the failover they took a leaf from MySQL's/Novell's book-of-cluster'ing. In this particular text all nodes of the cluster function identically...to the point where even when one dies *all* of them effectively goes down with the ship :-/ MySQL adds the extra 'functionality' of toasting your data for being a lazy sysadmin and not choosing to use PostgreSQL. So, it's not so much *why* DHCP in a RADIUS server, it's that it looks like the functionality was easy to add (like VMPS) and what other options other than ISC DHCP have you got? Although I'm yet to use it, it seems that it cannot be worse than ISC DHCP, and for me it's got a long term shelf life compared to the LDAP patch I'm maintaining. Cheers -- Alexander Clouter .sigmonster says: But Captain -- the engines can't take this much longer!
On Sun, 2009-06-07 at 12:22 +0100, Alexander Clouter wrote:
I have been using DHCP with a LDAP patch that is getting harder and harder to maintain. FreeRADIUS can pretty much do the same, I get to keep my LDAP policy schema stuff (and write a unlang glue to use it) and you get proper DHCP load-balancing/failover.
DHCP failover and load-balancing are not simple *at all*. I would be very interested to hear how freeradius does it (or plans to do it) hence my interest in the discussion. Are there any docs on how freeradius implements DHCP? And especially how it implements failover? Regards, K. -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Karl Auer (kauer@biplane.com.au) +61-2-64957160 (h) http://www.biplane.com.au/~kauer/ +61-428-957160 (mob) GPG fingerprint: 07F3 1DF9 9D45 8BCD 7DD5 00CE 4A44 6A03 F43A 7DEF
Karl Auer wrote:
On Sun, 2009-06-07 at 12:22 +0100, Alexander Clouter wrote:
I have been using DHCP with a LDAP patch that is getting harder and harder to maintain. FreeRADIUS can pretty much do the same, I get to keep my LDAP policy schema stuff (and write a unlang glue to use it) and you get proper DHCP load-balancing/failover.
DHCP failover and load-balancing are not simple *at all*.
They're trivial once you're storing leases in a transactional database. Arran
On Sun, Jun 7, 2009 at 8:09 PM, Arran Cudbard-Bell<a.cudbard-bell@sussex.ac.uk> wrote:
Karl Auer wrote:
On Sun, 2009-06-07 at 12:22 +0100, Alexander Clouter wrote:
I have been using DHCP with a LDAP patch that is getting harder and harder to maintain. FreeRADIUS can pretty much do the same, I get to keep my LDAP policy schema stuff (and write a unlang glue to use it) and you get proper DHCP load-balancing/failover.
DHCP failover and load-balancing are not simple *at all*.
They're trivial once you're storing leases in a transactional database.
Can freeradius also detect "rogue" clients which uses static IP address? If yes, this could be THE dhcp server I'm looking for. Last I check ISC's DHCP tries ping first, but newer Windows (with icmp echo disabled by default) makes it somewhat less useful. -- Fajar
On Sun, 2009-06-07 at 20:22 +0700, Fajar A. Nugraha wrote:
Last I check ISC's DHCP tries ping first, but newer Windows (with icmp echo disabled by default) makes it somewhat less useful.
No server can detect a "rogue" that is switched off. If it's switched on, your other clients *should* issue DHCPDECLINE responses if they detect the rogue in their subnets. However, it's not exactly reliable. It is more reliable with DHCPv6, because duplicate address detection is a standardised part of the address configuration process. DHCP is not a security tool. It never was and never will be. You have to look beyond DHCP for that. Regards, K. -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Karl Auer (kauer@biplane.com.au) +61-2-64957160 (h) http://www.biplane.com.au/~kauer/ +61-428-957160 (mob) GPG fingerprint: 07F3 1DF9 9D45 8BCD 7DD5 00CE 4A44 6A03 F43A 7DEF
Fajar A. Nugraha wrote:
On Sun, Jun 7, 2009 at 8:09 PM, Arran Cudbard-Bell<a.cudbard-bell@sussex.ac.uk> wrote:
Karl Auer wrote:
On Sun, 2009-06-07 at 12:22 +0100, Alexander Clouter wrote:
I have been using DHCP with a LDAP patch that is getting harder and harder to maintain. FreeRADIUS can pretty much do the same, I get to keep my LDAP policy schema stuff (and write a unlang glue to use it) and you get proper DHCP load-balancing/failover.
DHCP failover and load-balancing are not simple *at all*.
They're trivial once you're storing leases in a transactional database.
Can freeradius also detect "rogue" clients which uses static IP address? If yes, this could be THE dhcp server I'm looking for.
It'd be possible to implement this functionality with the policy language... yes. But it won't do it out of the box. You'd send an ARP-Request for the IP you were allocating, and if you get a response then you'd log it. You can use the exec module to achieve such functionality, or link it in with python/perl.
Last I check ISC's DHCP tries ping first, but newer Windows (with icmp echo disabled by default) makes it somewhat less useful.
I think ARP-Requests would work better than pings in this case; as almost all client side firewalls will allow them to pass. Thanks, Arran Arran
Fajar A. Nugraha <fajar@fajar.net> wrote:
On Sun, Jun 7, 2009 at 8:09 PM, Arran Cudbard-Bell<a.cudbard-bell@sussex.ac.uk> wrote:
Karl Auer wrote:
On Sun, 2009-06-07 at 12:22 +0100, Alexander Clouter wrote:
I have been using DHCP with a LDAP patch that is getting harder and harder to maintain. FreeRADIUS can pretty much do the same, I get to keep my LDAP policy schema stuff (and write a unlang glue to use it) and you get proper DHCP load-balancing/failover.
DHCP failover and load-balancing are not simple *at all*.
They're trivial once you're storing leases in a transactional database.
Can freeradius also detect "rogue" clients which uses static IP address? If yes, this could be THE dhcp server I'm looking for.
As already said, *nothing* can. Instead of arp'ing, which needs to be done on the local subnet, you can be clever with FreeRADIUS as you can use an external script to fire off an SNMP request to much the ARP table's of your switching infrastructure. You could replace the SNMP with ICMP echo's lovin' though.
Last I check ISC's DHCP tries ping first, but newer Windows (with icmp echo disabled by default) makes it somewhat less useful.
The better way to do this is get your network infrastructure to enforce this. Even really old Cisco switches support DHCP snooping, I understand HP and other venduh's have their own similar thing. http://www.cisco.com/web/DK/assets/docs/security2006/Security2006_Eric_Vynck... Cheers -- Alexander Clouter .sigmonster says: Faster, faster, you fool, you fool! -- Bill Cosby
Alexander Clouter wrote:
Fajar A. Nugraha <fajar@fajar.net> wrote:
On Sun, Jun 7, 2009 at 8:09 PM, Arran Cudbard-Bell<a.cudbard-bell@sussex.ac.uk> wrote:
Karl Auer wrote:
On Sun, 2009-06-07 at 12:22 +0100, Alexander Clouter wrote:
I have been using DHCP with a LDAP patch that is getting harder and harder to maintain. FreeRADIUS can pretty much do the same, I get to keep my LDAP policy schema stuff (and write a unlang glue to use it) and you get proper DHCP load-balancing/failover.
DHCP failover and load-balancing are not simple *at all*.
They're trivial once you're storing leases in a transactional database.
Can freeradius also detect "rogue" clients which uses static IP address? If yes, this could be THE dhcp server I'm looking for.
As already said, *nothing* can. Instead of arp'ing, which needs to be done on the local subnet, you can be clever with FreeRADIUS as you can use an external script to fire off an SNMP request to much the ARP table's of your switching infrastructure.
Hmm. Not quire sure what you mean here. I guess if one of your core routers supported it, you could have it send the ARP requests... But yeah you're right. Sending requests from the server wouldn't work for subnets on which the DHCP server was not directly connected.
You could replace the SNMP with ICMP echo's lovin' though.
Yeah just... things block ping. It's not very reliable.
Last I check ISC's DHCP tries ping first, but newer Windows (with icmp echo disabled by default) makes it somewhat less useful.
The better way to do this is get your network infrastructure to enforce this. Even really old Cisco switches support DHCP snooping, I understand HP and other venduh's have their own similar thing.
Yes. We have it enabled most of our smarter L2/3 switches on campus. Once it's combined with dynamic ARP protection or IP lockdown (like it can be on the ProCurve switches), then it makes life quite difficult for those statically assigning IPs. It's hideously broken on the 2600s though, doesn't process lease renewals properly. So ATM it's only good for preventing rogue DHCP servers, and little bits of compliance. Arran
Arran Cudbard-Bell <a.cudbard-bell@sussex.ac.uk> wrote:
The better way to do this is get your network infrastructure to enforce this. Even really old Cisco switches support DHCP snooping, I understand HP and other venduh's have their own similar thing.
Yes. We have it enabled most of our smarter L2/3 switches on campus. Once it's combined with dynamic ARP protection or IP lockdown (like it can be on the ProCurve switches), then it makes life quite difficult for those statically assigning IPs.
It's hideously broken on the 2600s though, doesn't process lease renewals properly. So ATM it's only good for preventing rogue DHCP servers, and little bits of compliance.
Wait till you look at the DHCP snooping on a Cisco WLC 4400. It is so picky about enforcing DHCP, that if the client already has a lease, it cannot ask for a new one[1] until the already assigned one has expired. Cisco's solution for the past year or so, have your leases cracked down to five minutes or less :-/ Cheers [1] say in the *ahem* uncommon *ahem* case that a client moves between AP's or disconnects, reconnects...or hell even reboots their computer -- Alexander Clouter .sigmonster says: Knowledge is power. -- Francis Bacon
Fajar A. Nugraha wrote:
Can freeradius also detect "rogue" clients which uses static IP address? If yes, this could be THE dhcp server I'm looking for.
No. But let's take a look at what happens when you add RADIUS to a bare DHCP implementation: 1) do MAC auth via RADIUS. Who cares what the MAC is. The key is to get them to do RADIUS. 1a) Or, do 802.1X. 2) Return a restrictive set of filter rules (or a VLAN) 3) If they do DHCP, use CoA to update the filter rules to allow packets FROM that source IP. (or switch VLANs) 3a) If they don't do DHCP, too bad for them: they can't pass any traffic.
Last I check ISC's DHCP tries ping first, but newer Windows (with icmp echo disabled by default) makes it somewhat less useful.
Exactly. Why bother testing to see if the network is what you expect? Simply enforce that it *is* what you want. If some idiot types in a static IP, too bad. He hasn't followed network policy, which requires him to do DHCP. The only (minor) problem is that these rules aren't written yet. This is why I'm giving the talk at LinuxTag: to get interest in people writing rules to keep their network secure. Alan DeKok.
Fajar A. Nugraha wrote:
Can freeradius also detect "rogue" clients which uses static IP address? If yes, this could be THE dhcp server I'm looking for.
In fact, it is much easier, when your access hardware (i.e. dslam's or switches) doing this. I mean features such as dhcp snooping and ip guard -- With best regards, Evgeniy Kozhuhovskiy Leader, Services team Minsk State Phone Network, RUE Beltelecom.
On Sun, 2009-06-07 at 14:09 +0100, Arran Cudbard-Bell wrote:
Karl Auer wrote:
DHCP failover and load-balancing are not simple *at all*.
They're trivial once you're storing leases in a transactional database.
With all due respect, Arran, no, they are not. Two DHCP servers in a failover relationship must communicate with each other, each maintaining information about the state of leases that the other has. If they do so via a shared database (which seems to be what you are suggesting, apologies if not) then the entire point of failover is lost. And that is quite apart from the carefully timed state management that must occur during takeover or recovery in the case where a server drops out, is not reachable by its peer or is deliberately taken offline. Not to mention the possibility of having several servers participating in various failover relationships. No - not trivial. Hence my interest in how freeradius will be doing all this. Regards, K. -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Karl Auer (kauer@biplane.com.au) +61-2-64957160 (h) http://www.biplane.com.au/~kauer/ +61-428-957160 (mob) GPG fingerprint: 07F3 1DF9 9D45 8BCD 7DD5 00CE 4A44 6A03 F43A 7DEF
Karl Auer <kauer@biplane.com.au> wrote:
On Sun, 2009-06-07 at 14:09 +0100, Arran Cudbard-Bell wrote:
Karl Auer wrote:
DHCP failover and load-balancing are not simple *at all*.
They're trivial once you're storing leases in a transactional database.
With all due respect, Arran, no, they are not.
Two DHCP servers in a failover relationship must communicate with each other, each maintaining information about the state of leases that the other has. If they do so via a shared database (which seems to be what you are suggesting, apologies if not) then the entire point of failover is lost.
The state lives in the database. ISC's DHCP has it's own 'database' which is just a flat text file...FreeRADIUS just puts everything in SQL. I think Alan's great example is shutdown one of your DHCP servers, scrub the dhcpd.leases file and fire it up again and watch what happens. Also if I see one more damn "peer owns all the leases" message in my logs, I'm likely to cause someone harm :)
And that is quite apart from the carefully timed state management that must occur during takeover or recovery in the case where a server drops out, is not reachable by its peer or is deliberately taken offline. Not to mention the possibility of having several servers participating in various failover relationships.
This is all solved by *load-balancing*. If your load-balancer cannot detect that a DHCP/RADIUS server is dead then you need to get a better load-balancer. The takeover and recovery is just something that helps people not need a load-balancer and spit out packets to the broadcast address of the VLAN both the DHCP servers sit in. We personally use Cisco's IOS (which is pretty naff, but just about usable[1]), but you could use a Linux based HA system. Cheers [1] http://www.digriz.org.uk/cisco-slb-radius -- Alexander Clouter .sigmonster says: Double!
On Sun, 2009-06-07 at 15:18 +0100, Alexander Clouter wrote:
The state lives in the database. ISC's DHCP has it's own 'database' which is just a flat text file...FreeRADIUS just puts everything in SQL.
Where you store the state is irrelevant, though it's easier to manage and more scalable if it's in a database. It's what you do with the state that is important.
This is all solved by *load-balancing*. If your load-balancer cannot detect that a DHCP/RADIUS server is dead then you need to get a better load-balancer.
I think we are talking quite different kinds of load balancing. I'm talking about balancing the leases between the two participants in a failover relationship, you are talking about packet-level load-balancing. My bad.
The takeover and recovery is just something that helps people not need a load-balancer and spit out packets to the broadcast address of the VLAN both the DHCP servers sit in.
What? Either you are talking about real DHCP failover or you are not. If both the DHCP servers in a failover relationship are in the same VLAN, you have a problem right there. Takeover and recovery are the entire point of having failover at all.
We personally use Cisco's IOS (which is pretty naff, but just about usable[1]), but you could use a Linux based HA system.
I think we are talking about different things here. But I'd still like to see whatever docs there may be on how freeradius does failover. Regards, K. -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Karl Auer (kauer@biplane.com.au) +61-2-64957160 (h) http://www.biplane.com.au/~kauer/ +61-428-957160 (mob) GPG fingerprint: 07F3 1DF9 9D45 8BCD 7DD5 00CE 4A44 6A03 F43A 7DEF
Alexander Clouter wrote:
Karl Auer <kauer@biplane.com.au> wrote:
On Sun, 2009-06-07 at 14:09 +0100, Arran Cudbard-Bell wrote:
Karl Auer wrote:
DHCP failover and load-balancing are not simple *at all*.
They're trivial once you're storing leases in a transactional database.
With all due respect, Arran, no, they are not.
Two DHCP servers in a failover relationship must communicate with each other, each maintaining information about the state of leases that the other has. If they do so via a shared database (which seems to be what you are suggesting, apologies if not) then the entire point of failover is lost.
The state lives in the database. ISC's DHCP has it's own 'database' which is just a flat text file...FreeRADIUS just puts everything in SQL.
I think Alan's great example is shutdown one of your DHCP servers, scrub the dhcpd.leases file and fire it up again and watch what happens. Also if I see one more damn "peer owns all the leases" message in my logs, I'm likely to cause someone harm :)
And that is quite apart from the carefully timed state management that must occur during takeover or recovery in the case where a server drops out, is not reachable by its peer or is deliberately taken offline. Not to mention the possibility of having several servers participating in various failover relationships.
This is all solved by *load-balancing*. If your load-balancer cannot detect that a DHCP/RADIUS server is dead then you need to get a better load-balancer.
There are also very intelligent ways to achieve load balancing between peers. One option is to calculate a skew time based on the difference between the leases allocated by the local server and the maximum number of leases allocated by a peer in the cluster. When a discover packet is received, the peer with the lowest number of leases allocated, responds immediately. The others wait skew_time to see if a DHCP Offer is broadcast by one of the other peers. If the local server receives no such offer after skew_time, it sends a DHCP offer. Problems with network latency... and it might not work as the code is currently... but still not incredibly hard to implement. Arran
Karl Auer wrote:
On Sun, 2009-06-07 at 14:09 +0100, Arran Cudbard-Bell wrote:
Karl Auer wrote:
DHCP failover and load-balancing are not simple *at all*.
They're trivial once you're storing leases in a transactional database.
With all due respect, Arran, no, they are not.
Two DHCP servers in a failover relationship must communicate with each other, each maintaining information about the state of leases that the other has. If they do so via a shared database (which seems to be what you are suggesting, apologies if not) then the entire point of failover is lost. Clustered MySQL still has support for transactions.
For purposes of resilience there is absolutely no requirement for DHCP servers to communicate with each other directly. They just need a common source of knowledge about DHCP lease allocation. Additionally there is no need for there to be this active/passive relationship between servers. RFC 2131 explicitly supports multiple DHCP servers operating on the same network, it even says that multiple DHCP servers can offer the same lease (it just makes the process a little less efficient). The key state change for a lease occurs when a client actually requests it. Granted this could be a major headache if both DHCP servers had to maintain local copies of lease information and and synchronise them; but with a resilient central lease database this problem disappears.
And that is quite apart from the carefully timed state management that must occur during takeover or recovery All this goes away. You run multiple servers in active/active mode; there is no takeover or recovery. All DHCP servers act as one, they all have an identical view of the state of the various leases. in the case where a server drops out, is not reachable by its peer or is deliberately taken offline. Not to mention the possibility of having several servers participating in various failover relationships.
*sigh* don't overcomplicate it. One of the good points you made was about load balancing; Any idea how this is done in existing DHCP server solutions? -- The strength of FreeRADIUS as a DHCP server, is that it ties DHCP functionality into a very rich policy environment. Servers like the ISC DHCP server don't integrate well with modern databases or directories, and their configuration syntax is often very baroque and inflexible. For sites moving towards policy driven networks, using FreeRADIUS as a DHCP server allows them to extend what they've already achieved with RADIUS/802.1X in terms of auto-configuration of the edge network, and bring it into the client domain. Arran
On Sun, 2009-06-07 at 17:20 +0100, Arran Cudbard-Bell wrote:
For purposes of resilience there is absolutely no requirement for DHCP servers to communicate with each other directly. They just need a common source of knowledge about DHCP lease allocation.
OK - but lets not talk about "DHCP failover" then, which has very specific meaning in a DHCP context. "Resilience" is a good term. You are basically talking about making sure the DHCP service stays up by using multiple servers accessing a common data source, and then making sure the data source stays available. Fair enough - but it isn't failover. And of course it means you can make the actual DHCP servers much simpler - for a start they don't have to implement failover :-) Probably a very good thing because as I believe I may have mentioned, it isn't trivail to do.
All this goes away. You run multiple servers in active/active mode; there is no takeover or recovery. All DHCP servers act as one, they all have an identical view of the state of the various leases.
Yes indeed. However the supply of that resilient common view requires quite a bit of cooperating hardware and software. You have made the DHCP side of things much easier, but the back end is now more complex. I'm not sure you get to call a resilient, high-availability clustered database "trivial" either :-) One down side is that you need all your DHCP servers to be the same; you lose interoperability. That may or may not be a problem. Another downside might be DHCP response time - how fast can you get addresses out when obtaining one involves a call into the database? Have you tested that side of things yet? (I'm not sure whether you have DHCP servers actually working this way or are talking about how they could or should work).
in the case where a server drops out, is not reachable by its peer or is deliberately taken offline. Not to mention the possibility of having several servers participating in various failover relationships.
*sigh* don't overcomplicate it.
Life complicates things. You have to deal with it. However, with a common data source driving your DHCP, you also don't have to worry about creating meshes of DHCP failover relationships, because failover has disappeared. It's one of the great things about DHCPv6, by the way - no more failover!
One of the good points you made was about load balancing; Any idea how this is done in existing DHCP server solutions?
Er - packet or DHCP-level balancing? We have never needed packet level load balancing; the servers we use have never come remotely close to needing it. I suppose a bigger network might need it, though in that case I'd be wanting a better DHCP server distribution rather than load balancing hardware. For DHCP balancing we've always just distributed the addresses 50/50. Regards, K. -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Karl Auer (kauer@biplane.com.au) +61-2-64957160 (h) http://www.biplane.com.au/~kauer/ +61-428-957160 (mob) GPG fingerprint: 07F3 1DF9 9D45 8BCD 7DD5 00CE 4A44 6A03 F43A 7DEF
Ok i'm going to try and draw this back into a central thread. On 7/6/09 17:57, Karl Auer wrote:
On Sun, 2009-06-07 at 17:20 +0100, Arran Cudbard-Bell wrote:
For purposes of resilience there is absolutely no requirement for DHCP servers to communicate with each other directly. They just need a common source of knowledge about DHCP lease allocation.
OK - but lets not talk about "DHCP failover" then, which has very specific meaning in a DHCP context. "Resilience" is a good term.
Karl; It has a very specific meaning in an ISC DHCP context. A clustered system could replicate fail-over behavior, but really there'd be little advantage in it doing so. Alan; I'd be interested in hearing what the 'corner cases' are where fail-over is advantageous.
You are basically talking about making sure the DHCP service stays up by using multiple servers accessing a common data source, and then making sure the data source stays available. Fair enough - but it isn't failover.
Can't find a definitive term in the OED or Merriam-Webster, but 'dictionary.com' *shudder*, defines it as: "Automatically switching to a different, redundant system upon failure or abnormal termination of the currently active system" There is no reason why this behavior couldn't be replicated on a request by request basis in a clustered environment. You just need to record the fact that a server has made an offer, as well as recording the fact that a lease has been made for an IP address.
And of course it means you can make the actual DHCP servers much simpler - for a start they don't have to implement failover :-) Probably a very good thing because as I believe I may have mentioned, it isn't trivail to do.
All this goes away. You run multiple servers in active/active mode; there is no takeover or recovery. All DHCP servers act as one, they all have an identical view of the state of the various leases.
Yes indeed. However the supply of that resilient common view requires quite a bit of cooperating hardware and software. You have made the DHCP side of things much easier, but the back end is now more complex. I'm not sure you get to call a resilient, high-availability clustered database "trivial" either :-)
It depends how seamless you want to make the fail-over process. You don't have to run truly clustered SQL, you can get away with a Master-> Slaves type arrangement. With MySQL at least, it's possible to nominate one of the replicas as a new master should the current master fail; this process can be automated to a large degree. Fail-back is much harder (apparently), and would be difficult to automate. I think one of the points Alan was making about using an SQL as backend lease storage mechanism, is that the mainstream SQL databases have proven track records when it comes to high availability configurations and scalability. The actual code base is maintained by people who know far more about the concepts behind synchronous replication of data than the people maintaining the ISC code or the other DHCP server implementations. Alex C; A small point about LDAP and DHCP leases. IIRC LDAP doesn't have any kind of 'locking' mechanism for objects/properties, or any way of defining index constraints. This makes it unsuitable for storing DHCP lease information in a clustered (or threaded) environment, as you cannot guarantee at the point of modifying the lease state, that the directory contents has not been modified.
One down side is that you need all your DHCP servers to be the same; you lose interoperability. That may or may not be a problem.
Well no. There's this lovely layer of abstraction between the DHCP server and the database. SQL is actually an ANSI standard, databases may implement extensions to it, but the base language can be used with any SQL compliant server, and/or client. So i'd argue that you mitigate against interoperability issues by using an SQL database as a lease store. It's certainly better defined than the ISC Failover protocol, and with some databases you also have stored procedures which offer an additional layer of abstraction on top. So long as you can modify the SQL queries used by the DHCP server, you should be able to get two servers from completely different vendors using the same database structure.
Another downside might be DHCP response time - how fast can you get addresses out when obtaining one involves a call into the database?
Our current production SQL server comes in at 0.0004 seconds for a SELECT statement against an indexed column containing 13,000 unique IP addresses. Insert statements will vary depending on the number indexes defined. But should be less than 50ms absolute maximum... With MySQL-INNODB you can run multiple concurrent insert and select statements on the same table.
Have you tested that side of things yet? (I'm not sure whether you have DHCP servers actually working this way or are talking about how they could or should work).
It's a little of both. I keep trying to find the time to give the DHCP server a through vetting; but fighting vendor issues has greatly reduced the amount of time I have to spend on fun things.
in the case where a server drops out, is not reachable by its peer or is deliberately taken offline. Not to mention the possibility of having several servers participating in various failover relationships.
*sigh* don't overcomplicate it.
Life complicates things. You have to deal with it.
Complexity has its root in simplicity. Complexity itself isn't a bad thing, sometimes things are necessarily complex. What makes a complex system bad (read unmaintainable) is poor stratification of the (simple) concepts it's built upon.
However, with a common data source driving your DHCP, you also don't have to worry about creating meshes of DHCP failover relationships, because failover has disappeared.
It's one of the great things about DHCPv6, by the way - no more failover!
Is anyone actually using that? What advantages does it have over the stateless auto-configuration protocol? (i've not really done that much reading as regards to IPv6 yet).
One of the good points you made was about load balancing; Any idea how this is done in existing DHCP server solutions?
Er - packet or DHCP-level balancing? We have never needed packet level load balancing; the servers we use have never come remotely close to needing it. I suppose a bigger network might need it,
We have a subnet with ~3000 hosts. After a campus wide power failure, it is conceivable that they'd all be trying to acquire leases at the same time, especially once the distribution layer is UPS backed. This would probably make the DHCP server sad.
though in that case I'd be wanting a better DHCP server distribution rather than load balancing hardware. For DHCP balancing we've always just distributed the addresses 50/50.
Ok. Regards, Arran -- Arran Cudbard-Bell (A.Cudbard-Bell@sussex.ac.uk), Authentication, Authorisation and Accounting Officer, Infrastructure Services (IT Services), E1-1-08, Engineering 1, University Of Sussex, Brighton, BN1 9QT DDI+FAX: +44 1273 873900 | INT: 3900 GPG: 86FF A285 1AA1 EE40 D228 7C2E 71A9 25BB 1E68 54A2
Arran Cudbard-Bell <A.Cudbard-Bell@sussex.ac.uk> wrote:
Alex C; A small point about LDAP and DHCP leases. IIRC LDAP doesn't have any kind of 'locking' mechanism for objects/properties, or any way of defining index constraints. This makes it unsuitable for storing DHCP lease information in a clustered (or threaded) environment, as you cannot guarantee at the point of modifying the lease state, that the directory contents has not been modified.
I'm not storing state in LDAP, I store policy in there. The 'state' (in the case of DHCP) will live in some postgresql database.
However, with a common data source driving your DHCP, you also don't have to worry about creating meshes of DHCP failover relationships, because failover has disappeared.
It's one of the great things about DHCPv6, by the way - no more failover!
Is anyone actually using that? What advantages does it have over the stateless auto-configuration protocol? (i've not really done that much reading as regards to IPv6 yet).
Problem with stateless is that in the long run (for organisations) I doubt people will use it. Why, RFC3041. Second reason, in the IPv6 world it's expected that you have *several* IPv6 addresses (mobile IPv6, local-link, SCTP gets exciting too here). It's going to make it awkward to deal with user accountability when most systems are built around the concept that the user has one IPv4 address...yet alone in addition several IPv6 addresses some of which vary over time. I think that's why a lot of organisations are not keen on stateless IPv6 address assignment but are keener on DHCPv6. I personally would just like an event driven (no SNMP polling...) method that lets me log address<->MAC address usage. I would ask for ideas, but this is all getting hugely OT.
Er - packet or DHCP-level balancing? We have never needed packet level load balancing; the servers we use have never come remotely close to needing it. I suppose a bigger network might need it,
We have a subnet with ~3000 hosts. After a campus wide power failure, it is conceivable that they'd all be trying to acquire leases at the same time, especially once the distribution layer is UPS backed. This would probably make the DHCP server sad.
Apparently it's easy to see around 9->10am at some places DHCP traffic persistantly run at about 50kB/s or more. Cheers -- Alexander Clouter .sigmonster says: Don't hit me!! I'm in the Twilight Zone!!!
Hi,
local-link, SCTP gets exciting too here). It's going to make it awkward to deal with user accountability when most systems are built around the concept that the user has one IPv4 address...yet alone in addition several IPv6 addresses some of which vary over time.
add into that dynamic IPv6 addreses in the link-local domain when systems want to talk to eachother. noone in the rest of the network needs to know those addresses so how/when do they get logged and known about? ;-)
I think that's why a lot of organisations are not keen on stateless IPv6 address assignment but are keener on DHCPv6. I personally would just like an event driven (no SNMP polling...) method that lets me log address<->MAC address usage.
aye. stateless router solicitated addreses just work but then you've got to keep track/log them...and they can often change, nice. DHCPv6 would be okay - ha. if only clients actually supported it!
Apparently it's easy to see around 9->10am at some places DHCP traffic persistantly run at about 50kB/s or more.
..and wonder what that '-' before the filename is for in syslog.conf? ;-) alan
A.L.M.Buxey@lboro.ac.uk wrote:
Hi,
local-link, SCTP gets exciting too here). It's going to make it awkward to deal with user accountability when most systems are built around the concept that the user has one IPv4 address...yet alone in addition several IPv6 addresses some of which vary over time.
add into that dynamic IPv6 addreses in the link-local domain when systems want to talk to eachother. noone in the rest of the network needs to know those addresses so how/when do they get logged and known about? ;-)
It's a tricky one, but you *do* need to record them. We've already had AUP-actionable abuse taking place over link-local mechanisms. Without the logging, we'd have been in a very awkward place.
I think that's why a lot of organisations are not keen on stateless IPv6 address assignment but are keener on DHCPv6. I personally would just like an event driven (no SNMP polling...) method that lets me log address<->MAC address usage.
aye. stateless router solicitated addreses just work but then you've got to keep track/log them...and they can often change, nice. DHCPv6 would be okay - ha. if only clients actually supported it!
If only the RFCs weren't broken...
On Mon, 2009-06-08 at 16:24 +0100, A.L.M.Buxey@lboro.ac.uk wrote:
add into that dynamic IPv6 addreses in the link-local domain when systems want to talk to eachother. noone in the rest of the network needs to know those addresses so how/when do they get logged and known about? ;-)
You don't need to know them - or rather, you already know them. They are derivable from the MAc addresses of the machines.
aye. stateless router solicitated addreses just work but then you've got to keep track/log them...and they can often change, nice. DHCPv6 would be okay - ha. if only clients actually supported it!
Clients took quite a while to support DHCP, remember those days? DHCPv6 is ready to go on MAC OSX and Linux, trivially installable in most Unix variants, and there's a straightforward client (Dibbler) available for Windows. Windows 7 *may* have a client built in. Regards, K. -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Karl Auer (kauer@biplane.com.au) +61-2-64957160 (h) http://www.biplane.com.au/~kauer/ +61-428-957160 (mob) GPG fingerprint: 07F3 1DF9 9D45 8BCD 7DD5 00CE 4A44 6A03 F43A 7DEF
Alexander Clouter wrote:
Is anyone actually using that? What advantages does it have over the stateless auto-configuration protocol? (i've not really done that much reading as regards to IPv6 yet).
Problem with stateless is that in the long run (for organisations) I doubt people will use it. Why, RFC3041. Second reason, in the IPv6 world it's expected that you have *several* IPv6 addresses (mobile IPv6, local-link, SCTP gets exciting too here). It's going to make it awkward to deal with user accountability when most systems are built around the concept that the user has one IPv4 address...yet alone in addition several IPv6 addresses some of which vary over time.
Stateful IPv6 (aka DHCPv6) is not there yet. There are numerous, serious problems with the IPv6 RFCs, in particular the exact semantics of the M/O bits. There is much discussion (some informed, some clueless) in the IETF mailing list archives, if you want to waste a day.
I think that's why a lot of organisations are not keen on stateless IPv6 address assignment but are keener on DHCPv6. I personally would just like an event driven (no SNMP polling...) method that lets me log address<->MAC address usage.
Whilst polling is awkward, event-driven has disadvantages too; specifically, high-rate changes (power outages). FWIW I changed our ARP polling to use an "expect" script against out Cisco 6500s, because the SNMP ipNetToMedia table is sloooowwwwww when it gets >20k entries, but the CLI table is sorted in hardware-order, and is thus fast.
I would ask for ideas, but this is all getting hugely OT.
Ideas for what?
Phil Mayers <p.mayers@imperial.ac.uk> wrote:
I think that's why a lot of organisations are not keen on stateless IPv6 address assignment but are keener on DHCPv6. I personally would just like an event driven (no SNMP polling...) method that lets me log address<->MAC address usage.
Whilst polling is awkward, event-driven has disadvantages too; specifically, high-rate changes (power outages).
We only have ~3000 hosts so we will be fine, plus event driven (for this purpose) is easy enough to handle via buffering. It's not like you need real-time data, just "one minute late" data. I don't know about you, but we usually get our AUP notices days after the event :)
FWIW I changed our ARP polling to use an "expect" script against out Cisco 6500s, because the SNMP ipNetToMedia table is sloooowwwwww when it gets >20k entries, but the CLI table is sorted in hardware-order, and is thus fast.
Worth remembering....don't happen to have a sensible Expect+Net::SSH setup there I can 'borrow'?
I would ask for ideas, but this is all getting hugely OT.
Ideas for what?
To keep this thread interesting, but it's FreeRADIUS content pretty much has reached zero and I'm waiting for someone to grumble :) Cheers -- Alexander Clouter .sigmonster says: Satire is tragedy plus time. -- Lenny Bruce
On Mon, 2009-06-08 at 17:49 +0100, Phil Mayers wrote:
Stateful IPv6 (aka DHCPv6) is not there yet. There are numerous, serious problems with the IPv6 RFCs, in particular the exact semantics of the M/O bits.
I have to disagree. It's "there" sufficiently to be used in the enterprise. Yes, you will have some surprises, but if you KISS for now, you'll have a generally acceptable experience. If you want to mix and match static, stateful and stateless in a single subnet, you'll want to test it first :-) Regards, K. -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Karl Auer (kauer@biplane.com.au) +61-2-64957160 (h) http://www.biplane.com.au/~kauer/ +61-428-957160 (mob) GPG fingerprint: 07F3 1DF9 9D45 8BCD 7DD5 00CE 4A44 6A03 F43A 7DEF
On Mon, 2009-06-08 at 14:57 +0100, Arran Cudbard-Bell wrote:
On 7/6/09 17:57, Karl Auer wrote:
OK - but lets not talk about "DHCP failover" then, which has very specific meaning in a DHCP context. "Resilience" is a good term.
Karl; It has a very specific meaning in an ISC DHCP context.
It's not a good sign that we bicker about terminology. Suffice it to say that "DHCP failover" is not ISC specific, it is implemented by several DHCP servers and has an (admittedly draft) standard behind it, namely draft-ietf-dhc-failover-xx.txt. Might be an RFC by now.
A clustered system could replicate fail-over behavior, but really there'd be little advantage in it doing so.
It could produce a resilient DHCP service. "Little advantage" is putting it very mildly; bending a clustered system to do failover as per the standard would be completely pointless. It could not replicate failover as defined in the draft without doing things like listening on the failover ports and doing direct server-to-server comms. You'd have to be desperate for interoperability.
There is no reason why this behavior couldn't be replicated on a request by request basis in a clustered environment. You just need to record the fact that a server has made an offer, as well as recording the fact that a lease has been made for an IP address.
Again you are not talking about DHCP failover as defined in the standard and implemented in much DHCP software. I think it is really important to distinguish between what you are talking about - which is a Good Thing! - and actual DHCP failover. It confuses the issue otherwise.
configurations and scalability. The actual code base is maintained by people who know far more about the concepts behind synchronous replication of data than the people maintaining the ISC code or the other DHCP server implementations.
I would be cautious about dumping on the skill levels of others. There are a great number of variables in these matters, and by no means all of them are technical. One of the benefits of ISC DHCP, for example, is that it can be deployed extremely easily, on a simple little low-powered server or two, and will then provide largely bullet-proof DHCP service *with failover*. It scales quite nicely from the small network up to a quite large network - obviously it tops out somewhere, and people needing real industrial strength move to something like the Nominum products. But ISC DHCP is working well in tens of thousands of installations of all different sizes right now. If it required a (relatively) complex database back end, it would not serve the needs of many as well as it does.
One down side is that you need all your DHCP servers to be the same; you lose interoperability. That may or may not be a problem.
Well no. There's this lovely layer of abstraction between the DHCP server and the database.
:-) You've missed my point a bit. I mean that if you have (say) an ISC DHCP server in your network, you can't do DHCP failover with it unless your server also speaks DHCP failover. As to using SQL as a lingua franca, I can see a world of pain right there. You would definitely need a standard to hold stuff together. Of course, the standards process is there for all, so go for it :-)
Our current production SQL server comes in at 0.0004 seconds for a SELECT statement against an indexed column containing 13,000 unique IP addresses. Insert statements will vary depending on the number indexes defined. But should be less than 50ms absolute maximum...
So taking 50ms as a worst case (because almost all actions will involve remembering state of some kind), you are looking at 20 DHCP "transactions" per second. Let's be kind and say you get ten times that - 200 per second. That doesn't seem much... see below.
Is anyone actually using [DHCPv6]? What advantages does it have over the stateless auto-configuration protocol? (i've not really done that much reading as regards to IPv6 yet).
I'll put my potted response in a separate email.
We have a subnet with ~3000 hosts. After a campus wide power failure, it is conceivable that they'd all be trying to acquire leases at the same time, especially once the distribution layer is UPS backed. This would probably make the DHCP server sad.
Not to play my network is bigger than your network, but we've had power outages that took out 30,000 clients. If they come back across (say) two minutes, that's an average of 1000 DHCP messages per second (discover, offer, request, ack). We've never had them all go, but we've had about a quarter of them go at once. The Nominum servers dealt with it just fine. Regards, K. -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Karl Auer (kauer@biplane.com.au) +61-2-64957160 (h) http://www.biplane.com.au/~kauer/ +61-428-957160 (mob) GPG fingerprint: 07F3 1DF9 9D45 8BCD 7DD5 00CE 4A44 6A03 F43A 7DEF
Karl Auer wrote:
It's not a good sign that we bicker about terminology. Suffice it to say that "DHCP failover" is not ISC specific, it is implemented by several DHCP servers
That looks to me like fail-over has a well-agreed upon meaning for everything *other* than DHCP.
... It could not replicate failover as defined in the draft ...
Which leads back to my comments on problem versus solution. The fail-over protocol does not work. Full-stop. If you think it works for you, it's because you've never ran into cases where it failed. Or, the implementation you're using doesn't follow the protocol as documented.
One of the benefits of ISC DHCP, for example, is that it can be deployed extremely easily, on a simple little low-powered server or two, and will then provide largely bullet-proof DHCP service *with failover*.
Where "failover" is defined as "sometimes works".
It scales quite nicely from the small network up to a quite large network - obviously it tops out somewhere, and people needing real industrial strength move to something like the Nominum products. But ISC DHCP is working well in tens of thousands of installations of all different sizes right now. If it required a (relatively) complex database back end, it would not serve the needs of many as well as it does.
If you look at the design of FreeRADIUS, you see a better way to do things. FreeRADIUS doesn't *require* a DB. But if you have one, it uses it. ISC DHCP *requires* flat-text files. If you have a DB, it doesn't care. On the other hand, administrators want to manage their systems using information in a DB. ISC either doesn't fit, or they have to write horrible "shim" software to copy data back and forth. Which they do. I'd like them to just run a DHCP server that can talk SQL. Why not?
:-) You've missed my point a bit. I mean that if you have (say) an ISC DHCP server in your network, you can't do DHCP failover with it unless your server also speaks DHCP failover.
That's a Good Thing.
As to using SQL as a lingua franca, I can see a world of pain right there. You would definitely need a standard to hold stuff together. Of course, the standards process is there for all, so go for it :-)
A public SQL schema. That's it.
Not to play my network is bigger than your network, but we've had power outages that took out 30,000 clients. If they come back across (say) two minutes, that's an average of 1000 DHCP messages per second (discover, offer, request, ack). We've never had them all go, but we've had about a quarter of them go at once. The Nominum servers dealt with it just fine.
ISC doesn't deal with that situation very well. At least part of the issue is the delays due to the failover protocol. And I'll get money that Nominum is getting such high performance by doing the kind of optimizations I'm talking about. Alan DeKok.
On Tue, 2009-06-09 at 04:15 +0200, Alan DeKok wrote:
Karl Auer wrote:
It's not a good sign that we bicker about terminology. Suffice it to say that "DHCP failover" is not ISC specific, it is implemented by several DHCP servers
That looks to me like fail-over has a well-agreed upon meaning for everything *other* than DHCP.
Yeah, you're probably right :-) But still.
The fail-over protocol does not work. Full-stop.
Unless you come up with some very clever definition of "does not work", that's just plain wrong, Alan. It clearly *does* work, most of the time for most of the people, and has been doing so in enterprises large and small for many years. It's OK to exaggerate for emphasis, but your statement flies in the face of years of real world experience with ISC DHCP and others. The fact that I haven't had a serious failure in the last eight years or so is a pretty good indicator to me that the protocol is robust *enough*.
If you think it works for you, it's because you've never ran into cases where it failed.
That is true of any protocol you care to name. It's also an unanswerable non-argument. Does inspection of the DHCP failover protocol reveal a theoretical failure mode to you? Or is it that the ISC DHCP implementation that has exhibited failures? One thing is very different to the other.
Or, the implementation you're using doesn't follow the protocol as documented.
That's possible. That never occurred to me, because it is allegedly interoperable with ISC DHCP. I will ask!
Where "failover" is defined as "sometimes works".
It almost always works. It works *by far* most of the time. Even with ISC DHCP. To the point where I have not ever seen it fail except due to bugs in an implementation. My experience is not all-encompassing - perhaps you have seen it fail when the protocol was properly implemented.
If you look at the design of FreeRADIUS, you see a better way to do things. FreeRADIUS doesn't *require* a DB. But if you have one, it uses it.
Yep, no argument there at all. Almost any software is more scalable on a DB than on text files, not to mention easier to maintain, extend, configure, back up, administer....
A public SQL schema. That's it.
There's a bit more to interoperability than merely a public schema. You need a public schema that everyone agrees to use, and that everyone agrees is at least a good schema. That means a standard. This is particularly important for deployment in government and large enterprises. If you are quick and lucky and have a good product, your schema might become a de-facto standard, of course.
ISC doesn't deal with that situation very well. At least part of the issue is the delays due to the failover protocol.
Yes. Or rather, it's delays in the operation of the failover protocol as implemented in ISC DHCP. Or I believe it to be - feel free to educate me otherwise.
And I'll get money that Nominum is getting such high performance by doing the kind of optimizations I'm talking about.
That could be. That is, their failover implementation may not follow the draft standard. However, if they were going to go non-standard, why not develop their own mechanism entirely? But I will ask them about this. Regards, K. -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Karl Auer (kauer@biplane.com.au) +61-2-64957160 (h) http://www.biplane.com.au/~kauer/ +61-428-957160 (mob) GPG fingerprint: 07F3 1DF9 9D45 8BCD 7DD5 00CE 4A44 6A03 F43A 7DEF
Karl Auer wrote:
The fail-over protocol does not work. Full-stop.
Unless you come up with some very clever definition of "does not work", that's just plain wrong, Alan. It clearly *does* work, most of the time for most of the people, and has been doing so in enterprises large and small for many years.
It does something. But it doesn't meet the goal of reliability. My issue is less with the protocol itself than with the belief that it *does* work. My experience with it has been less than positive.
The fact that I haven't had a serious failure in the last eight years or so is a pretty good indicator to me that the protocol is robust *enough*.
You've been lucky. See the RELNOTES that is included with ISC for a series of bug fixes to the protocol. Both the implementation and the protocol design have been changed substantially to avoid issues seen by real-live people in the field.
That is true of any protocol you care to name. It's also an unanswerable non-argument. Does inspection of the DHCP failover protocol reveal a theoretical failure mode to you?
Yes. A few quick tests demonstrated that failure. See earlier messages in this thread.
Or is it that the ISC DHCP implementation that has exhibited failures?
They've *admitted* failure. Publicly. --- FAILOVER: As of version 3.0.4, ISC has included a fix for an insidious bug in the failover implementation which, if left unchecked, could result in tying up all leases in transitional states (such as released, reset, or expired). The crux of the problem is the lack of retransmission of leases that rest in these states. The only way to solve this problem is to carry additional state on the lease data structures to indicate acknowledgement state. --- That doesn't inspire confidence. It's not just a bug, which even FreeRADIUS has had from time to time. The entire design of the protocol has mutated and changed based on discovery of something they missed... YEARS after the protocol was implemented. See also the massive changes in the protocol between 3.0 and 3.1. It's just like the duplicate detection cache implemented in FreeRADIUS nearly 10 years ago, and by myself in other servers before that. Yet it was only with the recent publication of RFC 5080 that some major commercial servers went "Oh, that's a good idea...", and implemented it. Until they did, they were subject to a number of *known* problems.
That's possible. That never occurred to me, because it is allegedly interoperable with ISC DHCP. I will ask!
http://www.tolly.net/ts/2008/Nominum/DHCP2.2/Tolly208319NominumDHCP.pdf They might be inter-operable. The major performance difference between the two proves to me that the protocol between the Nominum servers is *not* the same as the ones used between ISC servers, or between ISC and Nominum. i.e. ISC claims to implement the protocol. If its performance is so much worse than Nomimum, then either (a), ISC didn't implement the protocol as spec'd, or (b) Nominum didn't. And much of the rest of the performance difference is due to ISC *not* using simple algorithms like "dynamic hash tables". It's almost like ISC is *deliberately* bad, to make people go to Nominum. That's OK. It leaves a window of opportunity for me, to create a DHCP server that *isn't* deliberately bad.
It almost always works. It works *by far* most of the time. Even with ISC DHCP. To the point where I have not ever seen it fail except due to bugs in an implementation. My experience is not all-encompassing - perhaps you have seen it fail when the protocol was properly implemented.
Yes.
Yes. Or rather, it's delays in the operation of the failover protocol as implemented in ISC DHCP. Or I believe it to be - feel free to educate me otherwise.
I really don't know. I'm happy to say that both the protocol and the implementation are "less than optimal".
And I'll get money that Nominum is getting such high performance by doing the kind of optimizations I'm talking about.
That could be. That is, their failover implementation may not follow the draft standard. However, if they were going to go non-standard, why not develop their own mechanism entirely? But I will ask them about this.
I'm sure that they developed their own standard for communication between Nominum servers. Alan DeKok.
On Tue, 2009-06-09 at 08:06 +0200, Alan DeKok wrote:
It does something. But it doesn't meet the goal of reliability.
Ah, now that's different. But again, it's reliable *enough*. It does leave a nice big hole for people like Nominum to prodyuce something that is *very* reliable.
You've been lucky.
Perhaps you've just been unlucky? It's just as good an argument.
See the RELNOTES that is included with ISC for a series of bug fixes to the protocol. Both the implementation and the protocol design have been changed substantially to avoid issues seen by real-live people in the field.
Good. That's to be expected and a good thing.
Yes. A few quick tests demonstrated that failure. See earlier messages in this thread.
Nope - "tests" do not show a theoretical failure. Careful argument shows theoretical failure. Tests can only show a failure in an implementation that *may indicate* a theoretical failure. I'd really like to see the discussion of a theoretical failure (i.e., a case where failure must occur if the protocol is implemented correctly). I'm not stating this as some sort of challenge, I genuinely would like to see that discussion.
That doesn't inspire confidence. It's not just a bug, which even FreeRADIUS has had from time to time. The entire design of the protocol has mutated and changed based on discovery of something they missed... YEARS after the protocol was implemented. See also the massive changes in the protocol between 3.0 and 3.1.
Um - that's normal. For any protocol! It's good.
i.e. ISC claims to implement the protocol. If its performance is so much worse than Nomimum, then either (a), ISC didn't implement the protocol as spec'd, or (b) Nominum didn't.
Hm. Or Nominum implemented it better...
I really don't know. I'm happy to say that both the protocol and the implementation are "less than optimal".
Oh, we're in full agreement there.
I'm sure that they developed their own standard for communication between Nominum servers.
Watching it happen suggests very strongly that they are following the standard (such as it is) or something very similar. Whatever: Go for it, and I look forward to the new FreeDHCP server :-) Regards, K. -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Karl Auer (kauer@biplane.com.au) +61-2-64957160 (h) http://www.biplane.com.au/~kauer/ +61-428-957160 (mob) GPG fingerprint: 07F3 1DF9 9D45 8BCD 7DD5 00CE 4A44 6A03 F43A 7DEF
Karl Auer wrote:
Perhaps you've just been unlucky? It's just as good an argument.
Well-designed systems don't require luck to work. That's my argument.
See the RELNOTES that is included with ISC for a series of bug fixes to the protocol. Both the implementation and the protocol design have been changed substantially to avoid issues seen by real-live people in the field.
Good. That's to be expected and a good thing.
Umm.... no. It means they protocol was designed from an incomplete problem statement, and an incomplete knowledge of the system. That isn't good engineering practice.
Yes. A few quick tests demonstrated that failure. See earlier messages in this thread.
Nope - "tests" do not show a theoretical failure.
See earlier messages in this thread. I (a) found a theoretical issue with the protocol, and (b) demonstrated it in a live system. It's a fundamental design flaw. There are designs which are definitely better. Database replication, for one. Yes, it doesn't implement the various states that the ISC protocol uses. However, those states are largely there because of implementation decisions, rather than theoretical analysis. Alan DeKok.
On Tue, 2009-06-09 at 09:24 +0200, Alan DeKok wrote:
Umm.... no. It means they protocol was designed from an incomplete problem statement, and an incomplete knowledge of the system. That isn't good engineering practice.
Maybe - but it's the way a good many, in fact most, of the main protocols we use today have become what they are. People do their best, then the real world comes along and reminds them of all the things they forgot. It's normal for stuff to need fixing. This doesn't mean DHCP failover is a good protocol. There are enough legitimate gripes to throw rocks at.
See earlier messages in this thread. I (a) found a theoretical issue with the protocol, and (b) demonstrated it in a live system.
I missed it. What was it again?
Yes, it doesn't implement the various states that the ISC protocol uses. However, those states are largely there because of implementation decisions, rather than theoretical analysis.
You do need quite a few states for leases, and you need some mechanism for transitioning between those states in an orderly fashion, in a way that does not invalidating the contract you have with your DHCP clients. But these lease states aren't the same states as those used in the DHCP failover protocol. Seems to me you don't need *any* of those, because the servers simply do not have to communicate directly. They "communicate", if at all, through changing state in a shared database. Regards, K. -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Karl Auer (kauer@biplane.com.au) +61-2-64957160 (h) http://www.biplane.com.au/~kauer/ +61-428-957160 (mob) GPG fingerprint: 07F3 1DF9 9D45 8BCD 7DD5 00CE 4A44 6A03 F43A 7DEF
On 9/6/09 13:58, Karl Auer wrote:
On Tue, 2009-06-09 at 09:24 +0200, Alan DeKok wrote:
Umm.... no. It means they protocol was designed from an incomplete problem statement, and an incomplete knowledge of the system. That isn't good engineering practice.
Maybe - but it's the way a good many, in fact most, of the main protocols we use today have become what they are. People do their best, then the real world comes along and reminds them of all the things they forgot. It's normal for stuff to need fixing.
This doesn't mean DHCP failover is a good protocol. There are enough legitimate gripes to throw rocks at.
See earlier messages in this thread. I (a) found a theoretical issue with the protocol, and (b) demonstrated it in a live system.
I missed it. What was it again?
When we tried it back in 2007 with an Active/Active configuration, the two instances of ISC DHCPD started handing out duplicate leases completely arbitrarily. We scrapped the second instance and went down to a single one. Haven't tried it again since. It didn't work then... it may do now. Arran -- Arran Cudbard-Bell (A.Cudbard-Bell@sussex.ac.uk), Authentication, Authorisation and Accounting Officer, Infrastructure Services (IT Services), E1-1-08, Engineering 1, University Of Sussex, Brighton, BN1 9QT DDI+FAX: +44 1273 873900 | INT: 3900 GPG: 86FF A285 1AA1 EE40 D228 7C2E 71A9 25BB 1E68 54A2
On Tue, 2009-06-09 at 14:07 +0100, Arran Cudbard-Bell wrote:
See earlier messages in this thread. I (a) found a theoretical issue with the protocol, and (b) demonstrated it in a live system.
I missed it. What was it again?
When we tried it back in 2007 with an Active/Active configuration, the two instances of ISC DHCPD started handing out duplicate leases completely arbitrarily. We scrapped the second instance and went down to a single one. Haven't tried it again since.
Thanks - but that's not a theoretical problem (necessarily). I'm interested in the protocol itself; Alan has been talking about an error in the protocol which would lead to failure *even if the protocol were implemented correctly*. What does "Active/Active" mean? Presumably not the same as "primary/primary", which would be a configuration error... Regards, K. -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Karl Auer (kauer@biplane.com.au) +61-2-64957160 (h) http://www.biplane.com.au/~kauer/ +61-428-957160 (mob) GPG fingerprint: 07F3 1DF9 9D45 8BCD 7DD5 00CE 4A44 6A03 F43A 7DEF
On 9/6/09 14:20, Karl Auer wrote:
On Tue, 2009-06-09 at 14:07 +0100, Arran Cudbard-Bell wrote:
See earlier messages in this thread. I (a) found a theoretical issue with the protocol, and (b) demonstrated it in a live system.
I missed it. What was it again? When we tried it back in 2007 with an Active/Active configuration, the two instances of ISC DHCPD started handing out duplicate leases completely arbitrarily. We scrapped the second instance and went down to a single one. Haven't tried it again since.
Thanks - but that's not a theoretical problem (necessarily). I'm interested in the protocol itself; Alan has been talking about an error in the protocol which would lead to failure *even if the protocol were implemented correctly*.
What does "Active/Active" mean? Presumably not the same as "primary/primary", which would be a configuration error...
With a pair of servers running Active/Active means that both servers participate at the same time. In ISC terms this would be 'load sharing'. Active/Passive generally refers to some kind of redundancy arrangement. Regards, Arran -- Arran Cudbard-Bell (A.Cudbard-Bell@sussex.ac.uk), Authentication, Authorisation and Accounting Officer, Infrastructure Services (IT Services), E1-1-08, Engineering 1, University Of Sussex, Brighton, BN1 9QT DDI+FAX: +44 1273 873900 | INT: 3900 GPG: 86FF A285 1AA1 EE40 D228 7C2E 71A9 25BB 1E68 54A2
Hi,
When we tried it back in 2007 with an Active/Active configuration, the two instances of ISC DHCPD started handing out duplicate leases completely arbitrarily. We scrapped the second instance and went down to a single one. Haven't tried it again since.
It didn't work then... it may do now.
we run in an active/standby home-made configuration. the lease file is copied periodically from the live server to the standby (heck, we may lose some details if there is a problem but each lease file revision is logged) and then a process on the standby checks whether dhcpd is alive on the master. if it isnt, is starts dhcpd process ..if it detects the master is alive again it kills its process. why? when we tried any of the active/active and stanbdy methods in ISC DHCPD at the time it b0rked in many weird ways. this works. havent had time to get the 4.1.x runway time yet...been really waiting to get around to FR DHCP :-) alan
Karl Auer wrote:
Maybe - but it's the way a good many, in fact most, of the main protocols we use today have become what they are. People do their best, then the real world comes along and reminds them of all the things they forgot. It's normal for stuff to need fixing.
That's nice. Except that database replication was already a solved problem when the protocol was designed.
See earlier messages in this thread. I (a) found a theoretical issue with the protocol, and (b) demonstrated it in a live system.
I missed it. What was it again?
It doesn't have transaction numbers. Parts of the request/ack protocol is missing anything *other* than request/ack. It can't say "get me all leases since time T". It can't say "get me the leases since I last synced". What I did was to configure a primary and secondary. Let them sync. Then, take down the secondary, and delete it's lease database. When the secondary comes back up, the fail-over protocol does this: S: Send me leases P: I did! S: OK. And the secondary is quite happy to sit there with *zero* leases. It's really mind-boggling. Maybe they've fixed it in more recent versions, but it's still a catastrophic design error. A real replication protocol using techniques known since at least 1990 is: S: send me transactions since time 0 P: Hmm... I recall sending you transactions until time T, but OK... P: <here's all the leases from time 0..T'> S: OK. I'm synced at time T' P: Thanks, I'll remember that ... S: Can you send me updates since time T'? P: OK, here they are
You do need quite a few states for leases, and you need some mechanism for transitioning between those states in an orderly fashion, in a way that does not invalidating the contract you have with your DHCP clients.
Yes. So long as both servers can share the same view of what the client should be doing, they will work together seamlessly. Note that this does *not* mean that they share *all* information before responding to the client. Replication can be lazy in many, many, cases.
But these lease states aren't the same states as those used in the DHCP failover protocol. Seems to me you don't need *any* of those, because the servers simply do not have to communicate directly. They "communicate", if at all, through changing state in a shared database.
Yes. Alan DeKok.
Hi,
It's not a good sign that we bicker about terminology. Suffice it to say
whilst it was interesting that FreeRADIUS got DHCP support - certainly for those that want to ensure policy actually works - I never thought we'd get to have such fervent discussion about it :-) now, historical context - a lot of questions have been asked about eg freeRADIUS in 802.1X along the lines of 'authentication worked but client didnt get an address' - of course, FR never handed out addresses back then - it can now. however, the main question is when will FreeRADIUS have to have a new name? 8-) FRANAS ? (free radius, authentication and network access server) 8-) alan
A.L.M.Buxey@lboro.ac.uk wrote:
Hi,
It's not a good sign that we bicker about terminology. Suffice it to say
whilst it was interesting that FreeRADIUS got DHCP support - certainly for those that want to ensure policy actually works - I never thought we'd get to have such fervent discussion about it :-)
now, historical context - a lot of questions have been asked about eg freeRADIUS in 802.1X along the lines of 'authentication worked but client didnt get an address' - of course, FR never handed out addresses back then - it can now.
however, the main question is when will FreeRADIUS have to have a new name? 8-)
FRANAS ? (free radius, authentication and network access server) 8-)
You know I was thinking about this as well, it's not really a Network Access Server though (EAP terminology). It's hard... How many protocols does FR support now, VMPS, RADIUS (With WiMAX), EAP, DHCP. I'd suggest FreeNAC but I believe someone has already :P. FreeRNC - Free RADIUS and Network Control or FreeRaNCS (FRANCS) Arran
Arran Cudbard-Bell wrote:
It's hard... How many protocols does FR support now, VMPS, RADIUS (With WiMAX), EAP, DHCP. I'd suggest FreeNAC but I believe someone has already :P.
ARP? Tacacs+ maybe to come.
FreeRNC - Free RADIUS and Network Control
or
FreeRaNCS (FRANCS)
I'd avoid the word "free". Alan DeKok.
FreeRadius should always be called FreeRadius. It's almost a brand name, which is very well known, and has an excellent reputation. I agree the new stuff in FreeRadius needs way more attention. Virtually no one i know associates Free_Radius_ with VMPS, nor DHCP.
I'd avoid the word "free".
I'd agree. (Open- is taken too. What about Flex-"Something".) How could this be organized, in the source tree or elsewhere? A new server core maybe called Flex-"Something" without any knowledge of RADIUS, DHCP or the like? An "add-on" package called FreeRadius, another called FlexDHCP? Kind of confusing, there must be a better way. Best regards, Martin Lorentz
On 9/6/09 11:07, Martin Lorentz wrote:
FreeRadius should always be called FreeRadius. It's almost a brand name, which is very well known, and has an excellent reputation.
I agree the new stuff in FreeRadius needs way more attention. Virtually no one i know associates Free_Radius_ with VMPS, nor DHCP.
I'd avoid the word "free".
I'd agree. (Open- is taken too. What about Flex-"Something".)
How could this be organized, in the source tree or elsewhere? A new server core maybe called Flex-"Something" without any knowledge of RADIUS, DHCP or the like? An "add-on" package called FreeRadius, another called FlexDHCP? Kind of confusing, there must be a better way.
Whenever I think of Flex, I think of Flex-LM... and that's not a good association :( I know it's all 1980sish but we could bring back Synergy. "syn⋅er⋅gism –noun 1. the interaction of elements that when combined produce a total effect that is greater than the sum of the individual elements, contributions, etc." It actually fits what FreeRADIUS is moving towards; lots of protocols working together to provide control of a network. SyNC - Synergous/Synchronous Network Control, also reads as (Sync) SyNAC - Synergous/Synchronous Network Access Control Arran -- Arran Cudbard-Bell (A.Cudbard-Bell@sussex.ac.uk), Authentication, Authorisation and Accounting Officer, Infrastructure Services (IT Services), E1-1-08, Engineering 1, University Of Sussex, Brighton, BN1 9QT DDI+FAX: +44 1273 873900 | INT: 3900 GPG: 86FF A285 1AA1 EE40 D228 7C2E 71A9 25BB 1E68 54A2
Martin Lorentz wrote:
FreeRadius should always be called FreeRadius. It's almost a brand name, which is very well known, and has an excellent reputation.
I agree.
How could this be organized, in the source tree or elsewhere? A new server core maybe called Flex-"Something" without any knowledge of RADIUS, DHCP or the like? An "add-on" package called FreeRadius, another called FlexDHCP? Kind of confusing, there must be a better way.
I think the best way is to have a new project that contains additional documentation on the web explaining how to configure the additional protocols. We could also have a git tree based on FreeRADIUS, but with the RADIUS parts stripped out, for example. I've started some of that work in the git "master" branch. The various "listen" handlers are now in plug-in modules. The server core is gradually having all of the RADIUS portions removed. If I can figure out how to move the RADIUS state machine into the modules, then the core would be *completely* protocol agnostic. In the long term, migrating the FreeRADIUS code base into something more flexible is a good idea. Alan DeKok.
How about FreeRADIUS and FreeDHCP? The "FreeXXXXX" Project --------------------------------------------- | | | | | | FreeRADIUS | FreeDHCP | FreeTBD | ... | | Server | Server | Server | | --------------------------------------------- | | | "Magic Server Code" | | | --------------------------------------------- FreeRADIUS migrates to the "FreeXXXXX" Project which includes the "Magic Server Code", the FreeRADIUS Server, FreeDHCP Server, etc. There would be distributions for FreeRADIUS, FreeDHCP and one that included everything. Tim
-----Original Message----- From: freeradius-users- bounces+tim.sylvester=networkradius.com@lists.freeradius.org [mailto:freeradius-users- bounces+tim.sylvester=networkradius.com@lists.freeradius.org] On Behalf Of Alan DeKok Sent: Thursday, June 11, 2009 1:09 AM To: FreeRadius users mailing list Subject: Re: New name to reflect new functionality (was RE: DHCP code in 2.0.4+)
Martin Lorentz wrote:
FreeRadius should always be called FreeRadius. It's almost a brand name, which is very well known, and has an excellent reputation.
I agree.
How could this be organized, in the source tree or elsewhere? A new server core maybe called Flex-"Something" without any knowledge of RADIUS, DHCP or the like? An "add-on" package called FreeRadius, another called FlexDHCP? Kind of confusing, there must be a better way.
I think the best way is to have a new project that contains additional documentation on the web explaining how to configure the additional protocols. We could also have a git tree based on FreeRADIUS, but with the RADIUS parts stripped out, for example.
I've started some of that work in the git "master" branch. The various "listen" handlers are now in plug-in modules. The server core is gradually having all of the RADIUS portions removed. If I can figure out how to move the RADIUS state machine into the modules, then the core would be *completely* protocol agnostic.
In the long term, migrating the FreeRADIUS code base into something more flexible is a good idea.
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
A.L.M.Buxey@lboro.ac.uk wrote:
however, the main question is when will FreeRADIUS have to have a new name? 8-)
It won't. The name is already well known.
FRANAS ? (free radius, authentication and network access server) 8-)
I'm all for having an *additional* name, web site, etc. That will help draw new people in without affecting the current mindshare. The problem is picking a name that is nice, but isn't already registered. Alan DeKok.
On Mon, 2009-06-08 at 14:57 +0100, Arran Cudbard-Bell wrote:
Is anyone actually using [DHCPv6]? What advantages does it have over the stateless auto-configuration protocol?
This question gets asked a lot. Here's my potted response. Regards, K. Why is DHCP still important with IPv6? IPv6 has autoconfiguration - it can obtain an address fully automatically, given only a router advertising a prefix. Why then is DHCP still important? Here are some of the reasons. * DHCP can pass out an address in the *absence* of router advertisements. * DHCP can pass out nameserver information – search list, default domain, nameserver addresses. That doesn't happen with ordinary autoconf. * DHCP can do dynamic DNS. That is, it can register forward and reverse DNS entries for the addresses it gives out. While you could allow the individual clients to do this, it is generally better to have a few centralised servers making updates than to have thousands of clients doing it. Apart from anything else, it is easier to secure the updates. * DHCP lets you work with subnets that are not /64. There is a bit of a debate going on over whether we should or should not be a bit less profligate with v6 addresses; 18 billion billion does seem a lot for just one subnet. With DHCP you can allocate addresses in a subnet of any desired size, larger or smaller than /64. * DHCP lets you control what addresses are allocated – you can ensure, for example, that a particular machine always gets a particular address, or that allocated addresses always come from some particular range. * DHCP provides a natural hooking point for additional control (the server software has to support that though). * DHCP provides a natural logging point – you can see when clients come and go, what addresses they had from time to time, and so on. With modern switches you can get a lot of that off the ports via SNMP, but DHCP can centralise it. * DHCP allocates addresses that do not (necessarily) contain the MAC address of the client. Autoconfigured addresses contain the MAc address of the host, and some people have an issue with displaying, globally, the MAC address of a host. They see it as "leaking" local information onto the global Internet. Temporary addresses are also a solution to this. * DHCP provides a mechanism to delegate prefixes (hand out prefixes rather than individual addresses). All the above is DHCPv6 by the way. Most of it applies to DHCPv4 too. "If you have DHCP, would that not break the radvd/autoconfig stuff?" No, not at all. Autoconfig and DHCP play very well together, and you can mix and match. You can obtain your address by autoconf or from a DHCP server, or obtain one (or more!) from *both*. You can get the ancillary information like nameserver addresses for your autoconf address, or delivered with a DHCP address. You can statically configure an address and then go get the nameserver stuff from the DHCP server. And so on. For a very small network, where ancillary information like domains and nameservers can be manually configured and where a /64 is natural, you don't generally need DHCP. In networks where everything is dual stacked and you only need v4 nameserver info, you can get nameserver info via DHCPv4 and do without DHCPv6 (assuming none of the otherreasons for using it apply). In large networks of autonomous devices (like sensors, maybe) you probably don't need DHCP. In enterprise-style networks you probably do need DHCP. DHCPv4 is pretty straightforward; every OS has it in there by default. This is not the case for IPv6 yet. However, there are at least two very good DHCPv6 clients, WIDE and Dibbler. WIDE does the job for Unix, Dibbler works on Unix and Windows. Both good, both effective, both open and free, both easy to install. Dibbler is somewhat easier to configure. -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Karl Auer (kauer@biplane.com.au) +61-2-64957160 (h) http://www.biplane.com.au/~kauer/ +61-428-957160 (mob) GPG fingerprint: 07F3 1DF9 9D45 8BCD 7DD5 00CE 4A44 6A03 F43A 7DEF
Arran Cudbard-Bell wrote:
Alan; I'd be interested in hearing what the 'corner cases' are where fail-over is advantageous.
When you're too lazy to write your own solution, *or* where you've never run into problems with the protocol.
It depends how seamless you want to make the fail-over process. You don't have to run truly clustered SQL, you can get away with a Master-> Slaves type arrangement. With MySQL at least, it's possible to nominate one of the replicas as a new master should the current master fail; this process can be automated to a large degree. Fail-back is much harder (apparently), and would be difficult to automate.
Having active/active servers assign leases "live" from the same pool would be nice, too. That's only slightly harder to do. Alan DeKok.
Arran Cudbard-Bell wrote:
The key state change for a lease occurs when a client actually requests it. Granted this could be a major headache if both DHCP servers had to maintain local copies of lease information and and synchronise them; but with a resilient central lease database this problem disappears.
There's a lot of magic that happens behind the scenes with DHCP servers using the failover protocol. They maintain a *very* complex state machine where each lease has a large number of states and parameters associated with it. My time spent in the area convinced me that the complexity is unnecessary, poorly understood by everyone, and fragile.
And that is quite apart from the carefully timed state management that must occur during takeover or recovery All this goes away. You run multiple servers in active/active mode; there is no takeover or recovery. All DHCP servers act as one, they all have an identical view of the state of the various leases.
Karl is talking about the servers exchanging the state of the lease during the internal state changes that they undergo as they allocate the lease. Most DHCP people think it's necessary. I agree it's necessary in certain edge conditions. But also that the *normal* conditions don't need it. And it's so fragile that it causes more problems than it solves.
One of the good points you made was about load balancing; Any idea how this is done in existing DHCP server solutions?
Badly. MAC addresses are hashed (via a bad hash), and then mapped to unallocated IP's in a pool. If that IP is "owned" by the other server in a failover relationship, even *more* complexity results. The server has to ask the other server for more leases. If it's slow (or down), you get the dreaded "peer holds all leases" message. EVEN WHEN THERE ARE TONS OF FREE LEASES. Bizarre... I can't understand why it would be done that way, when other ways are simpler, more robust, and achieve the same goal. But that brings us back to the DHCP world view: The problem is defined to be "implement the failover protocol", and the solution is "the failover protocol". IMHO, admins don't care about it. They care about having a robust DHCP infrastructure with load-balancing and HA. That's relatively easy. And it's surprisingly easy to the people used to working with existing DHCP systems.
The strength of FreeRADIUS as a DHCP server, is that it ties DHCP functionality into a very rich policy environment. Servers like the ISC DHCP server don't integrate well with modern databases or directories, and their configuration syntax is often very baroque and inflexible.
Exactly.
For sites moving towards policy driven networks, using FreeRADIUS as a DHCP server allows them to extend what they've already achieved with RADIUS/802.1X in terms of auto-configuration of the edge network, and bring it into the client domain.
That's the goal. Alan DeKok.
Karl Auer wrote:
They're trivial once you're storing leases in a transactional database.
With all due respect, Arran, no, they are not.
Two DHCP servers in a failover relationship must communicate with each other, each maintaining information about the state of leases that the other has.
There's a miscommunication here. Arran is talking about two servers sharing a common view of their leases, and synchronizing that view with each other, to maintain a consistent response to client machines. You're talking about the ISC fail-over protocol. The two views just don't meet.
If they do so via a shared database (which seems to be what you are suggesting, apologies if not) then the entire point of failover is lost. And that is quite apart from the carefully timed state management that must occur during takeover or recovery in the case where a server drops out, is not reachable by its peer or is deliberately taken offline. Not to mention the possibility of having several servers participating in various failover relationships.
I disagree. Really. I spent most of a year working with DHCP in a previous life. Your comment is "correct", where the word "correct" has connotations of "theoretical analysis that conveniently ignores real-world situations". Like losing the entire d*mned lease database. Or working in settings like (1) enterprises, where 99.99% of lease renewals are for the same MAC/IP setting, or broadband/dial-up ISP's, who have interest in *not* handing out the same IP to the same MAC for any period of time,
No - not trivial.
90% of the situations are trivial to handle. The other 10% can go buy a $500,000 solution. The people selling the $500K solution are welcome to the support nightmare.
Hence my interest in how freeradius will be doing all this.
It will do it *very* well, or not at all. Alan DeKok.
This just in, from the horse's mouth (Nominum). While the failover protocol used in DCS is based on the standard draft, we have made some modifications where we deemed it necessary, and both for this and for performance reasons we do not claim interoperability with other implementations, nor do we support DCS in a failover configuration unless both peers are DCS. Our failover implementation is proven to work and it's better :-) Regards, K. -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Karl Auer (kauer@biplane.com.au) +61-2-64957160 (h) http://www.biplane.com.au/~kauer/ +61-428-957160 (mob) GPG fingerprint: 07F3 1DF9 9D45 8BCD 7DD5 00CE 4A44 6A03 F43A 7DEF
I used to work in product marketing. Let me translate this message.
This just in, from the horse's mouth (Nominum).
While the failover protocol used in DCS is based on the standard draft, we have made some modifications where we deemed it necessary,
"It was necessary to make modifications to the draft standard failover protocol because it doesn't work ..."
and both for this and for performance reasons we do not claim interoperability with other implementations,
"... and none of the implementations of the draft standard failover protocol interoperate."
nor do we support DCS in a failover configuration unless both peers are DCS.
"Our engineering, QA and tech support people refuse to support using the failover protocol between our DHCP server and the ISC DHCP server because the draft standard failover protocol and the ISC DHCP server implementation are broken."
Our failover implementation is proven to work and it's better :-)
"We fixed the problems with the draft standard DHCP failover protocol that is implemented in the ISC DHCP server." Tim
Karl Auer wrote:
DHCP failover and load-balancing are not simple *at all*.
As evidenced by the fact that the ISC fail-over protocol is horrible, and the implementation is almost as bad. Scratch that.. it's *terrible*. After toasting the leases once accidentally, I managed to prove to myself that this was a design feature. Let's say that the primary and secondary have the same configuration, and have synchronized leases. Stop the secondary, delete *all* leases, and bring it back up again. You can get into a state where the fail-over protocol does this: S: Send me the leases P: I did already! We're in sync! S: OK And the secondary has *zero* leases, and therefore wastes CPU cycles never handing out leases. WTF? I mean.. really. Is it that hard? Oh, and the server is O(N^2) in the number of leases. Why? Well... they don't use fancy concepts like "dynamically resizable hash tables". Fixed size hash tables were good enough in 1995, so they're good enough now, right? About 4 years ago I had a series of 200-400 line patches that would dramatically improve the performance of ISC. I got told that (1) it's impossible, and (2) if it was possible, it would require a drastic re-design of the server. When I told them that the patches were proven to work, and quoted *their own* code back at them showing where the patches could go, there was... nothing. They weren't malicious, they just had different priorities.
I would be very interested to hear how freeradius does it (or plans to do it) hence my interest in the discussion. Are there any docs on how freeradius implements DHCP? And especially how it implements failover?
There are little to no documentation on how it does DHCP. And it doesn't do fail-over. Why? Fail-over is hard. My experience is that the fail-over protocol doesn't help. From what I recall the last time I looked at it, it was missing key things, like "transaction numbers". Hence the failure case noted above. The *correct* conversation should have been: S: I'm at transaction #0: I have no leases! P: Geez.. my last recollection is that you were at 1000. Let me send you all of the updates from 0 to where we are now: 1010. S: Thanks! It's really not that hard. Database books describe replication protocols. They look very different from the DHCP fail-over protocol. So... for now, DHCP in FreeRADIUS is still experimental. If you want to use it, see raddb/sites-available/dhcp. DHCP fail-over will be supported when there's a fail-over protocol that *works*. And for most enterprise sites, you *don't* need a fail-over protocol. Really. Alan DeKok.
On Sun, 2009-06-07 at 22:25 +0200, Alan DeKok wrote:
Karl Auer wrote:
DHCP failover and load-balancing are not simple *at all*.
As evidenced by the fact that the ISC fail-over protocol is horrible, and the implementation is almost as bad.
Oh, I'm 100% with you on that.
Oh, and the server is O(N^2) in the number of leases. Why? Well... they don't use fancy concepts like "dynamically resizable hash tables".
True, but that's ISC's implementation. I use Nominum's DCS, which starts in seconds regardless of the number of leases, largely because it has a database backend (but not distributed or HA - highspeed, local and dedicated) and which is completely dynamically configurable. Nothing short of upgrading the server requires it to be stopped. And with one major exception (which was a *bug*), failover as implemented by Nominum has never, ever let us down. Let's not confuse the protocol with the implementation.
It's really not that hard. Database books describe replication protocols. They look very different from the DHCP fail-over protocol.
Though to be fair, database replication is *not* solving the same problem as failover is. Failover allows independent entities to maintain the same view of the data, and a view with severe constraints. The real mistakes were made defining DHCP itself, but for the time they did an OK job.
And for most enterprise sites, you *don't* need a fail-over protocol. Really.
Well... if they have a HA arrangement for DHCP such as that described here recently, maybe not. Otherwise we'll have to disagree on that. Regards, K. -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Karl Auer (kauer@biplane.com.au) +61-2-64957160 (h) http://www.biplane.com.au/~kauer/ +61-428-957160 (mob) GPG fingerprint: 07F3 1DF9 9D45 8BCD 7DD5 00CE 4A44 6A03 F43A 7DEF
Karl Auer wrote:
It's really not that hard. Database books describe replication protocols. They look very different from the DHCP fail-over protocol.
Though to be fair, database replication is *not* solving the same problem as failover is.
I'll have to disagree with you on that. There's a lot of database research on consistency. The DHCP failover protocol is an attempt to re-invent much of that, without using research that was well known at the time it was created. I just can't see how the failover protocol is *so* special that it falls into a class all by itself. It's just data replication, with certain constraints. Big whoop...
And for most enterprise sites, you *don't* need a fail-over protocol. Really.
Well... if they have a HA arrangement for DHCP such as that described here recently, maybe not. Otherwise we'll have to disagree on that.
Exactly. HA for DHCP is needed. Lazy replication is pretty much all you need when new machines are rare. And that's a *lot* easier to implement than the failover protocol. I don't think there's need for FreeRADIUS to implement a protocol that's broken *by design*. If failover works for you, it's likely because Nominum is using some other protocol than the one that ISC is using. Alan DeKok.
participants (10)
-
A.L.M.Buxey＠lboro.ac.uk -
Alan DeKok -
Alexander Clouter -
Arran Cudbard-Bell -
Evgeniy Kozhuhovskiy -
Fajar A. Nugraha -
Karl Auer -
Martin Lorentz -
Phil Mayers -
Tim Sylvester