I'm running FreeRadius 2.1.3 on RedHat Enterprise Linux configured as an Eduroam Radius proxy server. My Cisco Wireless Lan Controllers are constantly failing over the Accounting Servers, due to lack of response from the Home Servers, or so says the log. However, I believe the issue is that some remote institutions Radius Servers are ignoring the Accounting packets, and timing out my end, making it believe the Home Servers have failed to respond. FreeRadius responds by marking the Home server dead. It then sends a status-server query, to which is gets a reply, and enables the Dead Home server. I believe that's the sequence of events anyway. I captured some of that in debug mode: Rejecting request 288 due to lack of any response from home server x.x.x.x port 1813 Finished request 288. Cleaning up request 288 ID 205 with timestamp +1161 PROXY: Marking home server x.x.x.x port 1813 as zombie (it looks like it is dead). Sending Status-Server of id 55 to x.x.x.x port 1813 Message-Authenticator := 0x00000000000000000000000000000000 NAS-Identifier := "Status Check. Are you alive?" Waking up in 3.9 seconds. rad_recv: Access-Accept packet from host x.x.x.x port 1813, id=55, length=806 I don't have any control over Accounting Packets being accepted, or not, by other Eduroam members. Some do, some don't I imagine. Is there a configuration for FreeRadius that handles this situation cleanly? Seems to me that FR should check the Home server first, before marking it dead (at least). -Mike
On 5 Oct 2011, at 02:20, Mike Diggins wrote:
I'm running FreeRadius 2.1.3 on RedHat Enterprise Linux configured as an Eduroam Radius proxy server. My Cisco Wireless Lan Controllers are constantly failing over the Accounting Servers, due to lack of response from the Home Servers, or so says the log. However, I believe the issue is that some remote institutions Radius Servers are ignoring the Accounting packets, and timing out my end, making it believe the Home Servers have failed to respond. FreeRadius responds by marking the Home server dead. It then sends a status-server query, to which is gets a reply, and enables the Dead Home server. I believe that's the sequence of events anyway. I captured some of that in debug mode:
Rejecting request 288 due to lack of any response from home server x.x.x.x port 1813
Finished request 288.
Cleaning up request 288 ID 205 with timestamp +1161
PROXY: Marking home server x.x.x.x port 1813 as zombie (it looks like it is dead).
Sending Status-Server of id 55 to x.x.x.x port 1813 Message-Authenticator := 0x00000000000000000000000000000000 NAS-Identifier := "Status Check. Are you alive?" Waking up in 3.9 seconds.
rad_recv: Access-Accept packet from host x.x.x.x port 1813, id=55, length=806
I don't have any control over Accounting Packets being accepted, or not, by other Eduroam members. Some do, some don't I imagine. Is there a configuration for FreeRadius that handles this situation cleanly? Seems to me that FR should check the Home server first, before marking it dead (at least).
No i've talked about this with Alan. The proposed solution is an enhancement to the detail writer/reader functionality where the server automagically spawns new reader instances to listen on per domain detail files. For now i'd recommend using rlm_replicate, this allows you to send an entirely new request to the homeserver, whilst sending an Accounting-Response to the NAS. FR does not track the new request and so won't be able to retransmit. Just use something like: preacct { if(Realm && Realm != 'local'){ update control { Proxy-To-Realm := 'local' Replicate-To-Realm = 'eduroam' } } } accounting { replicate } Arran Cudbard-Bell a.cudbardb@freeradius.org Betelwiki, Betelwiki, Betelwiki.... http://wiki.freeradius.org/ !
On 10/05/2011 07:16 AM, Arran Cudbard-Bell wrote:
No i've talked about this with Alan. The proposed solution is an enhancement to the detail writer/reader functionality where the server automagically spawns new reader instances to listen on per domain detail files.
I guess that's ok, in that it stops an unresponsive realm blocking other realms, but wouldn't another solution be to add a config item to the detail reader to drop packets which are >X seconds old? That would help with another case - where a realm is accepting some accounting packets but not others (e.g. they've got a SQL config that is barfing on upper-case usernames...). Or, modify the detail reader to write a "done" marker byte into records which it has processed. p1 = ftell read packet send packet if ok p2 = ftell seek(p1) write('OK') seek(p2) else: sleep & retry Anyway - something else the OP can do today (aside from rlm_replicate) is to use follow the "robust-proxy-accounting" example, but instead of queueing failed packets, just "ok" them or write to a detail file.
Phil Mayers wrote:
I guess that's ok, in that it stops an unresponsive realm blocking other realms, but wouldn't another solution be to add a config item to the detail reader to drop packets which are >X seconds old?
if (Acct-Delay-Time > 3600) { ok } else { ... do proxy .... }
Or, modify the detail reader to write a "done" marker byte into records which it has processed.
That's a bit harder. Alan DeKok.
On 10/05/2011 09:26 AM, Alan DeKok wrote:
Phil Mayers wrote:
I guess that's ok, in that it stops an unresponsive realm blocking other realms, but wouldn't another solution be to add a config item to the detail reader to drop packets which are>X seconds old?
if (Acct-Delay-Time> 3600) { ok } else { ... do proxy .... }
Ah ha! Clever. I had forgotten the detail reader created/updated that attribute. Yay FreeRADIUS!
Or, modify the detail reader to write a "done" marker byte into records which it has processed.
That's a bit harder.
Tell me about it... we've got a home-grown system for logging DHCP requests via pcap (because ISC dhcpd can't do decent logging) and, along with a whole bunch of other crap, the process has to write to a logfile in chunks, with the ability to do this (mark one "row" as written) It was amazingly unreliable... I eventually resorted to writing them to an sqlite database as a "spool", on the grounds that "marking" them done/ignored was easier that way. One of these days I'll get the time to investigate FR DHCP...
On 5 Oct 2011, at 10:40, Phil Mayers wrote:
On 10/05/2011 09:26 AM, Alan DeKok wrote:
Phil Mayers wrote:
I guess that's ok, in that it stops an unresponsive realm blocking other realms, but wouldn't another solution be to add a config item to the detail reader to drop packets which are>X seconds old?
if (Acct-Delay-Time> 3600) { ok } else { ... do proxy .... }
Ah ha! Clever. I had forgotten the detail reader created/updated that attribute. Yay FreeRADIUS!
It's a bad way of doing it. At least with replicate every accounting packet has a chance... Using Acct-Delay-Time you'll end up dumping anywhere between 1-15 seconds accounting data for all realms if one realm is unreachable. -Arran Arran Cudbard-Bell a.cudbardb@freeradius.org Betelwiki, Betelwiki, Betelwiki.... http://wiki.freeradius.org/ !
Arran Cudbard-Bell wrote:
It's a bad way of doing it. At least with replicate every accounting packet has a chance... Using Acct-Delay-Time you'll end up dumping anywhere between 1-15 seconds accounting data for all realms if one realm is unreachable.
<shrug> if (Packet-Transmit-Counter > 5) { ok } else { ... proxy ... } If the home server doesn't get it after 5 tries, throw it away. In 2.1.10 & later, IIRC. Alan DeKok.
On 05/10/11 09:56, Arran Cudbard-Bell wrote:
On 5 Oct 2011, at 10:40, Phil Mayers wrote:
On 10/05/2011 09:26 AM, Alan DeKok wrote:
Phil Mayers wrote:
I guess that's ok, in that it stops an unresponsive realm blocking other realms, but wouldn't another solution be to add a config item to the detail reader to drop packets which are>X seconds old?
if (Acct-Delay-Time> 3600) { ok } else { ... do proxy .... }
Ah ha! Clever. I had forgotten the detail reader created/updated that attribute. Yay FreeRADIUS!
It's a bad way of doing it. At least with replicate every accounting packet has a chance... Using Acct-Delay-Time you'll end up dumping anywhere between 1-15 seconds accounting data for all realms if one realm is unreachable.
Whereas with rlm_replicate, you risk dropping arbitrary accounting packets because there is no retry. There is no ideal solution, because radius accounting was never designed for the kind of loosely-coupled federation that is Eduroam. For me, since most Eduroam sites don't care about receiving federated accounting, my primary concern is for my server to carry on functioning, and that means the detail file should not grow without bound. I don't really care how that happens - as long as it does.
On 5 Oct 2011, at 12:50, Phil Mayers wrote:
On 05/10/11 09:56, Arran Cudbard-Bell wrote:
On 5 Oct 2011, at 10:40, Phil Mayers wrote:
On 10/05/2011 09:26 AM, Alan DeKok wrote:
Phil Mayers wrote:
I guess that's ok, in that it stops an unresponsive realm blocking other realms, but wouldn't another solution be to add a config item to the detail reader to drop packets which are>X seconds old?
if (Acct-Delay-Time> 3600) { ok } else { ... do proxy .... }
Ah ha! Clever. I had forgotten the detail reader created/updated that attribute. Yay FreeRADIUS!
It's a bad way of doing it. At least with replicate every accounting packet has a chance... Using Acct-Delay-Time you'll end up dumping anywhere between 1-15 seconds accounting data for all realms if one realm is unreachable.
Whereas with rlm_replicate, you risk dropping arbitrary accounting packets because there is no retry. There is no ideal solution, because radius accounting was never designed for the kind of loosely-coupled federation that is Eduroam. For me, since most Eduroam sites don't care about receiving federated accounting, my primary concern is for my server to carry on functioning, and that means the detail file should not grow without bound. I don't really care how that happens - as long as it does.
True. Both solutions suck in their own unique ways. Roll on RADSEC. -Arran Arran Cudbard-Bell a.cudbardb@freeradius.org Betelwiki, Betelwiki, Betelwiki.... http://wiki.freeradius.org/ !
On Wed, 5 Oct 2011, Arran Cudbard-Bell wrote:
On 5 Oct 2011, at 02:20, Mike Diggins wrote:
I'm running FreeRadius 2.1.3 on RedHat Enterprise Linux configured as an Eduroam Radius proxy server. My Cisco Wireless Lan Controllers are constantly failing over the Accounting Servers, due to lack of response from the Home Servers, or so says the log. However, I believe the issue is that some remote institutions Radius Servers are ignoring the Accounting packets, and timing out my end, making it believe the Home Servers have failed to respond. FreeRadius responds by marking the Home server dead. It then sends a status-server query, to which is gets a reply, and enables the Dead Home server. I believe that's the sequence of events anyway. I captured some of that in debug mode:
Rejecting request 288 due to lack of any response from home server x.x.x.x port 1813
Finished request 288.
Cleaning up request 288 ID 205 with timestamp +1161
PROXY: Marking home server x.x.x.x port 1813 as zombie (it looks like it is dead).
Sending Status-Server of id 55 to x.x.x.x port 1813 Message-Authenticator := 0x00000000000000000000000000000000 NAS-Identifier := "Status Check. Are you alive?" Waking up in 3.9 seconds.
rad_recv: Access-Accept packet from host x.x.x.x port 1813, id=55, length=806
I don't have any control over Accounting Packets being accepted, or not, by other Eduroam members. Some do, some don't I imagine. Is there a configuration for FreeRadius that handles this situation cleanly? Seems to me that FR should check the Home server first, before marking it dead (at least).
No i've talked about this with Alan. The proposed solution is an enhancement to the detail writer/reader functionality where the server automagically spawns new reader instances to listen on per domain detail files.
For now i'd recommend using rlm_replicate, this allows you to send an entirely new request to the homeserver, whilst sending an Accounting-Response to the NAS. FR does not track the new request and so won't be able to retransmit.
Just use something like:
preacct { if(Realm && Realm != 'local'){ update control { Proxy-To-Realm := 'local' Replicate-To-Realm = 'eduroam' } } }
accounting { replicate }
Thanks for your suggestion. Do I drop this into my existing ..sites-available/default file? Does this add to what's there already, or do I replace what's there with this (and does the order matter)? Sorry, I have a very basic configuration and haven't had to delve into this. -Mike
Thanks for your suggestion. Do I drop this into my existing ..sites-available/default file? Does this add to what's there already, or do I replace what's there with this (and does the order matter)? Sorry, I have a very basic configuration and haven't had to delve into this.
Add it to sites-available/default in the existing sections (put replicate in the accounting {} section, put the other stuff in the preacct section... -Arran Arran Cudbard-Bell a.cudbardb@freeradius.org Betelwiki, Betelwiki, Betelwiki.... http://wiki.freeradius.org/ !
On Wed, 5 Oct 2011, Arran Cudbard-Bell wrote:
Thanks for your suggestion. Do I drop this into my existing ..sites-available/default file? Does this add to what's there already, or do I replace what's there with this (and does the order matter)? Sorry, I have a very basic configuration and haven't had to delve into this.
Add it to sites-available/default in the existing sections (put replicate in the accounting {} section, put the other stuff in the preacct section...
Tried this today but got an error. I'm running 2.1.3. Module: Checking accounting {...} for more modules to load /usr/local/freeradius/etc/raddb/sites-enabled/default[339]: Failed to find module "replicate". /usr/local/freeradius/etc/raddb/sites-enabled/default[336]: Errors parsing accounting section. -Mike
Mike Diggins wrote:
Tried this today but got an error. I'm running 2.1.3.
Upgrade.
Module: Checking accounting {...} for more modules to load /usr/local/freeradius/etc/raddb/sites-enabled/default[339]: Failed to find module "replicate". /usr/local/freeradius/etc/raddb/sites-enabled/default[336]: Errors parsing accounting section.
There's no "replicate" module in 2.1.3. Try 2.1.12 Alan DeKok
Hi,
I'm running FreeRadius 2.1.3 on RedHat Enterprise Linux configured as an Eduroam Radius proxy server. My Cisco Wireless Lan Controllers are constantly failing over the Accounting Servers, due to lack of response from the Home Servers, or so says the log. However, I believe the issue is that some remote institutions Radius Servers are ignoring the Accounting packets, and timing out my end, making it believe the Home Servers have failed to respond. FreeRadius responds by marking the Home server dead. It then sends a status-server query, to which is gets a reply, and enables the Dead Home server. I believe that's the sequence of events anyway. I captured some of that in debug mode:
the flow you captured is right - it tries, it doesnt get a response, it probes with status-server and marks server up again. thats fine...and the way it should work. regarding eduroam and proxying of accounting. yes. its a continual problem and one that I would like to see sorted...theres a large number of people who dont believe in the proxying of accounting (they dont believe in the accounting stuff really...) - I, however, like the accounting. the issue is that YOU know your NAS kit - and all the wierd things that it can do - sending 0 sesstion time and only using port 29 is the Cisco 'thing'.. what you arent ready for is the slew of junk from other sites NAS when they do accounting ..and be prepared for a whole lot of junk. (my current favourite is when 100% essential RADIUS attributes for accounting have been striped out by an over keen admin with the attribute filter on their RADIUS server. double fun!) I would suggest you take the accounting out of the 'live' stream and move to using one of the 'supplied with freeRADIUS' accounting virtual servers - decoupled-accounting, robust-proxy-accounting or copy-acct-to-home-server read those VS files and see which one appeals to you the most and which will suit your needs alan
participants (5)
-
Alan Buxey -
Alan DeKok -
Arran Cudbard-Bell -
Mike Diggins -
Phil Mayers