Re: segmentation fault freeradius 2.1.7 using rlm_sql
Upgraded freeradius to 2.1.11 (built from source)
Don't use 2.1.11 it segfaults, checkout the head of the 2.1.X branch in git
Notice how I DIDN'T suggest upgrading to 2.1.11, but to v2.1.x of git branch? There's a reason for that, and you just found out the hard way.
"Houston, we have a problem" ;-) This is not the first time a FreeRADIUS release was not ready for production when it was released. Those of us who package upstream projects for distribution worry a lot about stability and robustness. I've said this before so forgive me, but I'm going to reiterate it again. Please don't get mad at the messenger, I have only the best intentions with these observations. FreeRADIUS has some problems which other projects have avoided. * FreeRADIUS has no notion of a "stable release". Many projects maintain both a stable production version and a current version (which is not the same as the "tip", rather it's tagged in source code control, tested and released just like any other release, it's just got a few more features than the rock solid stable release). The rock solid stable release has been field proven, should have the absolute confidence of system administrators and be viable for multiple years (in other words you can install it and be confident once it's put in production you're good to go for several years. Occasionally a stable release needs a bug or security fix. When that occurs the stable release is surgically modified to fix exactly that one issue, it's minor version number is bumped. System administrators are never told to upgrade to a significant new version because of the bug/security issue, instead they reinstall a patched version of "stable". * FreeRADIUS has way too much churn for a critical system service. Think about other system services, how often do you see kerberos, bind, iptables, pam, MySQL, etc. going through significant revisions? Are the administrators of those services constantly being told to upgrade the service because of the bug/feature du jour? * The QE component of FreeRADIUS has proven to be inadequate. I know Alan runs a set of tests and he calls for testing prior to a new release. But we've seen the amount of testing which actually occurs is inadequate because releases have gone out with significant problems and those releases have gotten pushed into production. I think part of the problem is the frequent release schedule (measured in months) and the lack of a coordinated beta testing program. Releases should not occur until after they've successfully navigated a beta program. I humbly would suggest the following: * Create and maintain a "stable" version. * Organize a rigorous beta test program. * Slow down the release schedule, avoid the temptation to cut a new release because of minor new features. If production servers can't run successfully without a feature that's an indication the prior release was too hasty. Critical bug fixes should occur in the release branch and the release branch re-released. The release interval for a system service like FreeRADIUS should be measured in years, not months or weeks. Comments? Thoughts? Do you agree/disagree? John -- John Dennis <jdennis@redhat.com> Looking to carve out IT costs? www.redhat.com/carveoutcosts/
Houston, who's having the problem? :) Perhaps in a perfect world, everyone writing open-source is fully employed by RedHat, Google, IBM, or some other company who directly or indirectly benefits from the software. Where you have (paid) teams of people to patch, maintain, and test features and releases. Of course, we'd all would like to see and benefit from more rigorous testing and packaging in open-source. But, isn't that part of your (paid) job? :) I don't really see the argument being made why the FR team should do all that you ask; it doesn't put their kids through school or put food on their tables; and obviously doesn't keep people from using the software... You also suggest "Organize a rigorous beta test program." Well, do you test FR when asked to? Are you suggesting they _pay_ testers? Perhaps releasing new versions is the only way they eventually get people to test and report problems... I know you have the best intentions in mind, but I think it's wrong to imply the FR team does not... David On Tue, 2011-08-02 at 09:48 -0400, John Dennis wrote:
Upgraded freeradius to 2.1.11 (built from source)
Don't use 2.1.11 it segfaults, checkout the head of the 2.1.X branch in git
Notice how I DIDN'T suggest upgrading to 2.1.11, but to v2.1.x of git branch? There's a reason for that, and you just found out the hard way.
"Houston, we have a problem" ;-)
This is not the first time a FreeRADIUS release was not ready for production when it was released. Those of us who package upstream projects for distribution worry a lot about stability and robustness. I've said this before so forgive me, but I'm going to reiterate it again. Please don't get mad at the messenger, I have only the best intentions with these observations.
FreeRADIUS has some problems which other projects have avoided.
* FreeRADIUS has no notion of a "stable release". Many projects maintain both a stable production version and a current version (which is not the same as the "tip", rather it's tagged in source code control, tested and released just like any other release, it's just got a few more features than the rock solid stable release). The rock solid stable release has been field proven, should have the absolute confidence of system administrators and be viable for multiple years (in other words you can install it and be confident once it's put in production you're good to go for several years. Occasionally a stable release needs a bug or security fix. When that occurs the stable release is surgically modified to fix exactly that one issue, it's minor version number is bumped. System administrators are never told to upgrade to a significant new version because of the bug/security issue, instead they reinstall a patched version of "stable".
* FreeRADIUS has way too much churn for a critical system service. Think about other system services, how often do you see kerberos, bind, iptables, pam, MySQL, etc. going through significant revisions? Are the administrators of those services constantly being told to upgrade the service because of the bug/feature du jour?
* The QE component of FreeRADIUS has proven to be inadequate. I know Alan runs a set of tests and he calls for testing prior to a new release. But we've seen the amount of testing which actually occurs is inadequate because releases have gone out with significant problems and those releases have gotten pushed into production. I think part of the problem is the frequent release schedule (measured in months) and the lack of a coordinated beta testing program. Releases should not occur until after they've successfully navigated a beta program.
I humbly would suggest the following:
* Create and maintain a "stable" version.
* Organize a rigorous beta test program.
* Slow down the release schedule, avoid the temptation to cut a new release because of minor new features. If production servers can't run successfully without a feature that's an indication the prior release was too hasty. Critical bug fixes should occur in the release branch and the release branch re-released. The release interval for a system service like FreeRADIUS should be measured in years, not months or weeks.
Comments? Thoughts? Do you agree/disagree?
John
David Bird wrote:
Perhaps releasing new versions is the only way they eventually get people to test and report problems...
Yes. Look at the release history on this mailing list. "A new release will come soon, here's a link to a beta TGZ" <crickets> "OK, the release is ready" OMG! It doesn't work! "Arg... GTFO" Alan DeKok.
John Dennis wrote:
FreeRADIUS has some problems which other projects have avoided.
Sure. The reasons are pretty straightforward. The contribution from the community is small. The people who contribute get few rewards, and lots of arrows. The people who complain don't contribute. It's really that simple. All of the rest you posted below is engineering process. It's all nice. But it requires someone to do the work. And I don't see anyone volunteering. The few times I've asked, everyone says they're busy.
* FreeRADIUS has no notion of a "stable release". Many projects maintain both a stable production version and a current version (which is not the same as the "tip", rather it's tagged in source code control, tested and released just like any other release, it's just got a few more features than the rock solid stable release). The rock solid stable release has been field proven, should have the absolute confidence of system administrators and be viable for multiple years (in other words you can install it and be confident once it's put in production you're good to go for several years. Occasionally a stable release needs a bug or security fix. When that occurs the stable release is surgically modified to fix exactly that one issue, it's minor version number is bumped. System administrators are never told to upgrade to a significant new version because of the bug/security issue, instead they reinstall a patched version of "stable".
Great. That's done with 1.1.x. Who's going to do it for 2.x?
* FreeRADIUS has way too much churn for a critical system service. Think about other system services, how often do you see kerberos, bind, iptables, pam, MySQL, etc. going through significant revisions? Are the administrators of those services constantly being told to upgrade the service because of the bug/feature du jour?
Git is useful here. The difference between 2.1.10 and 2.1.11 is 4K LoC added, ~1K deleted, out of more than 80K. That's less than 5%. And a lot of that is adding "extern C" headers to files.
* The QE component of FreeRADIUS has proven to be inadequate. I know Alan runs a set of tests and he calls for testing prior to a new release. But we've seen the amount of testing which actually occurs is inadequate because releases have gone out with significant problems and those releases have gotten pushed into production. I think part of the problem is the frequent release schedule (measured in months) and the lack of a coordinated beta testing program. Releases should not occur until after they've successfully navigated a beta program.
Great. Help.
I humbly would suggest the following:
* Create and maintain a "stable" version.
Perhaps. I plan on releasing 3.0 within a few months. The 2.1.x branch can then be re-labeled 2.2.0. I can be marked "stable", if someone is willing to put work into maintaining it. And we're already doing this for 1.1.x. It works, it's shipped in many "legacy" packages. It hasn't changed in years.
* Organize a rigorous beta test program.
<crickets> Right. Lots of volunteers there.
* Slow down the release schedule, avoid the temptation to cut a new release because of minor new features. If production servers can't run successfully without a feature that's an indication the prior release was too hasty. Critical bug fixes should occur in the release branch and the release branch re-released. The release interval for a system service like FreeRADIUS should be measured in years, not months or weeks.
Absolutely not. There are a TON of changes being made to RADIUS. It's simply not feasible to have release intervals of years. And let me remind you: we DO have a "stable" release with a release interval of years. What happens? People don't even use the most recent version of that (1.1.8). Instead, they use an older version (1.1.3), because that's the only thing that THEIR OS VENDOR SUPPORTS. Hint? Then they ask for help here. It's ridiculous. I'll bet RH is making more money off of FreeRADIUS than I am. So if you want a stable release, contribute. Otherwise, I'm tired of the complaints. You've sent me email off-list complaining that YOUR CUSTOMERS are upset about the quality of the FR documentation. Fine. When I asked if you were willing to pay for better documentation, you didn't respond. And again here, you've posted a long message detailing work that *other people* are supposed to do. There is a conspicuous failure to volunteer for ANY of the things on your list. So... thanks for the contribution. Doing what you want (for free) is not really high on my list of priorities. I'll do what I can to make FR better. But that's only because I want to, and I'm spending *my* hard-earned money to do so. i.e. I'm putting MY money where my mouth is. I invite others to do the same. Alan DeKok.
Hi,
* FreeRADIUS has no notion of a "stable release". Many projects maintain
'stable' or 'stale' if you go for the 'stable' release of most daemons you will not have the ability to do certain things, the latest version will have those abilities. in the FR case, the most 'tested' version is often an older version...bugs come to light, new features requested etc are in the newer version, not backported down to older versions. it makes sense to do it that way
than the rock solid stable release). The rock solid stable release has been field proven, should have the absolute confidence of system administrators and be viable for multiple years (in other words you can install it and be confident once it's put in production you're good to go for several years.
rubbish. sorry, but the number of times I've had to upgrade eg BIND, ISC DHCPD, apache etc from 'stable' release due to a bug, or that months new vulnerability. i dont know of ANY server daemon that could be installed and left online for years - even if there was such a mythical beast, the server it runs on needs to be reloaded at least 4 times a year due to kernel/OS vulnerabilities too ;-)
security fix. When that occurs the stable release is surgically modified to fix exactly that one issue, it's minor version number is bumped.
thats the wrong way to go as you spend the effort putting features way back down in the tree - and often reverting behaviour already known or fixed. ideally you have a stable version and a test-devel version - which there has been before - however, who runs what? often people run safe and wont run a test-devel version where its needed - and only when the version is 'released' do they then run it and find a bug/issue that no-one else uncovered - this has happened several times this year with 2.1.x
* FreeRADIUS has way too much churn for a critical system service. Think about other system services, how often do you see kerberos, bind, iptables, pam, MySQL, etc. going through significant revisions? Are the administrators of those services constantly being told to upgrade the service because of the bug/feature du jour?
all the time. BIND and MySQL in particular. I'd add OpenSSH and Apache to the list of daemons always being upgraded.
those releases have gotten pushed into production. I think part of the problem is the frequent release schedule (measured in months) and the lack of a coordinated beta testing program. Releases should not occur until after they've successfully navigated a beta program.
I , for one, love having frequent releases of the server - it means we get new features. functions and fixes of behaviour rapidly in what has become a very agile area of network access. if you have a less frequent release you'll have a product with as much functionality as cisco ACS of MS NPS - ie bare and basic. dull and almost useless. of course, this depends on your market - if you are a dial-in user or run an ISP you might want the basic functions. there. done. free. dealt with. if, however, you are using it in an enterprise network environment and are looking at 802.1X with SoH, new rapid policy making, advanced external auditing using custom logging and maybe even wanting to use other RADIUS capabilities, CUI, CoA, linked to that months common backend - or wanting to use enhanced remote access systems like 'moonshot' then you need an agile project - FreeRADIUS is agile enough
* Organize a rigorous beta test program.
using what and who? most issues are found in demanding environments that cannot be modelled by some simulation - this isnt Apache where you can throw a tool at it. there isnt a 'generic config' that all people use. I'll find a bug in postgres that noone else has found because they all use another DB or flatfile.. or if they do use postgres they wont be throwing particular things at it (like storing a local copy of an accounting packet that wasnt received at the remote accounting server and couldnt be stored on sql-relay due to no disk space....) - thats just an example by the way! - by changing to 'stable' and having a remote 'test' release you can pretty much say goodbye to the default presence of leading-tree testers. i know most people will just take the latest stable. they may have 'stable' but they'll whine when they dont have a feature...and when posting to the list i'll say its in 'testing' and they'll ask when thats coming to 'stable - answer? probably never to that stable tree because the code changes to enable that feature are vast and the profiling wont be able to pick out any weird corner cases that might happen to that stable should the functions be backported. as alan said, 1.1.x is 'stable' its also damn bare of any flexible functionality ;-) alan
John Dennis wrote:
"Houston, we have a problem" ;-)
The "we" part is presumptuous in this case.
* FreeRADIUS has no notion of a "stable release". Many projects
That is matter of definition. My definition of the "stable release" is that bleeding edge 3.x snapshot that still appears to be stable about 6 months in the future. And yes, that's right, it means that whatever is "stable" will not be known until about 6 months after it has been "published". This mechanism works just fine for anyone running it in production servers or needing to wrap it in a distribution package.
I humbly would suggest the following:
* Create and maintain a "stable" version.
That is already being done, just adjust your definition.
* Organize a rigorous beta test program.
Well, sounds like a plan. Does that mean you are volunteering to organise it? -- Stephen. Confidence is that feeling you have before you understand the situation.
participants (5)
-
Alan Buxey -
Alan DeKok -
David Bird -
John Dennis -
Stephen R. van den Berg