segmentation fault freeradius 2.1.7 using rlm_sql
Hi all, freeRADIUS version 2.1.7 (package freeradius2 on centos 5.6) Radius is configured to store user info (auth) in ldap - running on same machine as freeradius, and use rlm_sql for accounting info (and simultaneous use checks) - running on remote machine. For some unknown reason radiusd keeps getting segmentation faults, every few days and even after several hours of work. Jul 27 04:01:03 RADIUS4 kernel: radiusd[29903]: segfault at 00000000000000c8 rip 00002aef39fb9cfb rsp 00000000456d6170 error 4 Last lines in the radius.log : [sql] expand: UPDATE radacct SET acctstoptime = '%S', acctsessiontime = '%{Acct-Session-Time}', acctinputoctets = '%{%{Acct-Input-Gigawords}:-0}' << 32 | '%{%{Acct-Input-Octets}:-0}', acctoutputoctets = '%{%{Acct-Output-Gigawords}:-0}' << 32 | '%{%{Acct-Output-Octets}:-0}', acctterminatecause = '%{Acct-Terminate-Cause}', acctstopdelay = '%{%{Acct-Delay-Time}:-0}', connectinfo_stop = '%{Connect-Info}' WHERE acctsessionid = '%{Acct-Session-Id}' AND username = '%{SQL-User-Name}' AND nasipaddress = '%{NAS-IP-Address}' -> UPDATE radacct SET acctstoptime = '2011-07-27 04:01:03', acctsessiontime = '0', acctinputoctets = '0' << 32 | '0', acctoutputoctets = '0' << 32 | [sql] expand: /var/log/radius/sqltrace.sql -> /var/log/radius/sqltrace.sql rlm_sql (sql): Trying to (re)connect unconnected handle 12.. rlm_sql (sql): Attempting to connect rlm_sql_mysql #12 since configuration files are quite large, I didn't attach complete dump or radius -X , let me know what other info is needed. Regards, Amir Tal.
I did, from 1.1.7 And still I get the exact same result (segmentation faults) -----Original Message----- From: freeradius-users-bounces+amir=ccc.co.il@lists.freeradius.org [mailto:freeradius-users-bounces+amir=ccc.co.il@lists.freeradius.org] On Behalf Of Alan DeKok Sent: Wednesday, July 27, 2011 4:33 PM To: FreeRadius users mailing list Subject: Re: segmentation fault freeradius 2.1.7 using rlm_sql Amir Tal wrote:
For some unknown reason radiusd keeps getting segmentation faults, every few days and even after several hours of work.
Upgrade. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On Thu, Jul 28, 2011 at 7:04 PM, Amir Tal <amir@ccc.co.il> wrote:
I did, from 1.1.7 And still I get the exact same result (segmentation faults)
I think what Allan means is upgrade to latest FR version, as (presumably) the bug is fixed in later version. Since RHEL/Centos 5 only comes with freeradius2-2.1.7, you need to either compile from source (latest git snapshot of v.2.1.x branch would be good, see https://github.com/alandekok/freeradius-server/tree/v2.1.x) or build your own RPM (see http://wiki.freeradius.org/Red_Hat_FAQ#How+to+build+an+SRPM) -- Fajar
-----Original Message----- From: freeradius-users-bounces+amir=ccc.co.il@lists.freeradius.org [mailto:freeradius-users-bounces+amir=ccc.co.il@lists.freeradius.org] On Behalf Of Alan DeKok Sent: Wednesday, July 27, 2011 4:33 PM To: FreeRadius users mailing list Subject: Re: segmentation fault freeradius 2.1.7 using rlm_sql
Amir Tal wrote:
For some unknown reason radiusd keeps getting segmentation faults, every few days and even after several hours of work.
Upgrade.
Alan DeKok.
Upgraded freeradius to 2.1.11 (built from source) This time it brought the whole machine down: radiusd[8644]: segfault at 00000000000000c8 rip 00002aad51190cfb rsp 00000000504b8770 error 4 radiusd[16348]: segfault at 00000000000000c8 rip 00002afffa168cfb rsp 0000000046c00170 error 4 radiusd[22540]: segfault at 00000000000004b8 rip 00002b10e5147850 rsp 000000004707c2c8 error 4 radiusd[30297]: segfault at 00000000000000c8 rip 00002b848c84bcfb rsp 0000000054924750 error 4 audispd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Call Trace: [<ffffffff800c961b>] out_of_memory+0x8e/0x2f3 [<ffffffff8002e3fc>] __wake_up+0x38/0x4f [<ffffffff8000f625>] __alloc_pages+0x27f/0x308 [<ffffffff8001300a>] __do_page_cache_readahead+0x96/0x179 [<ffffffff80013945>] filemap_nopage+0x14c/0x360 [<ffffffff80008964>] __handle_mm_fault+0x1fb/0x1039 [<ffffffff8006720b>] do_page_fault+0x4cb/0x874 [<ffffffff8005dde9>] error_exit+0x0/0x84 Mem-info: Node 0 DMA per-cpu: cpu 0 hot: high 0, batch 1 used:0 cpu 0 cold: high 0, batch 1 used:0 cpu 1 hot: high 0, batch 1 used:0 cpu 1 cold: high 0, batch 1 used:0 Node 0 DMA32 per-cpu: cpu 0 hot: high 186, batch 31 used:18 cpu 0 cold: high 62, batch 15 used:45 cpu 1 hot: high 186, batch 31 used:31 cpu 1 cold: high 62, batch 15 used:61 Node 0 Normal per-cpu: cpu 0 hot: high 186, batch 31 used:61 cpu 0 cold: high 62, batch 15 used:49 cpu 1 hot: high 186, batch 31 used:24 cpu 1 cold: high 62, batch 15 used:52 Node 0 HighMem per-cpu: empty Free pages: 22180kB (0kB HighMem) Active:535129 inactive:455633 dirty:0 writeback:0 unstable:0 free:5545 slab:3369 mapped-file:1085 mapped-anon:990241 pagetables:5050 Node 0 DMA free:10116kB min:16kB low:20kB high:24kB active:0kB inactive:0kB present:9748kB pages_scanned:0 all_unreclaimable? yes lowmem_reserve[]: 0 3000 4010 4010 Node 0 DMA32 free:10052kB min:6052kB low:7564kB high:9076kB active:1623644kB inactive:1405408kB present:3072160kB pages_scanned:19313612 all_unreclaimable? yes lowmem_reserve[]: 0 0 1010 1010 Node 0 Normal free:2012kB min:2036kB low:2544kB high:3052kB active:508040kB inactive:425828kB present:1034240kB pages_scanned:2193259 all_unreclaimable? yes lowmem_reserve[]: 0 0 0 0 Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 5*4kB 2*8kB 4*16kB 3*32kB 5*64kB 3*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10116kB Node 0 DMA32: 13*4kB 4*8kB 1*16kB 13*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10052kB Node 0 Normal: 17*4kB 19*8kB 0*16kB 0*32kB 0*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2012kB Node 0 HighMem: empty 1662 pagecache pages Swap cache: add 1095074, delete 1094514, find 2338993/2348642, race 0+1 Free swap = 0kB Total swap = 4096564kB Free swap: 0kB 1310720 pages of RAM 299836 reserved pages 8240 pages shared 560 pages swap cached Out of memory: Killed process 18723, UID 95, (radiusd). -----Original Message----- From: freeradius-users-bounces+amir=ccc.co.il@lists.freeradius.org [mailto:freeradius-users-bounces+amir=ccc.co.il@lists.freeradius.org] On Behalf Of Alan DeKok Sent: Wednesday, July 27, 2011 4:33 PM To: FreeRadius users mailing list Subject: Re: segmentation fault freeradius 2.1.7 using rlm_sql Amir Tal wrote:
For some unknown reason radiusd keeps getting segmentation faults, every few days and even after several hours of work.
Upgrade. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Upgraded freeradius to 2.1.11 (built from source)
Don't use 2.1.11 it segfaults, checkout the head of the 2.1.X branch in git
Notice how I DIDN'T suggest upgrading to 2.1.11, but to v2.1.x of git branch? There's a reason for that, and you just found out the hard way.
"Houston, we have a problem" ;-) This is not the first time a FreeRADIUS release was not ready for production when it was released. Those of us who package upstream projects for distribution worry a lot about stability and robustness. I've said this before so forgive me, but I'm going to reiterate it again. Please don't get mad at the messenger, I have only the best intentions with these observations. FreeRADIUS has some problems which other projects have avoided. * FreeRADIUS has no notion of a "stable release". Many projects maintain both a stable production version and a current version (which is not the same as the "tip", rather it's tagged in source code control, tested and released just like any other release, it's just got a few more features than the rock solid stable release). The rock solid stable release has been field proven, should have the absolute confidence of system administrators and be viable for multiple years (in other words you can install it and be confident once it's put in production you're good to go for several years. Occasionally a stable release needs a bug or security fix. When that occurs the stable release is surgically modified to fix exactly that one issue, it's minor version number is bumped. System administrators are never told to upgrade to a significant new version because of the bug/security issue, instead they reinstall a patched version of "stable". * FreeRADIUS has way too much churn for a critical system service. Think about other system services, how often do you see kerberos, bind, iptables, pam, MySQL, etc. going through significant revisions? Are the administrators of those services constantly being told to upgrade the service because of the bug/feature du jour? * The QE component of FreeRADIUS has proven to be inadequate. I know Alan runs a set of tests and he calls for testing prior to a new release. But we've seen the amount of testing which actually occurs is inadequate because releases have gone out with significant problems and those releases have gotten pushed into production. I think part of the problem is the frequent release schedule (measured in months) and the lack of a coordinated beta testing program. Releases should not occur until after they've successfully navigated a beta program. I humbly would suggest the following: * Create and maintain a "stable" version. * Organize a rigorous beta test program. * Slow down the release schedule, avoid the temptation to cut a new release because of minor new features. If production servers can't run successfully without a feature that's an indication the prior release was too hasty. Critical bug fixes should occur in the release branch and the release branch re-released. The release interval for a system service like FreeRADIUS should be measured in years, not months or weeks. Comments? Thoughts? Do you agree/disagree? John -- John Dennis <jdennis@redhat.com> Looking to carve out IT costs? www.redhat.com/carveoutcosts/
John Dennis wrote:
FreeRADIUS has some problems which other projects have avoided.
Sure. The reasons are pretty straightforward. The contribution from the community is small. The people who contribute get few rewards, and lots of arrows. The people who complain don't contribute. It's really that simple. All of the rest you posted below is engineering process. It's all nice. But it requires someone to do the work. And I don't see anyone volunteering. The few times I've asked, everyone says they're busy.
* FreeRADIUS has no notion of a "stable release". Many projects maintain both a stable production version and a current version (which is not the same as the "tip", rather it's tagged in source code control, tested and released just like any other release, it's just got a few more features than the rock solid stable release). The rock solid stable release has been field proven, should have the absolute confidence of system administrators and be viable for multiple years (in other words you can install it and be confident once it's put in production you're good to go for several years. Occasionally a stable release needs a bug or security fix. When that occurs the stable release is surgically modified to fix exactly that one issue, it's minor version number is bumped. System administrators are never told to upgrade to a significant new version because of the bug/security issue, instead they reinstall a patched version of "stable".
Great. That's done with 1.1.x. Who's going to do it for 2.x?
* FreeRADIUS has way too much churn for a critical system service. Think about other system services, how often do you see kerberos, bind, iptables, pam, MySQL, etc. going through significant revisions? Are the administrators of those services constantly being told to upgrade the service because of the bug/feature du jour?
Git is useful here. The difference between 2.1.10 and 2.1.11 is 4K LoC added, ~1K deleted, out of more than 80K. That's less than 5%. And a lot of that is adding "extern C" headers to files.
* The QE component of FreeRADIUS has proven to be inadequate. I know Alan runs a set of tests and he calls for testing prior to a new release. But we've seen the amount of testing which actually occurs is inadequate because releases have gone out with significant problems and those releases have gotten pushed into production. I think part of the problem is the frequent release schedule (measured in months) and the lack of a coordinated beta testing program. Releases should not occur until after they've successfully navigated a beta program.
Great. Help.
I humbly would suggest the following:
* Create and maintain a "stable" version.
Perhaps. I plan on releasing 3.0 within a few months. The 2.1.x branch can then be re-labeled 2.2.0. I can be marked "stable", if someone is willing to put work into maintaining it. And we're already doing this for 1.1.x. It works, it's shipped in many "legacy" packages. It hasn't changed in years.
* Organize a rigorous beta test program.
<crickets> Right. Lots of volunteers there.
* Slow down the release schedule, avoid the temptation to cut a new release because of minor new features. If production servers can't run successfully without a feature that's an indication the prior release was too hasty. Critical bug fixes should occur in the release branch and the release branch re-released. The release interval for a system service like FreeRADIUS should be measured in years, not months or weeks.
Absolutely not. There are a TON of changes being made to RADIUS. It's simply not feasible to have release intervals of years. And let me remind you: we DO have a "stable" release with a release interval of years. What happens? People don't even use the most recent version of that (1.1.8). Instead, they use an older version (1.1.3), because that's the only thing that THEIR OS VENDOR SUPPORTS. Hint? Then they ask for help here. It's ridiculous. I'll bet RH is making more money off of FreeRADIUS than I am. So if you want a stable release, contribute. Otherwise, I'm tired of the complaints. You've sent me email off-list complaining that YOUR CUSTOMERS are upset about the quality of the FR documentation. Fine. When I asked if you were willing to pay for better documentation, you didn't respond. And again here, you've posted a long message detailing work that *other people* are supposed to do. There is a conspicuous failure to volunteer for ANY of the things on your list. So... thanks for the contribution. Doing what you want (for free) is not really high on my list of priorities. I'll do what I can to make FR better. But that's only because I want to, and I'm spending *my* hard-earned money to do so. i.e. I'm putting MY money where my mouth is. I invite others to do the same. Alan DeKok.
Hi,
* FreeRADIUS has no notion of a "stable release". Many projects maintain
'stable' or 'stale' if you go for the 'stable' release of most daemons you will not have the ability to do certain things, the latest version will have those abilities. in the FR case, the most 'tested' version is often an older version...bugs come to light, new features requested etc are in the newer version, not backported down to older versions. it makes sense to do it that way
than the rock solid stable release). The rock solid stable release has been field proven, should have the absolute confidence of system administrators and be viable for multiple years (in other words you can install it and be confident once it's put in production you're good to go for several years.
rubbish. sorry, but the number of times I've had to upgrade eg BIND, ISC DHCPD, apache etc from 'stable' release due to a bug, or that months new vulnerability. i dont know of ANY server daemon that could be installed and left online for years - even if there was such a mythical beast, the server it runs on needs to be reloaded at least 4 times a year due to kernel/OS vulnerabilities too ;-)
security fix. When that occurs the stable release is surgically modified to fix exactly that one issue, it's minor version number is bumped.
thats the wrong way to go as you spend the effort putting features way back down in the tree - and often reverting behaviour already known or fixed. ideally you have a stable version and a test-devel version - which there has been before - however, who runs what? often people run safe and wont run a test-devel version where its needed - and only when the version is 'released' do they then run it and find a bug/issue that no-one else uncovered - this has happened several times this year with 2.1.x
* FreeRADIUS has way too much churn for a critical system service. Think about other system services, how often do you see kerberos, bind, iptables, pam, MySQL, etc. going through significant revisions? Are the administrators of those services constantly being told to upgrade the service because of the bug/feature du jour?
all the time. BIND and MySQL in particular. I'd add OpenSSH and Apache to the list of daemons always being upgraded.
those releases have gotten pushed into production. I think part of the problem is the frequent release schedule (measured in months) and the lack of a coordinated beta testing program. Releases should not occur until after they've successfully navigated a beta program.
I , for one, love having frequent releases of the server - it means we get new features. functions and fixes of behaviour rapidly in what has become a very agile area of network access. if you have a less frequent release you'll have a product with as much functionality as cisco ACS of MS NPS - ie bare and basic. dull and almost useless. of course, this depends on your market - if you are a dial-in user or run an ISP you might want the basic functions. there. done. free. dealt with. if, however, you are using it in an enterprise network environment and are looking at 802.1X with SoH, new rapid policy making, advanced external auditing using custom logging and maybe even wanting to use other RADIUS capabilities, CUI, CoA, linked to that months common backend - or wanting to use enhanced remote access systems like 'moonshot' then you need an agile project - FreeRADIUS is agile enough
* Organize a rigorous beta test program.
using what and who? most issues are found in demanding environments that cannot be modelled by some simulation - this isnt Apache where you can throw a tool at it. there isnt a 'generic config' that all people use. I'll find a bug in postgres that noone else has found because they all use another DB or flatfile.. or if they do use postgres they wont be throwing particular things at it (like storing a local copy of an accounting packet that wasnt received at the remote accounting server and couldnt be stored on sql-relay due to no disk space....) - thats just an example by the way! - by changing to 'stable' and having a remote 'test' release you can pretty much say goodbye to the default presence of leading-tree testers. i know most people will just take the latest stable. they may have 'stable' but they'll whine when they dont have a feature...and when posting to the list i'll say its in 'testing' and they'll ask when thats coming to 'stable - answer? probably never to that stable tree because the code changes to enable that feature are vast and the profiling wont be able to pick out any weird corner cases that might happen to that stable should the functions be backported. as alan said, 1.1.x is 'stable' its also damn bare of any flexible functionality ;-) alan
John Dennis <jdennis@redhat.com> writes:
* FreeRADIUS has way too much churn for a critical system service. Think about other system services, how often do you see kerberos, bind, iptables, pam, MySQL, etc. going through significant revisions? Are the administrators of those services constantly being told to upgrade the service because of the bug/feature du jour?
I think that's unfair. All of the other services you mention have their share of release related problems. The details are too off-topic on this list, but things like having "iptables" completely pulled out and replaced every now-and-then (remember ipchains? Heard of nftables?) springs to mind. Or the wait-forever-for-a-new-stable-release of MySQL. And, although I haven't verified this, I'd be surprised if the RedHat packaged versions of all those services didn't include quite a few bug-fixing patches. There is no such thing as a bugfree release.
* The QE component of FreeRADIUS has proven to be inadequate. I know Alan runs a set of tests and he calls for testing prior to a new release. But we've seen the amount of testing which actually occurs is inadequate because releases have gone out with significant problems and those releases have gotten pushed into production. I think part of the problem is the frequent release schedule (measured in months) and the lack of a coordinated beta testing program. Releases should not occur until after they've successfully navigated a beta program.
I humbly would suggest the following:
* Create and maintain a "stable" version.
* Organize a rigorous beta test program.
I guess no one is going to _oppose_ those suggestions. But both of them are kind of already there. You do have branches which are supposed to be stable. And you do have the call for testing you mention. The reason this is inadequate is probably lack of man power. How many users do actually test the proposed new release on a (near) production system? Not nearly enough, as shown by the bug reports coming in right after a release. Myself, I usually build and install such beta versions on our test system, but we do not really have a complete lab with hundreds of different big NASes and millions of real users. Some problems never show up before we put it on a real production server. Which is kind-of hard to get accepted for a beta test. The FreeRADIUS user mass is probably too low to get anything close to real testing without doing an actual release. Which brings us to the next point:
* Slow down the release schedule, avoid the temptation to cut a new release because of minor new features. If production servers can't run successfully without a feature that's an indication the prior release was too hasty. Critical bug fixes should occur in the release branch and the release branch re-released. The release interval for a system service like FreeRADIUS should be measured in years, not months or weeks.
I don't thing the 2.1.x history looks too bad. Except for a couple of extra bugfix releases early in the cycle, it's been 3-4-5 months between them: release_2_1_0: 2008-09-05 15:27:57 +0200 release_2_1_1: 2008-09-25 10:41:26 +0200 release_2_1_2: 2008-12-04 10:50:29 +0100 release_2_1_3: 2008-12-05 17:37:56 +0100 release_2_1_4: 2009-03-11 03:26:50 +0100 release_2_1_6: 2009-05-18 13:12:54 +0200 release_2_1_7: 2009-09-14 16:43:29 +0200 release_2_1_8: 2009-12-30 16:44:35 +0100 release_2_1_9: 2010-05-24 07:40:58 +0200 release_2_1_10: 2010-09-28 13:03:56 +0200 release_2_1_11: 2011-06-20 16:57:14 +0200
Comments? Thoughts? Do you agree/disagree?
I don't think discussing strategies is very useful given the obvious lack of contributions to the project. It's not like there are resources which could/should be (re)allocated anywhere. All available resources are already used in an extremely efficient way. There are just too few of them to do all that you want. But the project could most certainly need more contributors. Why not create and maintain a stable branch, like the stable kernel branches? Like the kernel, I don't think that's really a job for the main developer(s). It's more of a janitor job. Or why not improve documentation? The new Wiki opens possibilities for anyone to contribute again. And for beta testing: It's pretty obvious that the most important missing part are the testers. Bring them in. The positions are open... In my experience the FreeRADIUS project has never ever turned down any offer to help. A simple task, which doesn't even require any longterm commitment, is just squashing a few bugs the next time there is a call for testing. Just my humble thoughts, as a fellow FreeRADIUS user. Bjørn
On Tue, Aug 2, 2011 at 2:27 PM, Amir Tal <amir@ccc.co.il> wrote:
Upgraded freeradius to 2.1.11 (built from source)
This time it brought the whole machine down:
Notice how I DIDN'T suggest upgrading to 2.1.11, but to v2.1.x of git branch? There's a reason for that, and you just found out the hard way. -- Fajar
This is a production server, What is the latest stable version to use? Amir. -----Original Message----- From: freeradius-users-bounces+amir=ccc.co.il@lists.freeradius.org [mailto:freeradius-users-bounces+amir=ccc.co.il@lists.freeradius.org] On Behalf Of Fajar A. Nugraha Sent: Tuesday, August 02, 2011 11:02 AM To: FreeRadius users mailing list Subject: Re: segmentation fault freeradius 2.1.7 using rlm_sql On Tue, Aug 2, 2011 at 2:27 PM, Amir Tal <amir@ccc.co.il> wrote:
Upgraded freeradius to 2.1.11 (built from source)
This time it brought the whole machine down:
Notice how I DIDN'T suggest upgrading to 2.1.11, but to v2.1.x of git branch? There's a reason for that, and you just found out the hard way. -- Fajar - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On 2 Aug 2011, at 12:07, Amir Tal wrote:
This is a production server, What is the latest stable version to use?
2.1.x head, it'll soon be tagged as 2.1.12. All the dangerous stuff goes on in the master branch, the 2.1.X branch is relatively stable. -Arran
Amir.
-----Original Message----- From: freeradius-users-bounces+amir=ccc.co.il@lists.freeradius.org [mailto:freeradius-users-bounces+amir=ccc.co.il@lists.freeradius.org] On Behalf Of Fajar A. Nugraha Sent: Tuesday, August 02, 2011 11:02 AM To: FreeRadius users mailing list Subject: Re: segmentation fault freeradius 2.1.7 using rlm_sql
On Tue, Aug 2, 2011 at 2:27 PM, Amir Tal <amir@ccc.co.il> wrote:
Upgraded freeradius to 2.1.11 (built from source)
This time it brought the whole machine down:
Notice how I DIDN'T suggest upgrading to 2.1.11, but to v2.1.x of git branch? There's a reason for that, and you just found out the hard way.
-- Fajar - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Arran Cudbard-Bell a.cudbardb@freeradius.org RADIUS - Half the complexity of Diameter
participants (7)
-
Alan Buxey -
Alan DeKok -
Amir Tal -
Arran Cudbard-Bell -
Bjørn Mork -
Fajar A. Nugraha -
John Dennis