I'm beginning the process of replacing a home-grown RADIUS server with freeradius, a good idea on many many fronts. The server will interact with our backend databases in order to determine attributes to inject into the access-accept messages. For initial development, I've begun work on a perl script that is interacting with freeradius 2.1.10. I'm a little alarmed that a malfunctioning perl script can segfault the entire server. I was hoping that freeradius could compartmentalize a failing script, restarting it as necessary. Admittedly, during development, my scripts are not production-ready, and there are lots of safety checks that simply aren't there. But I'm worried that, going into production, we'll discover an unexpected corner-case that crashes our entire radius infrastructure (not a good thing). Is the best practice simply to make sure you're scripts are bullet-proof? Or is there a more stable method of interacting with an external resource? I chosen rlm_perl because the script didn't need to be instantiated every time (as with rlm_exec), but perhaps we'd be better off relying on rlm_exec if it is more tolerant of corner-case failures. Thanks, Norman
On 09/02/2011 09:43 AM, Norman Elton wrote:
I'm beginning the process of replacing a home-grown RADIUS server with freeradius, a good idea on many many fronts. The server will interact with our backend databases in order to determine attributes to inject into the access-accept messages. For initial development, I've begun work on a perl script that is interacting with freeradius 2.1.10. I'm a little alarmed that a malfunctioning perl script can segfault the entire server. I was hoping that freeradius could compartmentalize a failing script, restarting it as necessary.
Admittedly, during development, my scripts are not production-ready, and there are lots of safety checks that simply aren't there. But I'm worried that, going into production, we'll discover an unexpected corner-case that crashes our entire radius infrastructure (not a good thing).
Is the best practice simply to make sure you're scripts are bullet-proof? Or is there a more stable method of interacting with an external resource? I chosen rlm_perl because the script didn't need to be instantiated every time (as with rlm_exec), but perhaps we'd be better off relying on rlm_exec if it is more tolerant of corner-case failures.
Of course a script error shouldn't segfault the server. It would have been much more useful if you had explained what the script error was and a stack trace from the segfault. -- John Dennis <jdennis@redhat.com> Looking to carve out IT costs? www.redhat.com/carveoutcosts/
Of course a script error shouldn't segfault the server. It would have been much more useful if you had explained what the script error was and a stack trace from the segfault.
Oh, I've experienced lots of them! So many, in fact, that I figured it was a common and well understood occurrence. Let me come up with an easily reproducible example and I'll post the relevant information. Thanks Norman
Hi,
Oh, I've experienced lots of them! So many, in fact, that I figured it was a common and well understood occurrence. Let me come up with an easily reproducible example and I'll post the relevant information.
2.1.11 is out...and 2.1.12 is almost ready for release - does your system behave in the same way with 2.1.11? alan
Alan Buxey <A.L.M.Buxey@lboro.ac.uk> writes:
Oh, I've experienced lots of them! So many, in fact, that I figured it was a common and well understood occurrence. Let me come up with an easily reproducible example and I'll post the relevant information.
2.1.11 is out...and 2.1.12 is almost ready for release - does your system behave in the same way with 2.1.11?
Valid question of course, but do note that there aren't *any* changes to rlm_perl in those versions. Only a few new lines of documentation in the example script: bjorn@canardo:/usr/local/src/git/freeradius$ git diff --stat release_2_1_10..v2.1.x src/modules/rlm_perl/ src/modules/rlm_perl/example.pl | 13 +++++++++++++ 1 files changed, 13 insertions(+), 0 deletions(-) And FWIW, we've been using rlm_perl extensively with 2.1.10 without any segfaults. But then again, that might just be because we write bug free perl code :-) Bjørn
On Fri, Sep 02, 2011 at 07:16:26PM +0200, Bjørn Mork wrote:
Alan Buxey <A.L.M.Buxey@lboro.ac.uk> writes:
Oh, I've experienced lots of them! So many, in fact, that I figured it was a common and well understood occurrence. Let me come up with an easily reproducible example and I'll post the relevant information.
2.1.11 is out...and 2.1.12 is almost ready for release - does your system behave in the same way with 2.1.11?
Are you using a pre-built package for freeradius or one that you have built yourself? Perl can pull in so many different libraries that version differences can result in segfaults. Cheers, Ken
2.1.11 is out...and 2.1.12 is almost ready for release - does your system behave in the same way with 2.1.11?
Are you using a pre-built package for freeradius or one that you have built yourself?
I am using RedHat's pre-built packages, both FreeRADIUS and Perl. I have not tried newer versions, but it should be pretty straightforward to test.
And FWIW, we've been using rlm_perl extensively with 2.1.10 without any segfaults. But then again, that might just be because we write bug free perl code :-)
Oh I have no doubt that people are using rlm_perl trouble-free. I'm just a little concerned that a bug has the capability to crash the entire server. In development, not a big deal. In fact, it encourages good error recovery. But I'd rather not wake up in the middle of the night and find my entire RADIUS infrastructure has died due to an unexpected corner case. Usernames with unicode characters particularly terrify me.
Of course a script error shouldn't segfault the server. It would have been much more useful if you had explained what the script error was and a stack trace from the segfault.
I don't have a stack trace yet, but I've got an easily reproducible test case. This is on RedHat 6, using FreeRadius 2.1.10-5, perl 5.10.1-119, x86_64 architecture. From a fresh install, I cleared out sites-enabled and created a single enabled server: server srv-perl-crash { authorize { preprocess update control { Auth-Type := Accept } perl } authenticate { noop } post-auth { noop } preacct { noop } accounting { noop } } I route localhost to that server: client 127.0.0.1 { shortname = localhost secret = mysecret virtual_server = srv-perl-crash } And I define a very simple example.pl: use strict; use constant RLM_MODULE_OK=> 2;# /* the module is OK, continue */ sub authorize { my $i = 1/0; return RLM_MODULE_OK; } Obviously, a division by zero is a bad thing. But one would expect FreeRadius to stay online. I fire up the server, and test it with: radtest -x foo bar 127.0.0.1 1812 mysecret At first, it gives an error, but survives: rlm_perl: perl_embed:: module = /etc/raddb/example.pl , func = authorize exit status= Illegal division by zero at /etc/raddb/example.pl line 58. I receive an Access-Reject, and things are fine. I send a second request, and I get: rlm_perl: perl_embed:: module = /etc/raddb/example.pl , func = authorize exit status= Illegal division by zero at /etc/raddb/example.pl line 58. Segmentation fault And the server dies. Uh oh! Like I said, I will work on a stack trace. Just wanted to get this out on the list while it's fresh in my mind. Norman
I've used GDB to generate a stack trace, specifically using the instructions on http://freeradius.org/radiusd/doc/bugs. For this particular test case, I configured as I described above, but instead of a stripped-down example.pl, I just the one provided, but put "my $i = 1/0;" in the "test_call" subroutine towards the bottom of the script. This was reproduced using 2.1.11. Let me know if there is additional information I can provide, thanks! Norman ===== Starting program: /usr/local/sbin/radiusd -d /usr/local/etc/raddb -X [Thread debugging using libthread_db enabled] Program received signal SIGSEGV, Segmentation fault. 0x00007ffff5d8a02a in rlmperl_call (instance=<value optimized out>, request=0x8a5b30, function_name=0x781670 "authorize") at rlm_perl.c:725 725 exitstatus = POPi; Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.25.el6_1.3.x86_64 nss-softokn-freebl-3.12.9-3.el6.x86_64 perl-5.10.1-119.el6.x86_64 perl-libs-5.10.1-119.el6.x86_64 * 1 Thread 0x7ffff7fef700 (LWP 29993) 0x00007ffff5d8a02a in rlmperl_call (instance=<value optimized out>, request=0x8a5b30, function_name=0x781670 "authorize") at rlm_perl.c:725 Thread 1 (Thread 0x7ffff7fef700 (LWP 29993)): #0 0x00007ffff5d8a02a in rlmperl_call (instance=<value optimized out>, request=0x8a5b30, function_name=0x781670 "authorize") at rlm_perl.c:725 _sv = 0x411 sp = 0x967f50 inst = <value optimized out> vp = <value optimized out> exitstatus = 0 count = 1 n_a = 70 rad_reply_hv = 0x9519d0 rad_check_hv = 0x908a40 rad_config_hv = 0x0 rad_request_hv = 0x914aa0 rad_request_proxy_hv = 0x942640 rad_request_proxy_reply_hv = 0x951c10 interp = 0x967f50 #1 0x000000000041af53 in call_modsingle (component=1, c=<value optimized out>, request=<value optimized out>) at modcall.c:297 myresult = <value optimized out> #2 modcall (component=1, c=<value optimized out>, request=<value optimized out>) at modcall.c:670 myresult = <value optimized out> stack = {pointer = 1, priority = {0 <repeats 32 times>}, result = {0 <repeats 32 times>}, children = {<value optimized out> <repeats 32 times>}, start = { <value optimized out> <repeats 32 times>}} parent = 0x785c90 child = 0x7b0a80 sp = 0x7b0a80 if_taken = 0 was_if = 0 #3 0x0000000000417b33 in indexed_modcall (comp=1, idx=0, request=0x8a5b30) at modules.c:737 rcode = <value optimized out> list = 0x785c90 server = <value optimized out> #4 0x0000000000408646 in rad_authenticate (request=0x8a5b30) at auth.c:579 namepair = <value optimized out> check_item = <value optimized out> auth_item = 0x8a5d50 module_msg = <value optimized out> tmp = <value optimized out> result = <value optimized out> password = 0x4349da "" autz_retry = 0 '\000' autz_type = <value optimized out> #5 0x000000000042796e in radius_handle_request (request=0x8a5b30, fun=0x4083e0 <rad_authenticate>) at event.c:3780 No locals. #6 0x000000000041ed3d in thread_pool_addrequest (request=0x8a5b30, fun=0x4083e0 <rad_authenticate>) at threads.c:874 No locals. #7 0x0000000000428fee in event_socket_handler (xel=<value optimized out>, fd=<value optimized out>, ctx=0x7b1380) at event.c:3425 listener = 0x7b1380 fun = 0x4083e0 <rad_authenticate> request = 0x8a5b30 #8 0x00007ffff7bd343b in fr_event_loop (el=0x7b1e60) at event.c:413 ef = <value optimized out> i = <value optimized out> rcode = 1 maxfd = 12 when = {tv_sec = 1314989256, tv_usec = 664777} wake = <value optimized out> read_fds = {fds_bits = {1024, 0 <repeats 15 times>}} master_fds = {fds_bits = {7424, 0 <repeats 15 times>}} #9 0x000000000041be24 in main (argc=<value optimized out>, argv=<value optimized out>) at radiusd.c:408 rcode = <value optimized out> argval = <value optimized out> spawn_flag = 0 dont_fork = 1 flag = 0 act = {__sigaction_handler = {sa_handler = 0x41c100 <sig_fatal>, sa_sigaction = 0x41c100 <sig_fatal>}, sa_mask = {__val = {0 <repeats 16 times>}}, sa_flags = 0, sa_restorer = 0}
participants (5)
-
Alan Buxey -
Bjørn Mork -
John Dennis -
ktm＠rice.edu -
Norman Elton