Hi, I am running freeradius (2.1.8) with rlm_perl (5.10.1, USE_ITHREADS) on a Debian-Lenny system. The problem is radius fails with segfault – periodically and intermittently. I have no way to reliably reproduce the problem – it happens only in production, and it is impossible to reliably predict when or backtrace why. It seems that I am running into some kind of memory allocation error. Coredump type #3 (see below) is the most "popular" one; coredumps with backtrace going into perl seem to be rather random (it fails in different parts of libperl.so) - again, see below. I understand that freeradius has a newer version available - but I am hesitant to upgrade a production server without a very good reason. And I could not find such reason for an upgrade after reading the CHANGELOG for 2.1.10. But maybe I am wrong? Any ideas? Thank you! PS: bt for coredump #1: #0 0xb47d1ab6 in mysql_st_execute (sth=0xaf13a1a8, imp_sth=0xadf537a8) at dbdimp.c:3209 #1 0xb47da215 in XS_DBD__mysql__st_execute (my_perl=0xb831a08, cv=0xb1e16858) at mysql.xsi:588 #2 0xb70740ab in XS_DBI_dispatch () from /opt/server/lib/perl5/i686-linux-thread-multi/auto/DBI/DBI.so #3 0xb76a841e in Perl_pp_entersub () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #4 0xb76a6841 in Perl_runops_standard () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #5 0xb7641500 in Perl_call_sv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #6 0xb7642284 in Perl_call_pv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #7 0xb72026ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so #8 0x08063524 in modcall () #9 0x0805ffe7 in indexed_modcall () #10 0x080602fc in module_accounting () #11 0x0804e9d1 in rad_accounting () #12 0x0806df65 in radius_handle_request () #13 0x080661d0 in request_handler_thread () #14 0xb759c4c0 in start_thread () from /lib/i686/cmov/libpthread.so.0 #15 0xb734384e in clone () from /lib/i686/cmov/libc.so.6 bt for coredump #2: #0 0xb7690c80 in Perl_pad_undef () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #1 0xb764db67 in Perl_cv_undef () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #2 0xb76dcf5a in Perl_sv_clear () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #3 0xb76dd288 in Perl_sv_free2 () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #4 0xb76dd391 in Perl_sv_free () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #5 0xb76dcd57 in Perl_sv_clear () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #6 0xb76dd288 in Perl_sv_free2 () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #7 0xb70af022 in XS_Sys__Syslog_closelog_xs () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Sys/Syslog/Syslog.so #8 0xb76c941e in Perl_pp_entersub () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #9 0xb76c7841 in Perl_runops_standard () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #10 0xb7662500 in Perl_call_sv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #11 0xb7663284 in Perl_call_pv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #12 0xb72236ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so #13 0x08063524 in modcall () #14 0x0805ffe7 in indexed_modcall () #15 0x0806037c in module_authenticate () #16 0x0804f908 in rad_authenticate () #17 0x0806df65 in radius_handle_request () #18 0x080661d0 in request_handler_thread () #19 0xb75bd4c0 in start_thread () from /lib/i686/cmov/libpthread.so.0 #20 0xb736484e in clone () from /lib/i686/cmov/libc.so.6 bt for coredump #3: #0 0xb77a2424 in __kernel_vsyscall () #1 0xb72d6640 in raise () from /lib/i686/cmov/libc.so.6 #2 0xb72d8018 in abort () from /lib/i686/cmov/libc.so.6 #3 0xb731348d in ?? () from /lib/i686/cmov/libc.so.6 #4 0x00000005 in ?? () #5 0xb583f614 in ?? () #6 0x00000400 in ?? () #7 0xb73e97c8 in ?? () from /lib/i686/cmov/libc.so.6 #8 0x00000017 in ?? () #9 0xbfd8b66b in ?? () #10 0x00000020 in ?? () #11 0xb73e97e1 in ?? () from /lib/i686/cmov/libc.so.6 #12 0x00000002 in ?? () #13 0xb73e985c in ?? () from /lib/i686/cmov/libc.so.6 #14 0x00000023 in ?? () #15 0xb73e97e5 in ?? () from /lib/i686/cmov/libc.so.6 #16 0x00000004 in ?? () #17 0xb583fb43 in ?? () #18 0x00000008 in ?? () #19 0xb73e97eb in ?? () from /lib/i686/cmov/libc.so.6 #20 0x00000005 in ?? () #21 0xb72eb388 in vfprintf () from /lib/i686/cmov/libc.so.6 #22 0xb7319764 in ?? () from /lib/i686/cmov/libc.so.6 #23 0x00000002 in ?? () #24 0xb73e97c8 in ?? () from /lib/i686/cmov/libc.so.6 #25 0xbfd8b66b in ?? () #26 0xb73e985c in ?? () from /lib/i686/cmov/libc.so.6 #27 0xb583fb43 in ?? () #28 0xb73e985c in ?? () from /lib/i686/cmov/libc.so.6 #29 0x3079d6d0 in ?? () #30 0x61346564 in ?? () #31 0x00383132 in ?? () #32 0xb7401ff4 in ?? () from /lib/i686/cmov/libc.so.6 #33 0xb7403160 in ?? () from /lib/i686/cmov/libc.so.6 #34 0x0de4a218 in ?? () #35 0xb583fb78 in ?? () #36 0xb731b966 in free () from /lib/i686/cmov/libc.so.6 Backtrace stopped: frame did not save the PC bt for coredump #4: #0 0xb7673132 in Perl_av_fetch () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #1 0xb708c788 in DD_dump () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Data/Dumper/Dumper.so #2 0xb708d77f in DD_dump () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Data/Dumper/Dumper.so #3 0xb7090219 in XS_Data__Dumper_Dumpxs () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Data/Dumper/Dumper.so #4 0xb767541e in Perl_pp_entersub () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #5 0xb7673841 in Perl_runops_standard () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #6 0xb760e500 in Perl_call_sv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #7 0xb760f284 in Perl_call_pv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #8 0xb71cf6ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so #9 0x08063524 in modcall () #10 0x0805ffe7 in indexed_modcall () #11 0x0806037c in module_authenticate () #12 0x0804f908 in rad_authenticate () #13 0x0806df65 in radius_handle_request () #14 0x080661d0 in request_handler_thread () #15 0xb75694c0 in start_thread () from /lib/i686/cmov/libpthread.so.0 #16 0xb731084e in clone () from /lib/i686/cmov/libc.so.6
On 2011/01/04 09:59 PM, Anatoly Ivanov wrote:
Hi,
I am running freeradius (2.1.8) with rlm_perl (5.10.1, USE_ITHREADS) on a Debian-Lenny system.
The problem is radius fails with segfault – periodically and intermittently. I have no way to reliably reproduce the problem – it happens only in production, and it is impossible to reliably predict when or backtrace why.
It seems that I am running into some kind of memory allocation error.
Coredump type #3 (see below) is the most "popular" one; coredumps with backtrace going into perl seem to be rather random (it fails in different parts of libperl.so) - again, see below.
I understand that freeradius has a newer version available - but I am hesitant to upgrade a production server without a very good reason. And I could not find such reason for an upgrade after reading the CHANGELOG for 2.1.10. But maybe I am wrong?
Any ideas?
A complete gut feel after reading this says you have a hardware problem - faulty ram. Has this happened from the beginning or suddenly now? You can try memtesting (http://www.memtest.org/) the server, or a trick that I've found works sometimes (if you can't take the server out of production) to show a ram problem, is to compile a kernel. I've seen compilation fail at different stages with faulty RAM. I realiuse the advice might sound ridiculous, but it has worked for me before. -- Johan Meiring Cape PC Services CC Tel: (021) 883-8271 Fax: (021) 886-7782
Anatoly Ivanov <anatoly.ivanov@gmail.com> wrote:
I am running freeradius (2.1.8) with rlm_perl (5.10.1, USE_ITHREADS) on a Debian-Lenny system.
The problem is radius fails with segfault ??? periodically and intermittently. I have no way to reliably reproduce the problem ??? it happens only in production, and it is impossible to reliably predict when or backtrace why.
Stab in the dark, is your Perl code actually thread safe...including *all* the Perl modules you use? Seems DBD::MySQL is safe (that actually surprises me): http://search.cpan.org/~capttofu/DBD-mysql-4.018/lib/DBD/mysql.pm#MULTITHREA... Are you using any global variables in your script, and if so have you protected them from concurrent access? The really quick test is to tell freeradius not to enable threading to your module and comment out 'use threads(::shared)?;' in your code.
It seems that I am running into some kind of memory allocation error.
Coredump type #3 (see below) is the most "popular" one; coredumps with backtrace going into perl seem to be rather random (it fails in different parts of libperl.so) - again, see below.
Looks like if you install some more debugging libraries (libc6-dbg and any others you think might help, looks like you have already installed perl-debug) a few more blanks will be filled in. Cheers -- Alexander Clouter .sigmonster says: Avoid gunfire in the bathroom tonight.
On Jan 4, 2011, at 9:59 PM, Anatoly Ivanov wrote:
Hi,
I am running freeradius (2.1.8) with rlm_perl (5.10.1, USE_ITHREADS) on a Debian-Lenny system.
The problem is radius fails with segfault – periodically and intermittently. I have no way to reliably reproduce the problem – it happens only in production, and it is impossible to reliably predict when or backtrace why.
This is have to be thread problem. Use CLONE function in your perl code (this is executed per every new thread) to initialize per thread variables and DBI Connections.
It seems that I am running into some kind of memory allocation error.
Coredump type #3 (see below) is the most "popular" one; coredumps with backtrace going into perl seem to be rather random (it fails in different parts of libperl.so) - again, see below.
If you want to debug - search this list or google using keywords "debug rlm_perl"
I understand that freeradius has a newer version available - but I am hesitant to upgrade a production server without a very good reason. And I could not find such reason for an upgrade after reading the CHANGELOG for 2.1.10. But maybe I am wrong?
Any ideas?
Thank you!
Best Regards, Boian Jordanov Head of Voice Department tel. +359 2 4004 723 tel. +359 2 4004 002
participants (4)
-
Alexander Clouter -
Anatoly Ivanov -
Boian Jordanov -
Johan Meiring