rlm_perl.c questions

Fri Jan 7 00:06:50 CET 2011

Boian,

Thanks for the explanation.

I am rather limited in what I can do on a production system, so it is not
(administratively) easy to re-compile and re-deploy the code.

Right now I have configured the server to never kill a thread from
threadpool - by specifying max_spare_servers=max_servers. The server has not
crashed for two days now, but that may be a coincidence - low load during
the holidays helps with the stability. I should be able to tell for sure
next week.

I have a bunch of backtraces. Please let me know if you see anything
suspicious in there.
bt #1:

#0  0xb47d1ab6 in mysql_st_execute (sth=0xaf13a1a8, imp_sth=0xadf537a8) at
dbdimp.c:3209
#1  0xb47da215 in XS_DBD__mysql__st_execute (my_perl=0xb831a08,
cv=0xb1e16858) at mysql.xsi:588
#2  0xb70740ab in XS_DBI_dispatch () from
/opt/server/lib/perl5/i686-linux-thread-multi/auto/DBI/DBI.so
 #3  0xb76a841e in Perl_pp_entersub () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#4  0xb76a6841 in Perl_runops_standard () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#5  0xb7641500 in Perl_call_sv () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#6  0xb7642284 in Perl_call_pv () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#7  0xb72026ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so
#8  0x08063524 in modcall ()
#9  0x0805ffe7 in indexed_modcall ()
#10 0x080602fc in module_accounting ()
#11 0x0804e9d1 in rad_accounting ()
#12 0x0806df65 in radius_handle_request ()
#13 0x080661d0 in request_handler_thread ()
#14 0xb759c4c0 in start_thread () from /lib/i686/cmov/libpthread.so.0
#15 0xb734384e in clone () from /lib/i686/cmov/libc.so.6

bt #2:

#0  0xb7690c80 in Perl_pad_undef () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#1  0xb764db67 in Perl_cv_undef () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#2  0xb76dcf5a in Perl_sv_clear () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#3  0xb76dd288 in Perl_sv_free2 () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#4  0xb76dd391 in Perl_sv_free () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#5  0xb76dcd57 in Perl_sv_clear () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#6  0xb76dd288 in Perl_sv_free2 () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#7  0xb70af022 in XS_Sys__Syslog_closelog_xs () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Sys/Syslog/Syslog.so
#8  0xb76c941e in Perl_pp_entersub () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#9  0xb76c7841 in Perl_runops_standard () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#10 0xb7662500 in Perl_call_sv () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#11 0xb7663284 in Perl_call_pv () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#12 0xb72236ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so
#13 0x08063524 in modcall ()
#14 0x0805ffe7 in indexed_modcall ()
#15 0x0806037c in module_authenticate ()
#16 0x0804f908 in rad_authenticate ()
#17 0x0806df65 in radius_handle_request ()
#18 0x080661d0 in request_handler_thread ()
#19 0xb75bd4c0 in start_thread () from /lib/i686/cmov/libpthread.so.0
#20 0xb736484e in clone () from /lib/i686/cmov/libc.so.6

bt#3:

#0  0xb77a2424 in __kernel_vsyscall ()
#1  0xb72d6640 in raise () from /lib/i686/cmov/libc.so.6
#2  0xb72d8018 in abort () from /lib/i686/cmov/libc.so.6
#3  0xb731348d in __libc_message () from /lib/i686/cmov/libc.so.6
#4  0xb7319764 in malloc_printerr () from /lib/i686/cmov/libc.so.6
#5  0xb731b966 in free () from /lib/i686/cmov/libc.so.6
#6  0xb76d81b1 in Perl_safesysfree () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#7  0xb77040a8 in Perl_sv_clear () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#8  0xb7704288 in Perl_sv_free2 () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#9  0xb70d6022 in XS_Sys__Syslog_closelog_xs () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Sys/Syslog/Syslog.so
#10 0xb76f041e in Perl_pp_entersub () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#11 0xb76ee841 in Perl_runops_standard () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#12 0xb7689500 in Perl_call_sv () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#13 0xb768a284 in Perl_call_pv () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#14 0xb724a6ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so
#15 0x08063524 in modcall ()
#16 0x0805ffe7 in indexed_modcall ()
#17 0x0806037c in module_authenticate ()
#18 0x0804f908 in rad_authenticate ()
#19 0x0806df65 in radius_handle_request ()
#20 0x080661d0 in request_handler_thread ()
#21 0xb75e44c0 in start_thread () from /lib/i686/cmov/libpthread.so.0
#22 0xb738b84e in clone () from /lib/i686/cmov/libc.so.6

bt #4

#0  0xb7673132 in Perl_av_fetch () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#1  0xb708c788 in DD_dump () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Data/Dumper/Dumper.so
#2  0xb708d77f in DD_dump () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Data/Dumper/Dumper.so
#3  0xb7090219 in XS_Data__Dumper_Dumpxs () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Data/Dumper/Dumper.so
#4  0xb767541e in Perl_pp_entersub () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#5  0xb7673841 in Perl_runops_standard () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#6  0xb760e500 in Perl_call_sv () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#7  0xb760f284 in Perl_call_pv () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#8  0xb71cf6ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so
#9  0x08063524 in modcall ()
#10 0x0805ffe7 in indexed_modcall ()
#11 0x0806037c in module_authenticate ()
#12 0x0804f908 in rad_authenticate ()
#13 0x0806df65 in radius_handle_request ()
#14 0x080661d0 in request_handler_thread ()
#15 0xb75694c0 in start_thread () from /lib/i686/cmov/libpthread.so.0
#16 0xb731084e in clone () from /lib/i686/cmov/libc.so.6

bt #5:

#0  0xb771ef77 in Perl_newSVsv () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#1  0xb7712020 in Perl_pp_aassign () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#2  0xb770a841 in Perl_runops_standard () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#3  0xb76a5500 in Perl_call_sv () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#4  0xb76a6284 in Perl_call_pv () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#5  0xb72666ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so
#6  0x08063524 in modcall ()
#7  0x0805ffe7 in indexed_modcall ()
#8  0x0806037c in module_authenticate ()
#9  0x0804f908 in rad_authenticate ()
#10 0x0806df65 in radius_handle_request ()
#11 0x080661d0 in request_handler_thread ()
#12 0xb76004c0 in start_thread () from /lib/i686/cmov/libpthread.so.0
#13 0xb73a784e in clone () from /lib/i686/cmov/libc.so.6

bt #6:
#0  0xb76f4424 in __kernel_vsyscall ()
#1  0xb7228640 in raise () from /lib/i686/cmov/libc.so.6
#2  0xb722a018 in abort () from /lib/i686/cmov/libc.so.6
#3  0xb726548d in __libc_message () from /lib/i686/cmov/libc.so.6
#4  0xb726b764 in malloc_printerr () from /lib/i686/cmov/libc.so.6
#5  0xb726d966 in free () from /lib/i686/cmov/libc.so.6
#6  0xb762a1b1 in Perl_safesysfree () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#7  0xb76560a8 in Perl_sv_clear () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#8  0xb7656288 in Perl_sv_free2 () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#9  0xb7028022 in XS_Sys__Syslog_closelog_xs () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Sys/Syslog/Syslog.so
#10 0xb764241e in Perl_pp_entersub () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#11 0xb7640841 in Perl_runops_standard () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#12 0xb75db500 in Perl_call_sv () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#13 0xb75dc284 in Perl_call_pv () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so
#14 0xb719c6ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so
#15 0x08063524 in modcall ()
#16 0x0805ffe7 in indexed_modcall ()
#17 0x0806037c in module_authenticate ()
#18 0x0804f908 in rad_authenticate ()
#19 0x0806df65 in radius_handle_request ()
#20 0x080661d0 in request_handler_thread ()
#21 0xb75364c0 in start_thread () from /lib/i686/cmov/libpthread.so.0
#22 0xb72dd84e in clone () from /lib/i686/cmov/libc.so.6

Thanks,
Anatoly.

On Thu, Jan 6, 2011 at 1:26 AM, Boian Jordanov <bjordanov at orbitel.bg> wrote:

>
> On Jan 6, 2011, at 1:51 AM, Anatoly Ivanov wrote:
>
> > Hi everybody,
> >
> > I have a couple of questions regarding modules/rlm_perl/rlm_perl.c:
> > perl_detach(...):
> >
> > 1.  Is it correct to call PERL_SYS_TERM in detach()?
> >
> > I am experiencing quite random and unpredictable segfaults with radius
> 2.1.8
> > and rlm_perl, and it really looks like memory allocation/multithreading
> > error. I have very similar system running radius 2.1.7, and it works just
> > fine. The most significant difference in rlm_perl between 2.1.7 and 2.1.8
> is
> > that PERL_SYS_TERM call.
> >
> > According to the documentation
> > (http://perldoc.perl.org/perlembed.html), PERL_SYS_TERM
> > should be called only once. In our case, it is called once per detaching
> > thread.
>
> In fact it is called only once in the end of life of module. You can see
> that when a thread is destroyed is called
>
> /* Create Key */
> static void rlm_perl_make_key(pthread_key_t *key)
> {
>        pthread_key_create(key, rlm_destroy_perl);
> }
>
> rlm_destroy_perl which destroy only particular interp
>
> So the best will be to produce gdb trace file as i already wrote in my
> previous mail.
>
>
> >
> > 2. This code:
> >
> > embed = rad_malloc(4*(sizeof(char *)));
> > memset(embed, 0, sizeof(4*(sizeof(char *))));
> >
> > looks like a misprint to me. Not that it would cause any real errors, but
> it
> > is inaccurate anyway.
> >
> > Thank you,
> > Anatoly.
> > -
> > List info/subscribe/unsubscribe? See
> http://www.freeradius.org/list/devel.html
>
> Best Regards,
> Boian Jordanov
> Head of Voice Department
> tel. +359 2 4004 723
> tel. +359 2 4004 002
>
>
>
>
>
>
>
> -
> List info/subscribe/unsubscribe? See
> http://www.freeradius.org/list/devel.html
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freeradius.org/pipermail/freeradius-devel/attachments/20110106/65a81dfb/attachment.html>