Boian,<div><br></div><div>Thanks for the explanation.</div><div><br></div><div>I am rather limited in what I can do on a production system, so it is not (administratively) easy to re-compile and re-deploy the code.</div><div>
<br></div><div>Right now I have configured the server to never kill a thread from threadpool - by specifying max_spare_servers=max_servers. The server has not crashed for two days now, but that may be a coincidence - low load during the holidays helps with the stability. I should be able to tell for sure next week.</div>
<div><br></div><div>I have a bunch of backtraces. Please let me know if you see anything suspicious in there.</div><div><div>bt #1:</div><div><br></div><div>#0 0xb47d1ab6 in mysql_st_execute (sth=0xaf13a1a8, imp_sth=0xadf537a8) at dbdimp.c:3209</div>
<div>#1 0xb47da215 in XS_DBD__mysql__st_execute (my_perl=0xb831a08, cv=0xb1e16858) at mysql.xsi:588</div><div>#2 0xb70740ab in XS_DBI_dispatch () from /opt/server/lib/perl5/i686-linux-thread-multi/auto/DBI/DBI.so</div>
<div>
#3 0xb76a841e in Perl_pp_entersub () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#4 0xb76a6841 in Perl_runops_standard () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>
<div>#5 0xb7641500 in Perl_call_sv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#6 0xb7642284 in Perl_call_pv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>
<div>#7 0xb72026ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so</div><div>#8 0x08063524 in modcall ()</div><div>#9 0x0805ffe7 in indexed_modcall ()</div><div>#10 0x080602fc in module_accounting ()</div><div>
#11 0x0804e9d1 in rad_accounting ()</div><div>#12 0x0806df65 in radius_handle_request ()</div><div>#13 0x080661d0 in request_handler_thread ()</div><div>#14 0xb759c4c0 in start_thread () from /lib/i686/cmov/libpthread.so.0</div>
<div>#15 0xb734384e in clone () from /lib/i686/cmov/libc.so.6</div><div><br></div><div>bt #2:</div><div><br></div><div>#0 0xb7690c80 in Perl_pad_undef () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>
<div>#1 0xb764db67 in Perl_cv_undef () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#2 0xb76dcf5a in Perl_sv_clear () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>
<div>#3 0xb76dd288 in Perl_sv_free2 () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#4 0xb76dd391 in Perl_sv_free () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>
<div>#5 0xb76dcd57 in Perl_sv_clear () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#6 0xb76dd288 in Perl_sv_free2 () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>
<div>#7 0xb70af022 in XS_Sys__Syslog_closelog_xs () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Sys/Syslog/Syslog.so</div><div>#8 0xb76c941e in Perl_pp_entersub () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>
<div>#9 0xb76c7841 in Perl_runops_standard () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#10 0xb7662500 in Perl_call_sv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>
<div>#11 0xb7663284 in Perl_call_pv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#12 0xb72236ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so</div><div>#13 0x08063524 in modcall ()</div>
<div>#14 0x0805ffe7 in indexed_modcall ()</div><div>#15 0x0806037c in module_authenticate ()</div><div>#16 0x0804f908 in rad_authenticate ()</div><div>#17 0x0806df65 in radius_handle_request ()</div><div>#18 0x080661d0 in request_handler_thread ()</div>
<div>#19 0xb75bd4c0 in start_thread () from /lib/i686/cmov/libpthread.so.0</div><div>#20 0xb736484e in clone () from /lib/i686/cmov/libc.so.6</div><div><br></div><div><br></div><div>bt#3:</div><div><br></div><div>#0 0xb77a2424 in __kernel_vsyscall ()</div>
<div>#1 0xb72d6640 in raise () from /lib/i686/cmov/libc.so.6</div><div>#2 0xb72d8018 in abort () from /lib/i686/cmov/libc.so.6</div><div>#3 0xb731348d in __libc_message () from /lib/i686/cmov/libc.so.6</div><div>#4 0xb7319764 in malloc_printerr () from /lib/i686/cmov/libc.so.6</div>
<div>#5 0xb731b966 in free () from /lib/i686/cmov/libc.so.6</div><div>#6 0xb76d81b1 in Perl_safesysfree () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#7 0xb77040a8 in Perl_sv_clear () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>
<div>#8 0xb7704288 in Perl_sv_free2 () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#9 0xb70d6022 in XS_Sys__Syslog_closelog_xs () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Sys/Syslog/Syslog.so</div>
<div>#10 0xb76f041e in Perl_pp_entersub () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#11 0xb76ee841 in Perl_runops_standard () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>
<div>#12 0xb7689500 in Perl_call_sv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#13 0xb768a284 in Perl_call_pv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>
<div>#14 0xb724a6ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so</div><div>#15 0x08063524 in modcall ()</div><div>#16 0x0805ffe7 in indexed_modcall ()</div><div>#17 0x0806037c in module_authenticate ()</div>
<div>#18 0x0804f908 in rad_authenticate ()</div><div>#19 0x0806df65 in radius_handle_request ()</div><div>#20 0x080661d0 in request_handler_thread ()</div><div>#21 0xb75e44c0 in start_thread () from /lib/i686/cmov/libpthread.so.0</div>
<div>#22 0xb738b84e in clone () from /lib/i686/cmov/libc.so.6</div><div><br></div><div>bt #4</div><div><br></div><div>#0 0xb7673132 in Perl_av_fetch () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>
<div>#1 0xb708c788 in DD_dump () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Data/Dumper/Dumper.so</div><div>#2 0xb708d77f in DD_dump () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Data/Dumper/Dumper.so</div>
<div>#3 0xb7090219 in XS_Data__Dumper_Dumpxs () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Data/Dumper/Dumper.so</div><div>#4 0xb767541e in Perl_pp_entersub () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>
<div>#5 0xb7673841 in Perl_runops_standard () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#6 0xb760e500 in Perl_call_sv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>
<div>#7 0xb760f284 in Perl_call_pv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#8 0xb71cf6ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so</div><div>#9 0x08063524 in modcall ()</div>
<div>#10 0x0805ffe7 in indexed_modcall ()</div><div>#11 0x0806037c in module_authenticate ()</div><div>#12 0x0804f908 in rad_authenticate ()</div><div>#13 0x0806df65 in radius_handle_request ()</div><div>#14 0x080661d0 in request_handler_thread ()</div>
<div>#15 0xb75694c0 in start_thread () from /lib/i686/cmov/libpthread.so.0</div><div>#16 0xb731084e in clone () from /lib/i686/cmov/libc.so.6</div><div><br></div><div>bt #5:</div><div><br></div><div>#0 0xb771ef77 in Perl_newSVsv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>
<div>#1 0xb7712020 in Perl_pp_aassign () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#2 0xb770a841 in Perl_runops_standard () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>
<div>#3 0xb76a5500 in Perl_call_sv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#4 0xb76a6284 in Perl_call_pv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>
<div>#5 0xb72666ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so</div><div>#6 0x08063524 in modcall ()</div><div>#7 0x0805ffe7 in indexed_modcall ()</div><div>#8 0x0806037c in module_authenticate ()</div>
<div>#9 0x0804f908 in rad_authenticate ()</div><div>#10 0x0806df65 in radius_handle_request ()</div><div>#11 0x080661d0 in request_handler_thread ()</div><div>#12 0xb76004c0 in start_thread () from /lib/i686/cmov/libpthread.so.0</div>
<div>#13 0xb73a784e in clone () from /lib/i686/cmov/libc.so.6</div><div><br></div><div>bt #6:</div><div>#0 0xb76f4424 in __kernel_vsyscall ()</div><div>#1 0xb7228640 in raise () from /lib/i686/cmov/libc.so.6</div><div>
#2 0xb722a018 in abort () from /lib/i686/cmov/libc.so.6</div>
<div>#3 0xb726548d in __libc_message () from /lib/i686/cmov/libc.so.6</div><div>#4 0xb726b764 in malloc_printerr () from /lib/i686/cmov/libc.so.6</div><div>#5 0xb726d966 in free () from /lib/i686/cmov/libc.so.6</div><div>
#6 0xb762a1b1 in Perl_safesysfree () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#7 0xb76560a8 in Perl_sv_clear () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>
<div>#8 0xb7656288 in Perl_sv_free2 () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#9 0xb7028022 in XS_Sys__Syslog_closelog_xs () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Sys/Syslog/Syslog.so</div>
<div>#10 0xb764241e in Perl_pp_entersub () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#11 0xb7640841 in Perl_runops_standard () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>
<div>#12 0xb75db500 in Perl_call_sv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#13 0xb75dc284 in Perl_call_pv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>
<div>#14 0xb719c6ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so</div><div>#15 0x08063524 in modcall ()</div><div>#16 0x0805ffe7 in indexed_modcall ()</div><div>#17 0x0806037c in module_authenticate ()</div>
<div>#18 0x0804f908 in rad_authenticate ()</div><div>#19 0x0806df65 in radius_handle_request ()</div><div>#20 0x080661d0 in request_handler_thread ()</div><div>#21 0xb75364c0 in start_thread () from /lib/i686/cmov/libpthread.so.0</div>
<div>#22 0xb72dd84e in clone () from /lib/i686/cmov/libc.so.6</div><div><br></div></div><div>Thanks,</div><div>Anatoly.</div><div><br><br><div class="gmail_quote">On Thu, Jan 6, 2011 at 1:26 AM, Boian Jordanov <span dir="ltr"><<a href="mailto:bjordanov@orbitel.bg" target="_blank">bjordanov@orbitel.bg</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><br>
On Jan 6, 2011, at 1:51 AM, Anatoly Ivanov wrote:<br>
<br>
</div><div>> Hi everybody,<br>
><br>
> I have a couple of questions regarding modules/rlm_perl/rlm_perl.c:<br>
> perl_detach(...):<br>
><br>
> 1. Is it correct to call PERL_SYS_TERM in detach()?<br>
><br>
> I am experiencing quite random and unpredictable segfaults with radius 2.1.8<br>
> and rlm_perl, and it really looks like memory allocation/multithreading<br>
> error. I have very similar system running radius 2.1.7, and it works just<br>
> fine. The most significant difference in rlm_perl between 2.1.7 and 2.1.8 is<br>
> that PERL_SYS_TERM call.<br>
><br>
> According to the documentation<br>
> (<a href="http://perldoc.perl.org/perlembed.html" target="_blank">http://perldoc.perl.org/perlembed.html</a>), PERL_SYS_TERM<br>
> should be called only once. In our case, it is called once per detaching<br>
> thread.<br>
<br>
</div>In fact it is called only once in the end of life of module. You can see that when a thread is destroyed is called<br>
<br>
/* Create Key */<br>
static void rlm_perl_make_key(pthread_key_t *key)<br>
{<br>
pthread_key_create(key, rlm_destroy_perl);<br>
}<br>
<br>
rlm_destroy_perl which destroy only particular interp<br>
<br>
So the best will be to produce gdb trace file as i already wrote in my previous mail.<br>
<div><br>
<br>
><br>
> 2. This code:<br>
><br>
> embed = rad_malloc(4*(sizeof(char *)));<br>
> memset(embed, 0, sizeof(4*(sizeof(char *))));<br>
><br>
> looks like a misprint to me. Not that it would cause any real errors, but it<br>
> is inaccurate anyway.<br>
><br>
> Thank you,<br>
> Anatoly.<br>
</div><div><div></div><div>> -<br>
> List info/subscribe/unsubscribe? See <a href="http://www.freeradius.org/list/devel.html" target="_blank">http://www.freeradius.org/list/devel.html</a><br>
<br>
Best Regards,<br>
Boian Jordanov<br>
Head of Voice Department<br>
tel. +359 2 4004 723<br>
tel. +359 2 4004 002<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
-<br>
List info/subscribe/unsubscribe? See <a href="http://www.freeradius.org/list/devel.html" target="_blank">http://www.freeradius.org/list/devel.html</a><br>
</div></div></blockquote></div><br></div>