Boian,<div><br></div><div>Thanks for the explanation.</div><div><br></div><div>I am rather limited in what I can do on a production system, so it is not (administratively) easy to re-compile and re-deploy the code.</div><div>

<br></div><div>Right now I have configured the server to never kill a thread from threadpool - by specifying max_spare_servers=max_servers. The server has not crashed for two days now, but that may be a coincidence - low load during the holidays helps with the stability. I should be able to tell for sure next week.</div>

<div><br></div><div>I have a bunch of backtraces. Please let me know if you see anything suspicious in there.</div><div><div>bt #1:</div><div><br></div><div>#0  0xb47d1ab6 in mysql_st_execute (sth=0xaf13a1a8, imp_sth=0xadf537a8) at dbdimp.c:3209</div>


<div>#1  0xb47da215 in XS_DBD__mysql__st_execute (my_perl=0xb831a08, cv=0xb1e16858) at mysql.xsi:588</div><div>#2  0xb70740ab in XS_DBI_dispatch () from /opt/server/lib/perl5/i686-linux-thread-multi/auto/DBI/DBI.so</div>

<div>
#3  0xb76a841e in Perl_pp_entersub () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#4  0xb76a6841 in Perl_runops_standard () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>


<div>#5  0xb7641500 in Perl_call_sv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#6  0xb7642284 in Perl_call_pv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>


<div>#7  0xb72026ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so</div><div>#8  0x08063524 in modcall ()</div><div>#9  0x0805ffe7 in indexed_modcall ()</div><div>#10 0x080602fc in module_accounting ()</div><div>


#11 0x0804e9d1 in rad_accounting ()</div><div>#12 0x0806df65 in radius_handle_request ()</div><div>#13 0x080661d0 in request_handler_thread ()</div><div>#14 0xb759c4c0 in start_thread () from /lib/i686/cmov/libpthread.so.0</div>


<div>#15 0xb734384e in clone () from /lib/i686/cmov/libc.so.6</div><div><br></div><div>bt #2:</div><div><br></div><div>#0  0xb7690c80 in Perl_pad_undef () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>


<div>#1  0xb764db67 in Perl_cv_undef () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#2  0xb76dcf5a in Perl_sv_clear () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>


<div>#3  0xb76dd288 in Perl_sv_free2 () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#4  0xb76dd391 in Perl_sv_free () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>


<div>#5  0xb76dcd57 in Perl_sv_clear () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#6  0xb76dd288 in Perl_sv_free2 () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>


<div>#7  0xb70af022 in XS_Sys__Syslog_closelog_xs () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Sys/Syslog/Syslog.so</div><div>#8  0xb76c941e in Perl_pp_entersub () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>


<div>#9  0xb76c7841 in Perl_runops_standard () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#10 0xb7662500 in Perl_call_sv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>


<div>#11 0xb7663284 in Perl_call_pv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#12 0xb72236ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so</div><div>#13 0x08063524 in modcall ()</div>


<div>#14 0x0805ffe7 in indexed_modcall ()</div><div>#15 0x0806037c in module_authenticate ()</div><div>#16 0x0804f908 in rad_authenticate ()</div><div>#17 0x0806df65 in radius_handle_request ()</div><div>#18 0x080661d0 in request_handler_thread ()</div>


<div>#19 0xb75bd4c0 in start_thread () from /lib/i686/cmov/libpthread.so.0</div><div>#20 0xb736484e in clone () from /lib/i686/cmov/libc.so.6</div><div><br></div><div><br></div><div>bt#3:</div><div><br></div><div>#0  0xb77a2424 in __kernel_vsyscall ()</div>


<div>#1  0xb72d6640 in raise () from /lib/i686/cmov/libc.so.6</div><div>#2  0xb72d8018 in abort () from /lib/i686/cmov/libc.so.6</div><div>#3  0xb731348d in __libc_message () from /lib/i686/cmov/libc.so.6</div><div>#4  0xb7319764 in malloc_printerr () from /lib/i686/cmov/libc.so.6</div>


<div>#5  0xb731b966 in free () from /lib/i686/cmov/libc.so.6</div><div>#6  0xb76d81b1 in Perl_safesysfree () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#7  0xb77040a8 in Perl_sv_clear () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>


<div>#8  0xb7704288 in Perl_sv_free2 () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#9  0xb70d6022 in XS_Sys__Syslog_closelog_xs () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Sys/Syslog/Syslog.so</div>


<div>#10 0xb76f041e in Perl_pp_entersub () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#11 0xb76ee841 in Perl_runops_standard () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>


<div>#12 0xb7689500 in Perl_call_sv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#13 0xb768a284 in Perl_call_pv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>


<div>#14 0xb724a6ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so</div><div>#15 0x08063524 in modcall ()</div><div>#16 0x0805ffe7 in indexed_modcall ()</div><div>#17 0x0806037c in module_authenticate ()</div>


<div>#18 0x0804f908 in rad_authenticate ()</div><div>#19 0x0806df65 in radius_handle_request ()</div><div>#20 0x080661d0 in request_handler_thread ()</div><div>#21 0xb75e44c0 in start_thread () from /lib/i686/cmov/libpthread.so.0</div>


<div>#22 0xb738b84e in clone () from /lib/i686/cmov/libc.so.6</div><div><br></div><div>bt #4</div><div><br></div><div>#0  0xb7673132 in Perl_av_fetch () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>


<div>#1  0xb708c788 in DD_dump () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Data/Dumper/Dumper.so</div><div>#2  0xb708d77f in DD_dump () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Data/Dumper/Dumper.so</div>


<div>#3  0xb7090219 in XS_Data__Dumper_Dumpxs () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Data/Dumper/Dumper.so</div><div>#4  0xb767541e in Perl_pp_entersub () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>


<div>#5  0xb7673841 in Perl_runops_standard () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#6  0xb760e500 in Perl_call_sv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>


<div>#7  0xb760f284 in Perl_call_pv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#8  0xb71cf6ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so</div><div>#9  0x08063524 in modcall ()</div>


<div>#10 0x0805ffe7 in indexed_modcall ()</div><div>#11 0x0806037c in module_authenticate ()</div><div>#12 0x0804f908 in rad_authenticate ()</div><div>#13 0x0806df65 in radius_handle_request ()</div><div>#14 0x080661d0 in request_handler_thread ()</div>


<div>#15 0xb75694c0 in start_thread () from /lib/i686/cmov/libpthread.so.0</div><div>#16 0xb731084e in clone () from /lib/i686/cmov/libc.so.6</div><div><br></div><div>bt #5:</div><div><br></div><div>#0  0xb771ef77 in Perl_newSVsv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>


<div>#1  0xb7712020 in Perl_pp_aassign () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#2  0xb770a841 in Perl_runops_standard () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>


<div>#3  0xb76a5500 in Perl_call_sv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#4  0xb76a6284 in Perl_call_pv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>


<div>#5  0xb72666ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so</div><div>#6  0x08063524 in modcall ()</div><div>#7  0x0805ffe7 in indexed_modcall ()</div><div>#8  0x0806037c in module_authenticate ()</div>


<div>#9  0x0804f908 in rad_authenticate ()</div><div>#10 0x0806df65 in radius_handle_request ()</div><div>#11 0x080661d0 in request_handler_thread ()</div><div>#12 0xb76004c0 in start_thread () from /lib/i686/cmov/libpthread.so.0</div>


<div>#13 0xb73a784e in clone () from /lib/i686/cmov/libc.so.6</div><div><br></div><div>bt #6:</div><div>#0  0xb76f4424 in __kernel_vsyscall ()</div><div>#1  0xb7228640 in raise () from /lib/i686/cmov/libc.so.6</div><div>

#2  0xb722a018 in abort () from /lib/i686/cmov/libc.so.6</div>
<div>#3  0xb726548d in __libc_message () from /lib/i686/cmov/libc.so.6</div><div>#4  0xb726b764 in malloc_printerr () from /lib/i686/cmov/libc.so.6</div><div>#5  0xb726d966 in free () from /lib/i686/cmov/libc.so.6</div><div>


#6  0xb762a1b1 in Perl_safesysfree () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#7  0xb76560a8 in Perl_sv_clear () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>


<div>#8  0xb7656288 in Perl_sv_free2 () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#9  0xb7028022 in XS_Sys__Syslog_closelog_xs () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Sys/Syslog/Syslog.so</div>


<div>#10 0xb764241e in Perl_pp_entersub () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#11 0xb7640841 in Perl_runops_standard () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>


<div>#12 0xb75db500 in Perl_call_sv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div><div>#13 0xb75dc284 in Perl_call_pv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so</div>


<div>#14 0xb719c6ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so</div><div>#15 0x08063524 in modcall ()</div><div>#16 0x0805ffe7 in indexed_modcall ()</div><div>#17 0x0806037c in module_authenticate ()</div>


<div>#18 0x0804f908 in rad_authenticate ()</div><div>#19 0x0806df65 in radius_handle_request ()</div><div>#20 0x080661d0 in request_handler_thread ()</div><div>#21 0xb75364c0 in start_thread () from /lib/i686/cmov/libpthread.so.0</div>


<div>#22 0xb72dd84e in clone () from /lib/i686/cmov/libc.so.6</div><div><br></div></div><div>Thanks,</div><div>Anatoly.</div><div><br><br><div class="gmail_quote">On Thu, Jan 6, 2011 at 1:26 AM, Boian Jordanov <span dir="ltr">&lt;<a href="mailto:bjordanov@orbitel.bg" target="_blank">bjordanov@orbitel.bg</a>&gt;</span> wrote:<br>


<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><br>
On Jan 6, 2011, at 1:51 AM, Anatoly Ivanov wrote:<br>
<br>
</div><div>&gt; Hi everybody,<br>
&gt;<br>
&gt; I have a couple of questions regarding modules/rlm_perl/rlm_perl.c:<br>
&gt; perl_detach(...):<br>
&gt;<br>
&gt; 1.  Is it correct to call PERL_SYS_TERM in detach()?<br>
&gt;<br>
&gt; I am experiencing quite random and unpredictable segfaults with radius 2.1.8<br>
&gt; and rlm_perl, and it really looks like memory allocation/multithreading<br>
&gt; error. I have very similar system running radius 2.1.7, and it works just<br>
&gt; fine. The most significant difference in rlm_perl between 2.1.7 and 2.1.8 is<br>
&gt; that PERL_SYS_TERM call.<br>
&gt;<br>
&gt; According to the documentation<br>
&gt; (<a href="http://perldoc.perl.org/perlembed.html" target="_blank">http://perldoc.perl.org/perlembed.html</a>), PERL_SYS_TERM<br>
&gt; should be called only once. In our case, it is called once per detaching<br>
&gt; thread.<br>
<br>
</div>In fact it is called only once in the end of life of module. You can see that when a thread is destroyed is called<br>
<br>
/* Create Key */<br>
static void rlm_perl_make_key(pthread_key_t *key)<br>
{<br>
        pthread_key_create(key, rlm_destroy_perl);<br>
}<br>
<br>
rlm_destroy_perl which destroy only particular interp<br>
<br>
So the best will be to produce gdb trace file as i already wrote in my previous mail.<br>
<div><br>
<br>
&gt;<br>
&gt; 2. This code:<br>
&gt;<br>
&gt; embed = rad_malloc(4*(sizeof(char *)));<br>
&gt; memset(embed, 0, sizeof(4*(sizeof(char *))));<br>
&gt;<br>
&gt; looks like a misprint to me. Not that it would cause any real errors, but it<br>
&gt; is inaccurate anyway.<br>
&gt;<br>
&gt; Thank you,<br>
&gt; Anatoly.<br>
</div><div><div></div><div>&gt; -<br>
&gt; List info/subscribe/unsubscribe? See <a href="http://www.freeradius.org/list/devel.html" target="_blank">http://www.freeradius.org/list/devel.html</a><br>
<br>
Best Regards,<br>
Boian Jordanov<br>
Head of Voice Department<br>
tel. +359 2 4004 723<br>
tel. +359 2 4004 002<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
-<br>
List info/subscribe/unsubscribe? See <a href="http://www.freeradius.org/list/devel.html" target="_blank">http://www.freeradius.org/list/devel.html</a><br>
</div></div></blockquote></div><br></div>