<div>Boian,</div><div><br></div><div>The code of perm module is rather complicated, but the only global variable (used for DB access) is initialized in proper manner in CLONE:</div><div><div><br></div><div>my $dbh;</div><div>
<br></div><div>sub CLONE {</div><div> eval {</div><div> # ....</div><div> $dbh = DBI->connect(....);</div><div> #.....</div><div> };</div><div> #error handling</div><div>}</div></div><div>
<br></div><div>I will continue looking through the perl code, but so far it does look thread-safe. </div><div><br></div><div>BTW, I have another weird symptom on the system: sometimes I get these garbled messages in the syslog:</div>
<div><div><br></div><div>Jan 7 15:17:17 servername <98><A1>{<AE>P>{<AE>perl: rlm_perl: Added pair Digest-Attributes = 0x030a5240000003544552</div><div><div>Jan 7 15:17:17 servername <98><A1>{<AE>P>{<AE>perl: rlm_perl: Added pair X-Ascend-PW-Lifetime = 4294967295</div>
</div><div>Jan 7 15:18:23 servername <C8>aC<B7><80>kj#014perl: rlm_perl: Added pair Cisco-AVPair = call-id=<a href="mailto:b4986e20-c2524cb1-c348dc2e@192.168.0.1">b4986e20-c2524cb1-c348dc2e@192.168.0.1</a></div>
</div><div><br></div><div>Any idea why this may be happening? I use rsyslog for logging.</div><div><br></div>Unfortunately, I cannot send you the core (security policy limitations).<div><br></div><div>Thank you,</div><div>
Anatoly.</div><br><div class="gmail_quote">On Fri, Jan 7, 2011 at 1:54 AM, Boian Jordanov <span dir="ltr"><<a href="mailto:bjordanov@orbitel.bg">bjordanov@orbitel.bg</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Anatoly,<br>
<br>
I still think this is thread issue. How do you use/initialize connection to DB is it a global variable ?<br>
<br>
Can you send me core #1 ?<br>
<div><div></div><div class="h5"><br>
<br>
<br>
<br>
On Jan 7, 2011, at 1:06 AM, Anatoly Ivanov wrote:<br>
<br>
> Boian,<br>
><br>
> Thanks for the explanation.<br>
><br>
> I am rather limited in what I can do on a production system, so it is not<br>
> (administratively) easy to re-compile and re-deploy the code.<br>
><br>
> Right now I have configured the server to never kill a thread from<br>
> threadpool - by specifying max_spare_servers=max_servers. The server has not<br>
> crashed for two days now, but that may be a coincidence - low load during<br>
> the holidays helps with the stability. I should be able to tell for sure<br>
> next week.<br>
><br>
> I have a bunch of backtraces. Please let me know if you see anything<br>
> suspicious in there.<br>
> bt #1:<br>
><br>
> #0 0xb47d1ab6 in mysql_st_execute (sth=0xaf13a1a8, imp_sth=0xadf537a8) at<br>
> dbdimp.c:3209<br>
> #1 0xb47da215 in XS_DBD__mysql__st_execute (my_perl=0xb831a08,<br>
> cv=0xb1e16858) at mysql.xsi:588<br>
> #2 0xb70740ab in XS_DBI_dispatch () from<br>
> /opt/server/lib/perl5/i686-linux-thread-multi/auto/DBI/DBI.so<br>
> #3 0xb76a841e in Perl_pp_entersub () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #4 0xb76a6841 in Perl_runops_standard () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #5 0xb7641500 in Perl_call_sv () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #6 0xb7642284 in Perl_call_pv () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #7 0xb72026ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so<br>
> #8 0x08063524 in modcall ()<br>
> #9 0x0805ffe7 in indexed_modcall ()<br>
> #10 0x080602fc in module_accounting ()<br>
> #11 0x0804e9d1 in rad_accounting ()<br>
> #12 0x0806df65 in radius_handle_request ()<br>
> #13 0x080661d0 in request_handler_thread ()<br>
> #14 0xb759c4c0 in start_thread () from /lib/i686/cmov/libpthread.so.0<br>
> #15 0xb734384e in clone () from /lib/i686/cmov/libc.so.6<br>
><br>
> bt #2:<br>
><br>
> #0 0xb7690c80 in Perl_pad_undef () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #1 0xb764db67 in Perl_cv_undef () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #2 0xb76dcf5a in Perl_sv_clear () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #3 0xb76dd288 in Perl_sv_free2 () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #4 0xb76dd391 in Perl_sv_free () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #5 0xb76dcd57 in Perl_sv_clear () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #6 0xb76dd288 in Perl_sv_free2 () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #7 0xb70af022 in XS_Sys__Syslog_closelog_xs () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Sys/Syslog/Syslog.so<br>
> #8 0xb76c941e in Perl_pp_entersub () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #9 0xb76c7841 in Perl_runops_standard () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #10 0xb7662500 in Perl_call_sv () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #11 0xb7663284 in Perl_call_pv () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #12 0xb72236ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so<br>
> #13 0x08063524 in modcall ()<br>
> #14 0x0805ffe7 in indexed_modcall ()<br>
> #15 0x0806037c in module_authenticate ()<br>
> #16 0x0804f908 in rad_authenticate ()<br>
> #17 0x0806df65 in radius_handle_request ()<br>
> #18 0x080661d0 in request_handler_thread ()<br>
> #19 0xb75bd4c0 in start_thread () from /lib/i686/cmov/libpthread.so.0<br>
> #20 0xb736484e in clone () from /lib/i686/cmov/libc.so.6<br>
><br>
><br>
> bt#3:<br>
><br>
> #0 0xb77a2424 in __kernel_vsyscall ()<br>
> #1 0xb72d6640 in raise () from /lib/i686/cmov/libc.so.6<br>
> #2 0xb72d8018 in abort () from /lib/i686/cmov/libc.so.6<br>
> #3 0xb731348d in __libc_message () from /lib/i686/cmov/libc.so.6<br>
> #4 0xb7319764 in malloc_printerr () from /lib/i686/cmov/libc.so.6<br>
> #5 0xb731b966 in free () from /lib/i686/cmov/libc.so.6<br>
> #6 0xb76d81b1 in Perl_safesysfree () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #7 0xb77040a8 in Perl_sv_clear () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #8 0xb7704288 in Perl_sv_free2 () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #9 0xb70d6022 in XS_Sys__Syslog_closelog_xs () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Sys/Syslog/Syslog.so<br>
> #10 0xb76f041e in Perl_pp_entersub () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #11 0xb76ee841 in Perl_runops_standard () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #12 0xb7689500 in Perl_call_sv () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #13 0xb768a284 in Perl_call_pv () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #14 0xb724a6ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so<br>
> #15 0x08063524 in modcall ()<br>
> #16 0x0805ffe7 in indexed_modcall ()<br>
> #17 0x0806037c in module_authenticate ()<br>
> #18 0x0804f908 in rad_authenticate ()<br>
> #19 0x0806df65 in radius_handle_request ()<br>
> #20 0x080661d0 in request_handler_thread ()<br>
> #21 0xb75e44c0 in start_thread () from /lib/i686/cmov/libpthread.so.0<br>
> #22 0xb738b84e in clone () from /lib/i686/cmov/libc.so.6<br>
><br>
> bt #4<br>
><br>
> #0 0xb7673132 in Perl_av_fetch () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #1 0xb708c788 in DD_dump () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Data/Dumper/Dumper.so<br>
> #2 0xb708d77f in DD_dump () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Data/Dumper/Dumper.so<br>
> #3 0xb7090219 in XS_Data__Dumper_Dumpxs () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Data/Dumper/Dumper.so<br>
> #4 0xb767541e in Perl_pp_entersub () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #5 0xb7673841 in Perl_runops_standard () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #6 0xb760e500 in Perl_call_sv () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #7 0xb760f284 in Perl_call_pv () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #8 0xb71cf6ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so<br>
> #9 0x08063524 in modcall ()<br>
> #10 0x0805ffe7 in indexed_modcall ()<br>
> #11 0x0806037c in module_authenticate ()<br>
> #12 0x0804f908 in rad_authenticate ()<br>
> #13 0x0806df65 in radius_handle_request ()<br>
> #14 0x080661d0 in request_handler_thread ()<br>
> #15 0xb75694c0 in start_thread () from /lib/i686/cmov/libpthread.so.0<br>
> #16 0xb731084e in clone () from /lib/i686/cmov/libc.so.6<br>
><br>
> bt #5:<br>
><br>
> #0 0xb771ef77 in Perl_newSVsv () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #1 0xb7712020 in Perl_pp_aassign () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #2 0xb770a841 in Perl_runops_standard () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #3 0xb76a5500 in Perl_call_sv () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #4 0xb76a6284 in Perl_call_pv () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #5 0xb72666ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so<br>
> #6 0x08063524 in modcall ()<br>
> #7 0x0805ffe7 in indexed_modcall ()<br>
> #8 0x0806037c in module_authenticate ()<br>
> #9 0x0804f908 in rad_authenticate ()<br>
> #10 0x0806df65 in radius_handle_request ()<br>
> #11 0x080661d0 in request_handler_thread ()<br>
> #12 0xb76004c0 in start_thread () from /lib/i686/cmov/libpthread.so.0<br>
> #13 0xb73a784e in clone () from /lib/i686/cmov/libc.so.6<br>
><br>
> bt #6:<br>
> #0 0xb76f4424 in __kernel_vsyscall ()<br>
> #1 0xb7228640 in raise () from /lib/i686/cmov/libc.so.6<br>
> #2 0xb722a018 in abort () from /lib/i686/cmov/libc.so.6<br>
> #3 0xb726548d in __libc_message () from /lib/i686/cmov/libc.so.6<br>
> #4 0xb726b764 in malloc_printerr () from /lib/i686/cmov/libc.so.6<br>
> #5 0xb726d966 in free () from /lib/i686/cmov/libc.so.6<br>
> #6 0xb762a1b1 in Perl_safesysfree () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #7 0xb76560a8 in Perl_sv_clear () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #8 0xb7656288 in Perl_sv_free2 () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #9 0xb7028022 in XS_Sys__Syslog_closelog_xs () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Sys/Syslog/Syslog.so<br>
> #10 0xb764241e in Perl_pp_entersub () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #11 0xb7640841 in Perl_runops_standard () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #12 0xb75db500 in Perl_call_sv () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #13 0xb75dc284 in Perl_call_pv () from<br>
> /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so<br>
> #14 0xb719c6ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so<br>
> #15 0x08063524 in modcall ()<br>
> #16 0x0805ffe7 in indexed_modcall ()<br>
> #17 0x0806037c in module_authenticate ()<br>
> #18 0x0804f908 in rad_authenticate ()<br>
> #19 0x0806df65 in radius_handle_request ()<br>
> #20 0x080661d0 in request_handler_thread ()<br>
> #21 0xb75364c0 in start_thread () from /lib/i686/cmov/libpthread.so.0<br>
> #22 0xb72dd84e in clone () from /lib/i686/cmov/libc.so.6<br>
><br>
> Thanks,<br>
> Anatoly.<br>
><br>
><br>
> On Thu, Jan 6, 2011 at 1:26 AM, Boian Jordanov <<a href="mailto:bjordanov@orbitel.bg">bjordanov@orbitel.bg</a>> wrote:<br>
><br>
>><br>
>> On Jan 6, 2011, at 1:51 AM, Anatoly Ivanov wrote:<br>
>><br>
>>> Hi everybody,<br>
>>><br>
>>> I have a couple of questions regarding modules/rlm_perl/rlm_perl.c:<br>
>>> perl_detach(...):<br>
>>><br>
>>> 1. Is it correct to call PERL_SYS_TERM in detach()?<br>
>>><br>
>>> I am experiencing quite random and unpredictable segfaults with radius<br>
>> 2.1.8<br>
>>> and rlm_perl, and it really looks like memory allocation/multithreading<br>
>>> error. I have very similar system running radius 2.1.7, and it works just<br>
>>> fine. The most significant difference in rlm_perl between 2.1.7 and 2.1.8<br>
>> is<br>
>>> that PERL_SYS_TERM call.<br>
>>><br>
>>> According to the documentation<br>
>>> (<a href="http://perldoc.perl.org/perlembed.html" target="_blank">http://perldoc.perl.org/perlembed.html</a>), PERL_SYS_TERM<br>
>>> should be called only once. In our case, it is called once per detaching<br>
>>> thread.<br>
>><br>
>> In fact it is called only once in the end of life of module. You can see<br>
>> that when a thread is destroyed is called<br>
>><br>
>> /* Create Key */<br>
>> static void rlm_perl_make_key(pthread_key_t *key)<br>
>> {<br>
>> pthread_key_create(key, rlm_destroy_perl);<br>
>> }<br>
>><br>
>> rlm_destroy_perl which destroy only particular interp<br>
>><br>
>> So the best will be to produce gdb trace file as i already wrote in my<br>
>> previous mail.<br>
>><br>
>><br>
>>><br>
>>> 2. This code:<br>
>>><br>
>>> embed = rad_malloc(4*(sizeof(char *)));<br>
>>> memset(embed, 0, sizeof(4*(sizeof(char *))));<br>
>>><br>
>>> looks like a misprint to me. Not that it would cause any real errors, but<br>
>> it<br>
>>> is inaccurate anyway.<br>
>>><br>
>>> Thank you,<br>
>>> Anatoly.<br>
>>> -<br>
>>> List info/subscribe/unsubscribe? See<br>
>> <a href="http://www.freeradius.org/list/devel.html" target="_blank">http://www.freeradius.org/list/devel.html</a><br>
>><br>
>> Best Regards,<br>
>> Boian Jordanov<br>
>> Head of Voice Department<br>
>> tel. +359 2 4004 723<br>
>> tel. +359 2 4004 002<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> -<br>
>> List info/subscribe/unsubscribe? See<br>
>> <a href="http://www.freeradius.org/list/devel.html" target="_blank">http://www.freeradius.org/list/devel.html</a><br>
>><br>
<br>
Best Regards,<br>
Boian Jordanov<br>
Head of Voice Department<br>
tel. +359 2 4004 723<br>
tel. +359 2 4004 002<br>
<br>
<br>
<br>
<br>
<br>
<br>
</div></div></blockquote></div><br>