Boian, The code of perm module is rather complicated, but the only global variable (used for DB access) is initialized in proper manner in CLONE: my $dbh; sub CLONE { eval { # .... $dbh = DBI->connect(....); #..... }; #error handling } I will continue looking through the perl code, but so far it does look thread-safe. BTW, I have another weird symptom on the system: sometimes I get these garbled messages in the syslog: Jan 7 15:17:17 servername <98><A1>{<AE>P>{<AE>perl: rlm_perl: Added pair Digest-Attributes = 0x030a5240000003544552 Jan 7 15:17:17 servername <98><A1>{<AE>P>{<AE>perl: rlm_perl: Added pair X-Ascend-PW-Lifetime = 4294967295 Jan 7 15:18:23 servername <C8>aC<B7><80>kj#014perl: rlm_perl: Added pair Cisco-AVPair = call-id=b4986e20-c2524cb1-c348dc2e@192.168.0.1 Any idea why this may be happening? I use rsyslog for logging. Unfortunately, I cannot send you the core (security policy limitations). Thank you, Anatoly. On Fri, Jan 7, 2011 at 1:54 AM, Boian Jordanov <bjordanov@orbitel.bg> wrote:
Anatoly,
I still think this is thread issue. How do you use/initialize connection to DB is it a global variable ?
Can you send me core #1 ?
On Jan 7, 2011, at 1:06 AM, Anatoly Ivanov wrote:
Boian,
Thanks for the explanation.
I am rather limited in what I can do on a production system, so it is not (administratively) easy to re-compile and re-deploy the code.
Right now I have configured the server to never kill a thread from threadpool - by specifying max_spare_servers=max_servers. The server has not crashed for two days now, but that may be a coincidence - low load during the holidays helps with the stability. I should be able to tell for sure next week.
I have a bunch of backtraces. Please let me know if you see anything suspicious in there. bt #1:
#0 0xb47d1ab6 in mysql_st_execute (sth=0xaf13a1a8, imp_sth=0xadf537a8) at dbdimp.c:3209 #1 0xb47da215 in XS_DBD__mysql__st_execute (my_perl=0xb831a08, cv=0xb1e16858) at mysql.xsi:588 #2 0xb70740ab in XS_DBI_dispatch () from /opt/server/lib/perl5/i686-linux-thread-multi/auto/DBI/DBI.so #3 0xb76a841e in Perl_pp_entersub () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #4 0xb76a6841 in Perl_runops_standard () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #5 0xb7641500 in Perl_call_sv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #6 0xb7642284 in Perl_call_pv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #7 0xb72026ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so #8 0x08063524 in modcall () #9 0x0805ffe7 in indexed_modcall () #10 0x080602fc in module_accounting () #11 0x0804e9d1 in rad_accounting () #12 0x0806df65 in radius_handle_request () #13 0x080661d0 in request_handler_thread () #14 0xb759c4c0 in start_thread () from /lib/i686/cmov/libpthread.so.0 #15 0xb734384e in clone () from /lib/i686/cmov/libc.so.6
bt #2:
#0 0xb7690c80 in Perl_pad_undef () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #1 0xb764db67 in Perl_cv_undef () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #2 0xb76dcf5a in Perl_sv_clear () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #3 0xb76dd288 in Perl_sv_free2 () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #4 0xb76dd391 in Perl_sv_free () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #5 0xb76dcd57 in Perl_sv_clear () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #6 0xb76dd288 in Perl_sv_free2 () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #7 0xb70af022 in XS_Sys__Syslog_closelog_xs () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Sys/Syslog/Syslog.so
#8 0xb76c941e in Perl_pp_entersub () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #9 0xb76c7841 in Perl_runops_standard () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #10 0xb7662500 in Perl_call_sv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #11 0xb7663284 in Perl_call_pv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #12 0xb72236ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so #13 0x08063524 in modcall () #14 0x0805ffe7 in indexed_modcall () #15 0x0806037c in module_authenticate () #16 0x0804f908 in rad_authenticate () #17 0x0806df65 in radius_handle_request () #18 0x080661d0 in request_handler_thread () #19 0xb75bd4c0 in start_thread () from /lib/i686/cmov/libpthread.so.0 #20 0xb736484e in clone () from /lib/i686/cmov/libc.so.6
bt#3:
#0 0xb77a2424 in __kernel_vsyscall () #1 0xb72d6640 in raise () from /lib/i686/cmov/libc.so.6 #2 0xb72d8018 in abort () from /lib/i686/cmov/libc.so.6 #3 0xb731348d in __libc_message () from /lib/i686/cmov/libc.so.6 #4 0xb7319764 in malloc_printerr () from /lib/i686/cmov/libc.so.6 #5 0xb731b966 in free () from /lib/i686/cmov/libc.so.6 #6 0xb76d81b1 in Perl_safesysfree () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #7 0xb77040a8 in Perl_sv_clear () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #8 0xb7704288 in Perl_sv_free2 () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #9 0xb70d6022 in XS_Sys__Syslog_closelog_xs () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Sys/Syslog/Syslog.so
#10 0xb76f041e in Perl_pp_entersub () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #11 0xb76ee841 in Perl_runops_standard () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #12 0xb7689500 in Perl_call_sv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #13 0xb768a284 in Perl_call_pv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #14 0xb724a6ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so #15 0x08063524 in modcall () #16 0x0805ffe7 in indexed_modcall () #17 0x0806037c in module_authenticate () #18 0x0804f908 in rad_authenticate () #19 0x0806df65 in radius_handle_request () #20 0x080661d0 in request_handler_thread () #21 0xb75e44c0 in start_thread () from /lib/i686/cmov/libpthread.so.0 #22 0xb738b84e in clone () from /lib/i686/cmov/libc.so.6
bt #4
#0 0xb7673132 in Perl_av_fetch () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #1 0xb708c788 in DD_dump () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Data/Dumper/Dumper.so
#2 0xb708d77f in DD_dump () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Data/Dumper/Dumper.so
#3 0xb7090219 in XS_Data__Dumper_Dumpxs () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Data/Dumper/Dumper.so
#4 0xb767541e in Perl_pp_entersub () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #5 0xb7673841 in Perl_runops_standard () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #6 0xb760e500 in Perl_call_sv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #7 0xb760f284 in Perl_call_pv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #8 0xb71cf6ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so #9 0x08063524 in modcall () #10 0x0805ffe7 in indexed_modcall () #11 0x0806037c in module_authenticate () #12 0x0804f908 in rad_authenticate () #13 0x0806df65 in radius_handle_request () #14 0x080661d0 in request_handler_thread () #15 0xb75694c0 in start_thread () from /lib/i686/cmov/libpthread.so.0 #16 0xb731084e in clone () from /lib/i686/cmov/libc.so.6
bt #5:
#0 0xb771ef77 in Perl_newSVsv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #1 0xb7712020 in Perl_pp_aassign () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #2 0xb770a841 in Perl_runops_standard () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #3 0xb76a5500 in Perl_call_sv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #4 0xb76a6284 in Perl_call_pv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #5 0xb72666ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so #6 0x08063524 in modcall () #7 0x0805ffe7 in indexed_modcall () #8 0x0806037c in module_authenticate () #9 0x0804f908 in rad_authenticate () #10 0x0806df65 in radius_handle_request () #11 0x080661d0 in request_handler_thread () #12 0xb76004c0 in start_thread () from /lib/i686/cmov/libpthread.so.0 #13 0xb73a784e in clone () from /lib/i686/cmov/libc.so.6
bt #6: #0 0xb76f4424 in __kernel_vsyscall () #1 0xb7228640 in raise () from /lib/i686/cmov/libc.so.6 #2 0xb722a018 in abort () from /lib/i686/cmov/libc.so.6 #3 0xb726548d in __libc_message () from /lib/i686/cmov/libc.so.6 #4 0xb726b764 in malloc_printerr () from /lib/i686/cmov/libc.so.6 #5 0xb726d966 in free () from /lib/i686/cmov/libc.so.6 #6 0xb762a1b1 in Perl_safesysfree () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #7 0xb76560a8 in Perl_sv_clear () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #8 0xb7656288 in Perl_sv_free2 () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #9 0xb7028022 in XS_Sys__Syslog_closelog_xs () from
/opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/auto/Sys/Syslog/Syslog.so
#10 0xb764241e in Perl_pp_entersub () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #11 0xb7640841 in Perl_runops_standard () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #12 0xb75db500 in Perl_call_sv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #13 0xb75dc284 in Perl_call_pv () from /opt/server/lib/perl5/5.10.1/i686-linux-thread-multi/CORE/libperl.so #14 0xb719c6ad in rlmperl_call () from /opt/server/radius/lib/rlm_perl.so #15 0x08063524 in modcall () #16 0x0805ffe7 in indexed_modcall () #17 0x0806037c in module_authenticate () #18 0x0804f908 in rad_authenticate () #19 0x0806df65 in radius_handle_request () #20 0x080661d0 in request_handler_thread () #21 0xb75364c0 in start_thread () from /lib/i686/cmov/libpthread.so.0 #22 0xb72dd84e in clone () from /lib/i686/cmov/libc.so.6
Thanks, Anatoly.
On Thu, Jan 6, 2011 at 1:26 AM, Boian Jordanov <bjordanov@orbitel.bg> wrote:
On Jan 6, 2011, at 1:51 AM, Anatoly Ivanov wrote:
Hi everybody,
I have a couple of questions regarding modules/rlm_perl/rlm_perl.c: perl_detach(...):
1. Is it correct to call PERL_SYS_TERM in detach()?
I am experiencing quite random and unpredictable segfaults with radius 2.1.8 and rlm_perl, and it really looks like memory allocation/multithreading error. I have very similar system running radius 2.1.7, and it works
just
fine. The most significant difference in rlm_perl between 2.1.7 and 2.1.8 is that PERL_SYS_TERM call.
According to the documentation (http://perldoc.perl.org/perlembed.html), PERL_SYS_TERM should be called only once. In our case, it is called once per detaching thread.
In fact it is called only once in the end of life of module. You can see that when a thread is destroyed is called
/* Create Key */ static void rlm_perl_make_key(pthread_key_t *key) { pthread_key_create(key, rlm_destroy_perl); }
rlm_destroy_perl which destroy only particular interp
So the best will be to produce gdb trace file as i already wrote in my previous mail.
2. This code:
embed = rad_malloc(4*(sizeof(char *))); memset(embed, 0, sizeof(4*(sizeof(char *))));
looks like a misprint to me. Not that it would cause any real errors,
but it
is inaccurate anyway.
Thank you, Anatoly. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/devel.html
Best Regards, Boian Jordanov Head of Voice Department tel. +359 2 4004 723 tel. +359 2 4004 002
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/devel.html
Best Regards, Boian Jordanov Head of Voice Department tel. +359 2 4004 723 tel. +359 2 4004 002