Re: Freeradius-Users Digest, Vol 108, Issue 20
Hi, First you´d better upgrade to 2.2.4; so many things have improved. We are serving around 2500 wifi customers, and the whole process of authentication takes less than 20 ms, and we do use a MySQL database. I advise to avoid external perl routines as possible, and have a look at unlang; are you sure you are also measuring correctly things? I used to use a couple of scripts that had their own setup, and though I had 200 ms when I was having 30ms in the past, until it dawned in me it was the time for the perl environment and script to load and setup the environment to work. Regards ------------------------------
Message: 4 Date: Fri, 04 Apr 2014 16:46:11 +0700 From: Eugene Grosbein <fr@grosbein.net> To: freeradius-users@lists.freeradius.org Subject: DHCP module and external DHCP relay Message-ID: <533E7F63.5020109@grosbein.net> Content-Type: text/plain; charset=us-ascii
Hi!
We use FreeRADIUS 2.2.0 as DHCP server with rlm_perl to connect to external database to run stored procedure there that returns IP/mask/DNS etc for option82-enabled requests.
The database is pretty fast and serves hundreds of parallel requests in 1 second or less.
FreeRADIUS receives all DHCP requests from external high loaded DHCP relay and threats them all as coming from single DHCP client.
Our perl script does extensive logging through direct syslog connection and I see that there is great delay between the moment of DHCP request arrival as shown by tcpdump and the moment when post_auth perl function is invoked by FreeRADIUS.
This delay is about 6 seconds and some DHCP clients do not wait for so long.
Is it possible to eliminate these delays? Here are lines from radiusd log:
Request 14280134 has been waiting in the processing queue for 6 seconds. Check that all databases are running properly! Discarding duplicate request from client dhcp port 67 - ID: 1381353109 due to unfinished request 14280311
Again, our database processes most request in less than 1 second and very seldom in 2 seconds but mentioned problem is persistent.
Eugene Grosbein
------------------------------
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
End of Freeradius-Users Digest, Vol 108, Issue 20 *************************************************
On 04.04.2014 19:59, Rui Ribeiro wrote:
Hi,
First you´d better upgrade to 2.2.4; so many things have improved.
I've tried 2.2.4 when it was released. It coredumps all the way with dhcp and rlm_perl while using many hundreds of threads, 2.2.0 is much more stable in our setup and 2.2.4 is unusable.
We are serving around 2500 wifi customers, and the whole process of authentication takes less than 20 ms, and we do use a MySQL database.
We use MS SQL database and I need perl DBI to keep persistent pool of database connections.
I advise to avoid external perl routines as possible, and have a look at unlang; are you sure you are also measuring correctly things?
There is no place to make mistake: tcpdump shows precise timestamps noting moment of request arrival. And then syslogd marks a moment then perl code starts: there is 6 seconds delay in between. Do you have DHCP Relay in front of FreeRADIUS? This is culprit here as it seems that FreeRADIS dhcp module mixes all request from single DHCP relay to the single queue.
I used to use a couple of scripts that had their own setup, and though I had 200 ms when I was having 30ms in the past, until it dawned in me it was the time for the perl environment and script to load and setup the environment to work.
I use fixed number of perl threads not allowing them to die/recreate. This way I have no extra overhead after initial creation of threads "to load and setup the environment". Eugene Grosbein
Hi,
I've tried 2.2.4 when it was released. It coredumps all the way with dhcp and rlm_perl while using many hundreds of threads, 2.2.0 is much more stable in our setup and 2.2.4 is unusable.
report the bugs - we run 2.2.4 on busy servers with no issues (touch wood!! crazy to say that on a friday! ;-) ) I'm suprised you arent finding even worse issues on 2.2.0(!)
We use MS SQL database and I need perl DBI to keep persistent pool of database connections.
use 3.0.2 :-)
There is no place to make mistake: tcpdump shows precise timestamps noting moment of request arrival. And then syslogd marks a moment then perl code starts: there is 6 seconds delay in between.
..thats...just weird. we dont get such issues with PERL. are you using the global caching of PERL module? alan
On 04.04.2014 22:35, A.L.M.Buxey@lboro.ac.uk wrote:
Hi,
I've tried 2.2.4 when it was released. It coredumps all the way with dhcp and rlm_perl while using many hundreds of threads, 2.2.0 is much more stable in our setup and 2.2.4 is unusable.
report the bugs - we run 2.2.4 on busy servers with no issues (touch wood!! crazy to say that on a friday! ;-) ) I'm suprised you arent finding even worse issues on 2.2.0(!)
I found these issues with 2.2.0 but we can workaround them. 2.2.4 is much, much gravy in our setup and we could not find workarounds for all these coredumps.
We use MS SQL database and I need perl DBI to keep persistent pool of database connections.
use 3.0.2 :-)
There is no place to make mistake: tcpdump shows precise timestamps noting moment of request arrival. And then syslogd marks a moment then perl code starts: there is 6 seconds delay in between.
..thats...just weird. we dont get such issues with PERL. are you using the global caching of PERL module?
What is "global caching"?
Hi,
I found these issues with 2.2.0 but we can workaround them. 2.2.4 is much, much gravy in our setup and we could not find workarounds for all these coredumps.
I'm hearing you but I'm not seeing any debug/bugs reported. that means that when 2.2.5 comes out it might not fix these things either...as theres no record of them regarding using variables globally across threads... its not clearly documented in rlm_perl docs etc - as its a function of PERL and you'll find examples of doing this on the mailing list history...but basically read the PERL docs about threads eg http://perldoc.perl.org/threads.html alan
On 05.04.2014 21:39, A.L.M.Buxey@lboro.ac.uk wrote:
Hi,
I found these issues with 2.2.0 but we can workaround them. 2.2.4 is much, much gravy in our setup and we could not find workarounds for all these coredumps.
I'm hearing you but I'm not seeing any debug/bugs reported. that means that when 2.2.5 comes out it might not fix these things either...as theres no record of them
I observe memory corruption while using freeraidus-2.2.4 with perl/DBI/ODBC/freetds. Memory corruption occur while rlm_perl runs embedded perl and lead to coredumps. That is, I added some checks to freeradius C code and see that before execution jumps to the perl, memory structures of C code are fine but they are damaged on exiting from the perl to C again. Then coredump is imminent then. With 2.2.0 the whole construct works much more stable, coredumps are very seldom.
regarding using variables globally across threads... its not clearly documented in rlm_perl docs etc - as its a function of PERL and you'll find examples of doing this on the mailing list history...but basically read the PERL docs about threads eg
Yes, I use variables shared betwheen perl threads. I use corresponding locking for them very carefully.
Eugene Grosbein wrote:
I observe memory corruption while using freeraidus-2.2.4 with perl/DBI/ODBC/freetds. Memory corruption occur while rlm_perl runs embedded perl and lead to coredumps. That is, I added some checks to freeradius C code and see that before execution jumps to the perl, memory structures of C code are fine but they are damaged on exiting from the perl to C again.
*Which* C structures? Knowing that would help. Just saying "C structures" is unhelpful, because all of FreeRADIUS uses C structures, as does Perl.
Then coredump is imminent then.
Hmm... the rlm_perl changes from 2.2.0 to 2.2.4 are very small. Mainly adding mutex locks around the perl clone calls. Please run valgrind on the 2.2.4 code. e.g. $ valgrind radiusd -f If there are any errors with the Perl code, please post them here. Alan DeKok.
participants (4)
-
A.L.M.Buxey＠lboro.ac.uk -
Alan DeKok -
Eugene Grosbein -
Rui Ribeiro