FreeRadius Losing packets in Multi-Threads mode
Hello everybody, I'm currently running freeradius on a debian (squeeze) server # uname -a Linux test 2.6.32-5-amd64 #1 SMP Mon Oct 3 03:59:20 UTC 2011 x86_64 GNU/Linux # freeradius -v freeradius: FreeRADIUS Version 2.1.10, for host x86_64-pc-linux-gnu, built on Nov 14 2010 at 21:12:30 The problem is that the server seems to lose accounting packets in Multithread mode Logging with module "detail" or "linelog" logs only a very little amount (0.5%) of the transmitted packets. The client receive all the answer, but my problem is that I rely on those logs for an other program. Example: # echo "Message-Authenticator = 0x00" | radclient 127.0.0.1 acct testing123 -c 2500000 > temp && wc -l temp && wc -l /var/log/freeradius/radacct/127.0.0.1/detail-20111025 2500000 temp 84224 /var/log/freeradius/radacct/127.0.0.1/detail-20111025 This only happen when using FreeRadius in Multithread mode, while using it in single-thread (freeradius -s), all the packets are logged correctly. Anyone has a hint on this problem? Regards, Pierre Rondou
Pierre Rondou wrote:
The problem is that the server seems to lose accounting packets in Multithread mode
It's possible that you're simply sending packets too fast. If the server doesn't read them from the socket quickly enough, the kernel will simply discard them.
Logging with module "detail" or "linelog" logs only a very little amount (0.5%) of the transmitted packets. The client receive all the answer, but my problem is that I rely on those logs for an other program.
It should handle a lot more than .5% of the packets. It does on my system.
This only happen when using FreeRadius in Multithread mode, while using it in single-thread (freeradius -s), all the packets are logged correctly.
Anyone has a hint on this problem?
What *else* is the server doing? Databases? Exec'd programs? Alan DeKok.
Hello Alan, See answers inline Regards, Pierre Alan DeKok a écrit :
Pierre Rondou wrote:
The problem is that the server seems to lose accounting packets in Multithread mode It's possible that you're simply sending packets too fast. If the server doesn't read them from the socket quickly enough, the kernel will simply discard them.
Well, then, why is this only happening in the multi-thread mode? If it was a kernel issue, wouldn't it drop the packets in the same way for the single-thread version?
Logging with module "detail" or "linelog" logs only a very little amount (0.5%) of the transmitted packets. The client receive all the answer, but my problem is that I rely on those logs for an other program. It should handle a lot more than .5% of the packets. It does on my system.
This only happen when using FreeRadius in Multithread mode, while using it in single-thread (freeradius -s), all the packets are logged correctly.
Anyone has a hint on this problem? What *else* is the server doing? Databases? Exec'd programs?
Actually, it's a test server (HP Proliant 360 with 16 Gb ram and quad-core proc), so nothing is really running on it. MongoDB is installed (but nothing is using it), monit and ssh are installed as well, and that's it. The debian was installed with basically nothing.
Pierre Rondou wrote:
It's possible that you're simply sending packets too fast. If the server doesn't read them from the socket quickly enough, the kernel will simply discard them.
Well, then, why is this only happening in the multi-thread mode? If it was a kernel issue, wouldn't it drop the packets in the same way for the single-thread version?
No. I mean that the threading code has overhead which doesn't exist in the non-threaded mode. Lock contention is a *major* source of lost CPU time in threaded applications. If the server is too slow to process requests, the kernel will throw away the UDP packets. This happens when the server is slow... whether it's threaded or not.
Actually, it's a test server (HP Proliant 360 with 16 Gb ram and quad-core proc), so nothing is really running on it. MongoDB is installed (but nothing is using it), monit and ssh are installed as well, and that's it. The debian was installed with basically nothing.
Well... that doesn't sound right. Accounting works for me... Alan DeKok.
Alan DeKok <aland@deployingradius.com> writes:
Pierre Rondou wrote:
It's possible that you're simply sending packets too fast. If the server doesn't read them from the socket quickly enough, the kernel will simply discard them.
Well, then, why is this only happening in the multi-thread mode? If it was a kernel issue, wouldn't it drop the packets in the same way for the single-thread version?
No. I mean that the threading code has overhead which doesn't exist in the non-threaded mode. Lock contention is a *major* source of lost CPU time in threaded applications.
If the server is too slow to process requests, the kernel will throw away the UDP packets. This happens when the server is slow... whether it's threaded or not.
But then the client won't get the acks and retransmit. If I understood correctly, then the problem is the packets *are* acked but not logged. File locking problem when multiple threads are attempting to update the same file? Bjørn
Bjørn Mork wrote:
File locking problem when multiple threads are attempting to update the same file?
Maybe... the detail file is marked "not thread-safe", which means that the server core takes care of locking it. i.e. only one thread at a time will be inside of the detail module. Alan DeKok.
On 26 October 2011 04:48, Bjørn Mork <bjorn@mork.no> wrote:
If the server is too slow to process requests, the kernel will throw away the UDP packets. This happens when the server is slow... whether it's threaded or not.
But then the client won't get the acks and retransmit. If I understood correctly, then the problem is the packets *are* acked but not logged.
Maybe my whole understanding of UDP is flawed, I thought the whole point of using UDP instead of TCP is not to have ACKS and retransmits? Cheers, Andrej
Andrej <andrej.groups@gmail.com> writes:
On 26 October 2011 04:48, Bjørn Mork <bjorn@mork.no> wrote:
If the server is too slow to process requests, the kernel will throw away the UDP packets. This happens when the server is slow... whether it's threaded or not.
But then the client won't get the acks and retransmit. If I understood correctly, then the problem is the packets *are* acked but not logged.
Maybe my whole understanding of UDP is flawed, I thought the whole point of using UDP instead of TCP is not to have ACKS and retransmits?
Not nessarily. The point is that UDP leaves that up to the application. For RADIUS accounting this is described in http://tools.ietf.org/html/rfc2866#section-2 <quote> It is recommended that the client continue attempting to send the Accounting-Request packet until it receives an acknowledgement, using some form of backoff. If no response is returned within a length of time, the request is re- sent a number of times. The client can also forward requests to an alternate server or servers in the event that the primary server is down or unreachable. An alternate server can be used either after a number of tries to the primary server fail, or in a round-robin fashion. Retry and fallback algorithms are the topic of current research and are not specified in detail in this document. </quote> I don't know if all clients follow this recommendation, but a lot of them do. Bjørn
Hi,
But then the client won't get the acks and retransmit. If I understood correctly, then the problem is the packets *are* acked but not logged.
Maybe my whole understanding of UDP is flawed, I thought the whole point of using UDP instead of TCP is not to have ACKS and retransmits?
at the service level - the software has do deal with problems instead of the TCPIP stack. however, as Alan said. in single thread mode, you only have one process dealing with requests.....so one single open connection to SQL, one single sesion to LDAP etc etc (whatever you use) - eg even a local file with PERL. with multithread mode, you have many threads - all of which can be hitting your SQL or LDAP at same time...or trying to write to the same file in a PERL module....so you have to look at the speed/ability of your backend....the jump from a single query ata time to concurrent queries may have tipped your balance alan
participants (5)
-
Alan Buxey -
Alan DeKok -
Andrej -
Bjørn Mork -
Pierre Rondou