One thing that I forgot to mention is that Server1 doesn't support sending status check with status-server, so I had configure status check with request. I can't blame the Server1 to not respond to accounting requests, because I can see the accounting responses ... So I really don't know what is going on .... <....> home_server Server11 { type = acct ipaddr = <Server1 IP address> port = 1813 response_window = 5 status_check = request username = "suntech" password = "<password>" secret = <secret> } home_server_pool Server1_POOL { type = fail-over home_server = Server11 } realm Server1 { type = radius acct_pool = Server1_POOL secret = <secret> <....> Cristina Miyata wrote:
We have just upgraded to FreeRADIUS 2.1.6, but unfortunately the problem persists:
Server1/: total 1008236 -rw------- 1 root root 714751062 Jul 29 11:53 detail-20090729 -rw------- 1 root root 316653344 Jul 29 07:29 detail.work <= stuck!
A 700M detail file? Wow... it should really be rather a lot smaller than that. I would suggest adding %H (at least) to the filename, which will shrink them in size by a factor of 24.
We got sometimes the following message for Server2:
Thu Jul 23 15:00:45 2009 : Proxy: No outstanding request was found for proxy reply from home server <Server2 IP address> port 1813 - ID 142 And several messages for Server1:
Wed Jul 29 11:36:53 2009 : Error: Rejecting request 3993531 due to lack of any response from home server <Server1 IP address> port 1813 Wed Jul 29 11:36:53 2009 : Error: PROXY: Marking home server <Server1 IP address> port 1813 as zombie (it looks like it is dead). Wed Jul 29 11:37:28 2009 : Info: Suspicious proxy state... continuing Wed Jul 29 11:37:30 2009 : Error: Rejecting request 3998634 due to lack of any response from home server <Server1 IP address> port 1813
Your home servers are dead or dying. That's not good.
In an attempt to force the revival of Server1, we scheduled the execution of the following command every minute:
radmin -e "set home_server state <Server1 IP address> 1813 alive"
Uh... that won't help. What happens when it's still down? This is a *very* bad idea. You should use the normal status checks to determine if a home server is alive.
But still, it seems to stuck ... I checked with "tcpdump port 1813 and host <Server1 IP address>" that even though the details.work for Server1 is freezed, FreeRADIUS is sending accounting requests to Server1 and it is receiving accounting responses.
Yes.... it doesn't *modify* the detail file while it's being processed. It processes the whole file, (sending packets the whole time), and then deletes the file when it's done. If the home servers are almost down, it will *continue* to process the detail file, and it will *continue* to send packets until it's done. That's what you're seeing. If the home servers are down, it will STOP proxying packets, and it will STOP reading the detail file... because the home servers are down.
I have many many questions of how FreeRADIUS proxy works. Could someone please help us understand what we doing wrong?
The entire functionality of proxying is documented in the configuration files.
1) Do you think that Server1 detail.work get "stuck", because FreeRADIUS detected that some of the accouting requests in the detail.work didn't have a response from Server1?
That is how the process is *documented* as working. See raddb/sites-available/copy-acct-to-home-server.
2) FreeRADIUS is still sending accounting requests and receiving responses for Server1, just because we are setting Server1 alive? When we do this, it starts processing the details.work from the beginning of the file? After sometime, FreeRADIUS proxy stops completely. Is it because FreeRADIUS had too many accounting requests without responses?
I have no idea what that means.
3) Server1 receives all accounting requests received by FreeRADIUS, and Server2 receives accounting requests that matches a filter. I've noticed that Server1 and Server2 accounting responses for the same accounting requests have the same Packet Identifier, and that the NAS frequently reuses this Packet Identifier. Do you think that FreeRADIUS can get lost in this situation?
No. Alan DeKok. ---------[ Received Mail Content ]---------- Subject : [RE]Problem with proxied accounting Date : Wed, 29 Jul 2009 11:19:14 -0400 (EDT)
From : "Cristina Miyata" <cmiyata@lycos.com> To : <freeradius-users@lists.freeradius.org>
p {margin-top:0px;margin-bottom:0px;} Thanks, Alan for the advice! We have just upgraded to FreeRADIUS 2.1.6, but unfortunately the problem persists: Server1/: total 1008236 -rw------- 1 root root 714751062 Jul 29 11:53 detail-20090729 -rw------- 1 root root 316653344 Jul 29 07:29 detail.work <= stuck! We got sometimes the following message for Server2: Thu Jul 23 15:00:45 2009 : Proxy: No outstanding request was found for proxy reply from home server <Server2 IP address> port 1813 - ID 142 And several messages for Server1: Wed Jul 29 11:36:53 2009 : Error: Rejecting request 3993531 due to lack of any response from home server <Server1 IP address> port 1813 Wed Jul 29 11:36:53 2009 : Error: PROXY: Marking home server <Server1 IP address> port 1813 as zombie (it looks like it is dead). Wed Jul 29 11:37:28 2009 : Info: Suspicious proxy state... continuing Wed Jul 29 11:37:30 2009 : Error: Rejecting request 3998634 due to lack of any response from home server <Server1 IP address> port 1813 <...> In an attempt to force the revival of Server1, we scheduled the execution of the following command every minute: radmin -e "set home_server state <Server1 IP address> 1813 alive" But still, it seems to stuck ... I checked with "tcpdump port 1813 and host <Server1 IP address>" that even though the details.work for Server1 is freezed, FreeRADIUS is sending accounting requests to Server1 and it is receiving accounting responses. # tcpdump port 1813 and host <Server1 IP address> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes 11:39:27.667255 IP <FreeRADIUS IP address>.1814 > Server1.radius-acct: RADIUS, Accounting Request (4), id: 0x0c length: 202 11:39:27.675969 IP Server1.radius-acct > <FreeRADIUS IP address>.1814: RADIUS, Accounting Response (5), id: 0x0c length: 20 <...> I have many many questions of how FreeRADIUS proxy works. Could someone please help us understand what we doing wrong? 1) Do you think that Server1 detail.work get "stuck", because FreeRADIUS detected that some of the accouting requests in the detail.work didn't have a response from Server1? 2) FreeRADIUS is still sending accounting requests and receiving responses for Server1, just because we are setting Server1 alive? When we do this, it starts processing the details.work from the beginning of the file? After sometime, FreeRADIUS proxy stops completely. Is it because FreeRADIUS had too many accounting requests without responses? 3) Server1 receives all accounting requests received by FreeRADIUS, and Server2 receives accounting requests that matches a filter. I've noticed that Server1 and Server2 accounting responses for the same accounting requests have the same Packet Identifier, and that the NAS frequently reuses this Packet Identifier. Do you think that FreeRADIUS can get lost in this situation? Please help us! Thanks, Cristina Miyata ----------------- Cristina Miyata wrote:
We are using Freeradius 2.1.1 and we send accounting RADIUS to 2 different servers called Server1 and Server2. In order to do so, we created two proxy servers and 3 detailed accouting logs: detail (stored in the server), detail1 (processed by the proxy server that send accounting to Server1) and detail2 (processed by the proxy server that send accounting to Server2).
I'd suggest upgrading to 2.1.6. It fixes some issues when reading from detail files. Alan DeKok. ---------[ Received Mail Content ]---------- Subject : Problem with proxied accounting Date : Tue, 21 Jul 2009 21:28:58 -0400 (EDT)
From : "Cristina Miyata" <cmiyata@lycos.com> To : <freeradius-users@lists.freeradius.org>
p {margin-top:0px;margin-bottom:0px;} Dear FreeRADIUS Users, We are using Freeradius 2.1.1 and we send accounting RADIUS to 2 different servers called Server1 and Server2. In order to do so, we created two proxy servers and 3 detailed accouting logs: detail (stored in the server), detail1 (processed by the proxy server that send accounting to Server1) and detail2 (processed by the proxy server that send accounting to Server2). For a while, the proxy serves works fine. Then one of them starts logging reject request due to lack of any response from home server <Server1 IP address> port 1813: Tue Jul 21 22:03:29 2009 : Error: Rejecting request 38447540 due to lack of any response from home server <Server1 IP address> port 1813 Tue Jul 21 22:03:29 2009 : Error: PROXY: Marking home server <Server1 IP address> port 1813 as zombie (it looks like it is dead). The proxy server for Server2 also stops working from time to time, but doesn't log any errors in "radius.log" file. The details file for the proxy gets larger and larger, and it seems to be consumed very very slowly (can see accounting being sent to Server2) or not consumed (simply stops sending accouting to Server2) by the proxy server: total 4265476 -rw------- 1 root root 1246037041 Jul 20 23:59 detail-20090720 -rw------- 1 root root 3117400401 Jul 21 21:49 detail-20090721 -rw------- 1 root root 107664 Jul 20 16:59 detail.work Server's CPU utilization is low and memory utilization is high (4GB in use and 33MB free). Does anyone has a clue what is going on with our FreeRADIUS? Really don't know what to do. Appreciate any help. Thanks, Cristina Miyata