Some people have seen the detail file listener go "crazy", and use lots of CPU. I've managed to reproduce the problem, and have committed a fix to the "stable" tree. Please see http://git.freeradius.org/pre/ for tar files && debian files containing the fix. Or, see http://git.freeradius.org/ for instructions on grabbing the latest "stable" code from git. Alan DeKok.
Alan DeKok wrote:
Some people have seen the detail file listener go "crazy", and use lots of CPU. I've managed to reproduce the problem, and have committed a fix to the "stable" tree.
Please see http://git.freeradius.org/pre/ for tar files && debian files containing the fix. Or, see http://git.freeradius.org/ for instructions on grabbing the latest "stable" code from git.
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Alan, I just tried the 2.1.5 pre-release and the issue with the detail file listener going crazy seems to have been corrected. Thank you. However, the issue of the spooled detail files being deleted remains. Doing my best to analyze the debug file and some copies of the listener detail files before they are deleted, it appears that FR starts to read the detail file (i.e. detail.work) and then starts a loop where it keeps adding the data that is read back into the main listener file (i.e. detail-20090413). It appears to do this a number of times and then stops and deletes all of the files in the listener directory. If you need any additional information from me, please let me know. I could send you examples of the listener detail files off list if it will be of assistance. Thanks, Jim L.
JDL wrote:
I just tried the 2.1.5 pre-release and the issue with the detail file listener going crazy seems to have been corrected. Thank you.
That's good.
However, the issue of the spooled detail files being deleted remains. Doing my best to analyze the debug file and some copies of the listener detail files before they are deleted, it appears that FR starts to read the detail file (i.e. detail.work) and then starts a loop where it keeps adding the data that is read back into the main listener file (i.e. detail-20090413).
The detail module does NOT write to the detail file if the request was read from a detail file. However, it checks this only for accounting. On top of that, your configuration clearly logs to the detail file *twice*. Once when the packet is read from the network, and then again when it's read from the detail file. The solution is simple: 1) Don't write to the detail file twice. See raddb/sites-available/robust-proxy-accounting. Note that the listen section that reads from the detail file is in a virtual server. AND the "accounting" section for that virtual server does NOT log to the detail file. 2) Ensure that only the "accounting" section is logging to the detail file, and not any others.
It appears to do this a number of times and then stops and deletes all of the files in the listener directory.
Because it has processed all of the packets in all of the files. The fact that it's logged them *again* to a detail file is little more than a misconfiguration on your local system. Alan DeKok.
Alan DeKok wrote:
On top of that, your configuration clearly logs to the detail file *twice*. Once when the packet is read from the network, and then again when it's read from the detail file.
I assume you are referring to this server acct_detail.imaginenet { accounting { detail detail.imaginenet } } I was logging the accounting packets twice to try to locate the missing information. The listener should only see the second one which writes to the listener directory, correct?
The solution is simple:
1) Don't write to the detail file twice. I have commented out ALL detail lines from all active configuration files (including sites-enabled/default) except for the one that corresponds to the robust-proxy-accounting example. See raddb/sites-available/robust-proxy-accounting. My configuration matches robust-proxy-accounting as much as it can and still work in my site (see attached). Note that the listen section that reads from the detail file is in a virtual server. AND the "accounting" section for that virtual server does NOT log to the detail file.
The accounting section for that virtual server matches robust-proxy-acounting. <cid:part1.04050609.01060002@ImagineNet.net>
2) Ensure that only the "accounting" section is logging to the detail file, and not any others.
Done. The files are still being deleted when the home accounting server is down. I have attached my site configuration. Jim L. # -*- text -*- ###################################################################### # # This is a sample configuration for robust proxy accounting. # accounting packets are proxied, OR logged locally if all # home servers are down. When the home servers come back up, # the accounting packets are forwarded. # # This method enables the server to proxy all packets to the # home servers when they're up, AND to avoid writing to the # detail file in most situations. # # In most situations, proxying of accounting messages is done # in a "pass-through" fashion. If the home server does not # respond, then the proxy server does not respond to the NAS. # That means that the NAS must retransmit packets, sometimes # forever. This example shows how the proxy server can still # respond to the NAS, even if all home servers are down. # # This configuration could be done MUCH more simply if ALL # packets were written to the detail file. But that would # involve a lot more disk writes, which may not be a good idea. # # This file is NOT meant to be used as-is. It needs to be # edited to match your local configuration. # # $Id$ # ###################################################################### # Authentication Servers ###################################################################### home_server auth_home1.imaginenet.net { type = auth ipaddr = 192.168.78.115 port = 1812 secret = xxxxxxxx # Mark this home server alive ONLY when it starts being responsive status_check = status-server # Set the response timeout aggressively low. # You MAY have to increase this, depending on tests with # your local installation. response_window = 6 # response_window = 20 # zombie_period = 40 # revive_interval = 120 # check_interval = 30 # num_answers_to_alive = 3 } home_server auth_home2.imaginenet.net { type = auth ipaddr = 192.168.78.6 port = 1812 secret = xxxxxxxx # Mark this home server alive ONLY when it starts being responsive status_check = status-server # Set the response timeout aggressively low. # You MAY have to increase this, depending on tests with # your local installation. response_window = 6 # response_window = 20 # zombie_period = 40 # revive_interval = 120 # check_interval = 30 # num_answers_to_alive = 3 } home_server_pool auth_pool.imaginenet { type = load-balance # type = fail-over home_server = auth_home1.imaginenet.net home_server = auth_home2.imaginenet.net } # Accounting Servers ###################################################################### home_server acct_home1.imaginenet.net { type = acct ipaddr = 192.168.78.115 port = 1813 secret = xxxxxxxx # Mark this home server alive ONLY when it starts being responsive status_check = status-server # Set the response timeout aggressively low. # You MAY have to increase this, depending on tests with # your local installation. response_window = 6 # response_window = 20 # zombie_period = 40 # revive_interval = 120 # check_interval = 30 # num_answers_to_alive = 3 } home_server acct_detail.imaginenet { virtual_server = acct_detail.imaginenet } home_server_pool acct_pool.imaginenet { type = load-balance # type = fail-over home_server = acct_home1.imaginenet.net # If all home servers are down, try a home server that is a local virtual server. fallback = acct_detail.imaginenet # for pre/post-proxy policies virtual_server = ImagineNet } # Realms ###################################################################### realm imaginenet.net { auth_pool = auth_pool.imaginenet acct_pool = acct_pool.imaginenet nostrip } realm linkup.us { auth_pool = auth_pool.imaginenet acct_pool = acct_pool.imaginenet nostrip } # (3) Define a realm for these home servers. # It should NOT be used as part of normal proxying decisions! realm acct_realm.imaginenet { acct_pool = acct_pool.imaginenet nostrip } # Detail File Writer ###################################################################### # (4) Define a detail file writer. # raddb/modules/imaginenet contains the following #detail detail.imaginenet { # detailfile = ${radacctdir}/ImagineNet/detail-%Y%m%d:%H #} # (5) Define the virtual server to write the packets to the detail file # This will be called when ALL home servers are down, because of the # "fallback" configuration in the home server pool. server acct_detail.imaginenet { #preacct { # attr_filter.imaginenet #} accounting { #detail detail.imaginenet } } # (6) Define a virtual server to handle pre/post-proxy re-writing server ImagineNet { # Read accounting packets from the detail file(s) for # the home server. # # Note that you can have only ONE "listen" section reading # detail files from a particular directory. That is why the # destination host name is used as part of the directory name # below. Having two "listen" sections reading detail files # from the same directory WILL cause problems. The packets # may be read by one, the other, or both "listen" sections. listen { type = detail filename = "${radacctdir}/ImagineNet/detail-*:*" #filename = "${radacctdir}/ImagineNet/detail-*" load_factor = 10 } pre-proxy { # Insert pre-proxy rules here } post-proxy { # Insert post-proxy rules here # This will be called when the CURRENT packet failed # to be proxied. This may happen when one home server # suddenly goes down, even though another home server # may be alive. # # i.e. the current request has run out of time, so it # cannot fail over to another (possibly) alive server. # # We want to respond to the NAS, so that it can stop # re-sending the packet. We write the packet to the # "detail" file, where it will be read, and sent to # another home server. # Post-Proxy-Type Fail { detail.imaginenet } } # All packets read from the detail file are proxied back to # the home servers. # # The normal pre/post-proxy rules are applied to them, too. # # If the home servers are STILL down, then the server stops # reading the detail file, and queues the packets for a later # retransmission. The Post-Proxy-Type "Fail" handler is NOT # called. # # When the home servers come back up, the packets are forwarded, # and the detail file processed as normal. accounting { # You may want accounting policies here... update control { Proxy-To-Realm := "acct_realm.imaginenet" } } }
I have performed the following actions, however, the listener files are still being deleted when the home accounting server is down. Alan DeKok wrote:
1) Don't write to the detail file twice.
I have commented out ALL detail lines from all active configuration files (including sites-enabled/default) except for the one that corresponds to the robust-proxy-accounting example.
See raddb/sites-available/robust-proxy-accounting.
My configuration matches robust-proxy-accounting as much as it can and still work in my site (see attached).
Note that the listen section that reads from the detail file is in a virtual server. AND the "accounting" section for that virtual server does NOT log to the detail file.
The accounting section for that virtual server matches robust-proxy-accounting.
2) Ensure that only the "accounting" section is logging to the detail file, and not any others.
Done. I have attached my site configuration for reference. Jim L. # -*- text -*- ###################################################################### # # This is a sample configuration for robust proxy accounting. # accounting packets are proxied, OR logged locally if all # home servers are down. When the home servers come back up, # the accounting packets are forwarded. # # This method enables the server to proxy all packets to the # home servers when they're up, AND to avoid writing to the # detail file in most situations. # # In most situations, proxying of accounting messages is done # in a "pass-through" fashion. If the home server does not # respond, then the proxy server does not respond to the NAS. # That means that the NAS must retransmit packets, sometimes # forever. This example shows how the proxy server can still # respond to the NAS, even if all home servers are down. # # This configuration could be done MUCH more simply if ALL # packets were written to the detail file. But that would # involve a lot more disk writes, which may not be a good idea. # # This file is NOT meant to be used as-is. It needs to be # edited to match your local configuration. # # $Id$ # ###################################################################### # Authentication Servers ###################################################################### home_server auth_home1.imaginenet.net { type = auth ipaddr = 192.168.78.115 port = 1812 secret = xxxxxxxx # Mark this home server alive ONLY when it starts being responsive status_check = status-server # Set the response timeout aggressively low. # You MAY have to increase this, depending on tests with # your local installation. response_window = 6 # response_window = 20 # zombie_period = 40 # revive_interval = 120 # check_interval = 30 # num_answers_to_alive = 3 } home_server auth_home2.imaginenet.net { type = auth ipaddr = 192.168.78.6 port = 1812 secret = xxxxxxxx # Mark this home server alive ONLY when it starts being responsive status_check = status-server # Set the response timeout aggressively low. # You MAY have to increase this, depending on tests with # your local installation. response_window = 6 # response_window = 20 # zombie_period = 40 # revive_interval = 120 # check_interval = 30 # num_answers_to_alive = 3 } home_server_pool auth_pool.imaginenet { type = load-balance # type = fail-over home_server = auth_home1.imaginenet.net home_server = auth_home2.imaginenet.net } # Accounting Servers ###################################################################### home_server acct_home1.imaginenet.net { type = acct ipaddr = 192.168.78.115 port = 1813 secret = xxxxxxxx # Mark this home server alive ONLY when it starts being responsive status_check = status-server # Set the response timeout aggressively low. # You MAY have to increase this, depending on tests with # your local installation. response_window = 6 # response_window = 20 # zombie_period = 40 # revive_interval = 120 # check_interval = 30 # num_answers_to_alive = 3 } home_server acct_detail.imaginenet { virtual_server = acct_detail.imaginenet } home_server_pool acct_pool.imaginenet { type = load-balance # type = fail-over home_server = acct_home1.imaginenet.net # If all home servers are down, try a home server that is a local virtual server. fallback = acct_detail.imaginenet # for pre/post-proxy policies virtual_server = ImagineNet } # Realms ###################################################################### realm imaginenet.net { auth_pool = auth_pool.imaginenet acct_pool = acct_pool.imaginenet nostrip } realm linkup.us { auth_pool = auth_pool.imaginenet acct_pool = acct_pool.imaginenet nostrip } # (3) Define a realm for these home servers. # It should NOT be used as part of normal proxying decisions! realm acct_realm.imaginenet { acct_pool = acct_pool.imaginenet nostrip } # Detail File Writer ###################################################################### # (4) Define a detail file writer. # raddb/modules/imaginenet contains the following #detail detail.imaginenet { # detailfile = ${radacctdir}/ImagineNet/detail-%Y%m%d:%H #} # (5) Define the virtual server to write the packets to the detail file # This will be called when ALL home servers are down, because of the # "fallback" configuration in the home server pool. server acct_detail.imaginenet { #preacct { # attr_filter.imaginenet #} accounting { #detail detail.imaginenet } } # (6) Define a virtual server to handle pre/post-proxy re-writing server ImagineNet { # Read accounting packets from the detail file(s) for # the home server. # # Note that you can have only ONE "listen" section reading # detail files from a particular directory. That is why the # destination host name is used as part of the directory name # below. Having two "listen" sections reading detail files # from the same directory WILL cause problems. The packets # may be read by one, the other, or both "listen" sections. listen { type = detail filename = "${radacctdir}/ImagineNet/detail-*:*" #filename = "${radacctdir}/ImagineNet/detail-*" load_factor = 10 } pre-proxy { # Insert pre-proxy rules here } post-proxy { # Insert post-proxy rules here # This will be called when the CURRENT packet failed # to be proxied. This may happen when one home server # suddenly goes down, even though another home server # may be alive. # # i.e. the current request has run out of time, so it # cannot fail over to another (possibly) alive server. # # We want to respond to the NAS, so that it can stop # re-sending the packet. We write the packet to the # "detail" file, where it will be read, and sent to # another home server. # Post-Proxy-Type Fail { detail.imaginenet } } # All packets read from the detail file are proxied back to # the home servers. # # The normal pre/post-proxy rules are applied to them, too. # # If the home servers are STILL down, then the server stops # reading the detail file, and queues the packets for a later # retransmission. The Post-Proxy-Type "Fail" handler is NOT # called. # # When the home servers come back up, the packets are forwarded, # and the detail file processed as normal. accounting { # You may want accounting policies here... update control { Proxy-To-Realm := "acct_realm.imaginenet" } } }
JDL wrote:
I have performed the following actions, however, the listener files are still being deleted when the home accounting server is down.
Please run it in debugging mode. The output is likely too large to post here, but only debugging mode will tell you what's happening. It will print out when it's renaming a file, reading a file, etc.
My configuration matches robust-proxy-accounting as much as it can and still work in my site (see attached).
OK... I'll take a look at repeating this myself. Alan DeKok.
I have performed the following actions, however, the listener files are still being deleted when the home accounting server is down. # Mark this home server alive ONLY when it starts being responsive status_check = status-server You have altered this from robust-proxy-accounting example. Ivan Kalik Kalik Informatika ISP
participants (3)
-
Alan DeKok -
Ivan Kalik -
JDL