Freeradius is not restarting properly (fails to quit and becomes a zombie process)
Over the last few days I've been having a recurring problem. Whenever I start Freeradius either with radiusd in a terminal or as a service in Debian, I can not restart/kill radiusd properly if it's authenticated any clients. Restarting the service says it's successful but the radius log states that port 1812 is already in use. "top" shows 100% cpu usage after I attempt to restart radiusd. In addition, kill will not work. I need to use kill -9. No errors are thrown when I try to kill it in debug mode either. It just says exiting and sits there but doesn't die. The only change I have made to radiusd.conf was to set the user and group to nobody and nogroup respectively. I've copied the contents of my eap.conf configuration file below. # -*- text -*- # # Whatever you do, do NOT set 'Auth-Type := EAP'. The server # is smart enough to figure this out on its own. The most # common side effect of setting 'Auth-Type := EAP' is that the # users then cannot use ANY other authentication method. # # $Id: eap.conf,v 1.4.4.3 2006/04/28 18:25:03 aland Exp $ # eap { # Invoke the default supported EAP type when # EAP-Identity response is received. # # The incoming EAP messages DO NOT specify which EAP # type they will be using, so it MUST be set here. # # For now, only one default EAP type may be used at a time. # # If the EAP-Type attribute is set by another module, # then that EAP type takes precedence over the # default type configured here. # default_eap_type = tls # A list is maintained to correlate EAP-Response # packets with EAP-Request packets. After a # configurable length of time, entries in the list # expire, and are deleted. # timer_expire = 60 # There are many EAP types, but the server has support # for only a limited subset. If the server receives # a request for an EAP type it does not support, then # it normally rejects the request. By setting this # configuration to "yes", you can tell the server to # instead keep processing the request. Another module # MUST then be configured to proxy the request to # another RADIUS server which supports that EAP type. # # If another module is NOT configured to handle the # request, then the request will still end up being # rejected. ignore_unknown_eap_types = no # Cisco AP1230B firmware 12.2(13)JA1 has a bug. When given # a User-Name attribute in an Access-Accept, it copies one # more byte than it should. # # We can work around it by configurably adding an extra # zero byte. cisco_accounting_username_bug = no # Supported EAP-types # # We do NOT recommend using EAP-MD5 authentication # for wireless connections. It is insecure, and does # not provide for dynamic WEP keys. # md5 { } # Cisco LEAP # # We do not recommend using LEAP in new deployments. See: # http://www.securiteam.com/tools/5TP012ACKE.html # # Cisco LEAP uses the MS-CHAP algorithm (but not # the MS-CHAP attributes) to perform it's authentication. # # As a result, LEAP *requires* access to the plain-text # User-Password, or the NT-Password attributes. # 'System' authentication is impossible with LEAP. # leap { } # Generic Token Card. # # Currently, this is only permitted inside of EAP-TTLS, # or EAP-PEAP. The module "challenges" the user with # text, and the response from the user is taken to be # the User-Password. # # Proxying the tunneled EAP-GTC session is a bad idea, # the users password will go over the wire in plain-text, # for anyone to see. # gtc { # The default challenge, which many clients # ignore.. #challenge = "Password: " # The plain-text response which comes back # is put into a User-Password attribute, # and passed to another module for # authentication. This allows the EAP-GTC # response to be checked against plain-text, # or crypt'd passwords. # # If you say "Local" instead of "PAP", then # the module will look for a User-Password # configured for the request, and do the # authentication itself. # auth_type = PAP } ## EAP-TLS # # To generate ctest certificates, run the script # # ../scripts/certs.sh # # The documents on http://www.freeradius.org/doc # are old, but may be helpful. # # See also: # # http://www.dslreports.com/forum/remark,9286052~mode=flat # tls { private_key_password = private_key_file = /etc/ssl/newCA/certs/server_key.pem # If Private key & Certificate are located in # the same file, then private_key_file & # certificate_file must contain the same file # name. certificate_file = /etc/ssl/newCA/certs/server_cert.pem # Trusted Root CA list CA_file = /etc/ssl/newCA/cacert.pem dh_file = /etc/ssl/newCA/certs/dh dh_key_length = 3072 random_file = /etc/ssl/newCA/certs/random CA_path = /etc/ssl/newCA/crl/ # # This can never exceed the size of a RADIUS # packet (4096 bytes), and is preferably half # that, to accomodate other attributes in # RADIUS packet. On most APs the MAX packet # length is configured between 1500 - 1600 # In these cases, fragment size should be # 1024 or less. # fragment_size = 1024 # include_length is a flag which is # by default set to yes If set to # yes, Total Length of the message is # included in EVERY packet we send. # If set to no, Total Length of the # message is included ONLY in the # First packet of a fragment series. # include_length = yes # Check the Certificate Revocation List # # 1) Copy CA certificates and CRLs to same directory. # 2) Execute 'c_rehash <CA certs&CRLs Directory>'. # 'c_rehash' is OpenSSL's command. # 3) Add 'CA_path=<CA certs&CRLs directory>' # to radiusd.conf's tls section. # 4) uncomment the line below. # 5) Restart radiusd check_crl = yes # # If check_cert_issuer is set, the value will # be checked against the DN of the issuer in # the client certificate. If the values do not # match, the cerficate verification will fail, # rejecting the user. # check_cert_issuer = "/C=US/ST=NJ/L=East Brunswick/CN=Jason Wittlin-Cohen's ROOT CA" # # If check_cert_cn is set, the value will # be xlat'ed and checked against the CN # in the client certificate. If the values # do not match, the certificate verification # will fail rejecting the user. # # This check is done only if the previous # "check_cert_issuer" is not set, or if # the check succeeds. # check_cert_cn = %{User-Name} # # Set this option to specify the allowed # TLS cipher suites. The format is listed # in "man 1 ciphers". cipher_list = "HIGH" } # The TTLS module implements the EAP-TTLS protocol, # which can be described as EAP inside of Diameter, # inside of TLS, inside of EAP, inside of RADIUS... # # Surprisingly, it works quite well. # # The TTLS module needs the TLS module to be installed # and configured, in order to use the TLS tunnel # inside of the EAP packet. You will still need to # configure the TLS module, even if you do not want # to deploy EAP-TLS in your network. Users will not # be able to request EAP-TLS, as it requires them to # have a client certificate. EAP-TTLS does not # require a client certificate. # #ttls { # The tunneled EAP session needs a default # EAP type which is separate from the one for # the non-tunneled EAP module. Inside of the # TTLS tunnel, we recommend using EAP-MD5. # If the request does not contain an EAP # conversation, then this configuration entry # is ignored. # default_eap_type = md5 # The tunneled authentication request does # not usually contain useful attributes # like 'Calling-Station-Id', etc. These # attributes are outside of the tunnel, # and normally unavailable to the tunneled # authentication request. # # By setting this configuration entry to # 'yes', any attribute which NOT in the # tunneled authentication request, but # which IS available outside of the tunnel, # is copied to the tunneled request. # # allowed values: {no, yes} # copy_request_to_tunnel = no # The reply attributes sent to the NAS are # usually based on the name of the user # 'outside' of the tunnel (usually # 'anonymous'). If you want to send the # reply attributes based on the user name # inside of the tunnel, then set this # configuration entry to 'yes', and the reply # to the NAS will be taken from the reply to # the tunneled request. # # allowed values: {no, yes} # use_tunneled_reply = no #} # # The tunneled EAP session needs a default EAP type # which is separate from the one for the non-tunneled # EAP module. Inside of the TLS/PEAP tunnel, we # recommend using EAP-MS-CHAPv2. # # The PEAP module needs the TLS module to be installed # and configured, in order to use the TLS tunnel # inside of the EAP packet. You will still need to # configure the TLS module, even if you do not want # to deploy EAP-TLS in your network. Users will not # be able to request EAP-TLS, as it requires them to # have a client certificate. EAP-PEAP does not # require a client certificate. # # peap { # The tunneled EAP session needs a default # EAP type which is separate from the one for # the non-tunneled EAP module. Inside of the # PEAP tunnel, we recommend using MS-CHAPv2, # as that is the default type supported by # Windows clients. # default_eap_type = mschapv2 # the PEAP module also has these configuration # items, which are the same as for TTLS. # copy_request_to_tunnel = no # use_tunneled_reply = no # When the tunneled session is proxied, the # home server may not understand EAP-MSCHAP-V2. # Set this entry to "no" to proxy the tunneled # EAP-MSCHAP-V2 as normal MSCHAPv2. # proxy_tunneled_request_as_eap = yes #} # # This takes no configuration. # # Note that it is the EAP MS-CHAPv2 sub-module, not # the main 'mschap' module. # # Note also that in order for this sub-module to work, # the main 'mschap' module MUST ALSO be configured. # # This module is the *Microsoft* implementation of MS-CHAPv2 # in EAP. There is another (incompatible) implementation # of MS-CHAPv2 in EAP by Cisco, which FreeRADIUS does not # currently support. # mschapv2 { } }
Jason Wittlin-Cohen wrote:
Over the last few days I've been having a recurring problem. Whenever I start Freeradius either with radiusd in a terminal or as a service in Debian, I can not restart/kill radiusd properly if it's authenticated any clients. Restarting the service says it's successful but the radius log states that port 1812 is already in use. "top" shows 100% cpu usage after I attempt to restart radiusd. In addition, kill will not work. I need to use kill -9. No errors are thrown when I try to kill it in debug mode either. It just says exiting and sits there but doesn't die.
Howdy Jason, Might you get any useful info by running radiusd with strace? Cheers, -- James Wakefield, Unix Administrator, Information Technology Services Division Deakin University, Geelong, Victoria 3217 Australia. Phone: 03 5227 8690 International: +61 3 5227 8690 Fax: 03 5227 8866 International: +61 3 5227 8866 E-mail: james.wakefield@deakin.edu.au Website: http://www.deakin.edu.au
select(5, [3 4], NULL, NULL, {6, 0}) = 1 (in [3], left {5, 992000}) time(NULL) = 1159497421 recvfrom(3, "\1\1\0\227\247\326\245\\\207\222(\352H\305\311\213\300"..., 4096, 0, {sa_family=AF_INET, sin_port=htons(2054), sin_addr=inet_addr("192.168.0.1")}, [16]) = 151 write(1, "rad_recv: Access-Request packet "..., 77rad_recv: Access-Request packet from host 192.168.0.1:2054, id=1, length=151 ) = 77 time(NULL) = 1159497421 write(1, "\tUser-Name = \"Jason Wittlin-Cohe"..., 35 User-Name = "Jason Wittlin-Cohen" ) = 35 write(1, "\tNAS-IP-Address = 192.168.0.1\n", 30 NAS-IP-Address = 192.168.0.1 ) = 30 write(1, "\tCalled-Station-Id = \"00160112eb"..., 36 Called-Station-Id = "00160112ebda" ) = 36 write(1, "\tCalling-Station-Id = \"00095b934"..., 37 Calling-Station-Id = "00095b93459e" ) = 37 write(1, "\tNAS-Identifier = \"00160112ebda\""..., 33 NAS-Identifier = "00160112ebda" ) = 33 write(1, "\tNAS-Port = 8\n", 14 NAS-Port = 8 ) = 14 write(1, "\tFramed-MTU = 1400\n", 19 Framed-MTU = 1400 ) = 19 write(1, "\tState = 0x8570d74429dcf8507949a"..., 44 State = 0x8570d74429dcf8507949ae638bd52940 ) = 44 write(1, "\tNAS-Port-Type = Wireless-802.11"..., 33 NAS-Port-Type = Wireless-802.11 ) = 33 write(1, "\tEAP-Message = 0x020800060d00\n", 30 EAP-Message = 0x020800060d00 ) = 30 write(1, "\tMessage-Authenticator = 0xb781d"..., 60 Message-Authenticator = 0xb781dd8563450fa51bff3ce9be35dac3 ) = 60 time(NULL) = 1159497421 write(1, " Processing the authorize secti"..., 51 Processing the authorize section of radiusd.conf ) = 51 time(NULL) = 1159497421 write(1, "modcall: entering group authoriz"..., 48modcall: entering group authorize for request 8 ) = 48 time(NULL) = 1159497421 write(1, " modcall[authorize]: module \"pr"..., 67 modcall[authorize]: module "preprocess" returns ok for request 8 ) = 67 time(NULL) = 1159497421 write(1, " modcall[authorize]: module \"ch"..., 63 modcall[authorize]: module "chap" returns noop for request 8 ) = 63 time(NULL) = 1159497421 write(1, " modcall[authorize]: module \"ms"..., 65 modcall[authorize]: module "mschap" returns noop for request 8 ) = 65 time(NULL) = 1159497421 write(1, " rlm_realm: No \'@\' in User-Na"..., 82 rlm_realm: No '@' in User-Name = "Jason Wittlin-Cohen", looking up realm NULL ) = 82 time(NULL) = 1159497421 time(NULL) = 1159497421 write(1, " rlm_realm: No such realm \"NU"..., 36 rlm_realm: No such realm "NULL" ) = 36 time(NULL) = 1159497421 write(1, " modcall[authorize]: module \"su"..., 65 modcall[authorize]: module "suffix" returns noop for request 8 ) = 65 time(NULL) = 1159497421 write(1, " rlm_eap: EAP packet type respo"..., 50 rlm_eap: EAP packet type response id 8 length 6 ) = 50 time(NULL) = 1159497421 write(1, " rlm_eap: No EAP Start, assumin"..., 68 rlm_eap: No EAP Start, assuming it's an on-going EAP conversation ) = 68 time(NULL) = 1159497421 write(1, " modcall[authorize]: module \"ea"..., 65 modcall[authorize]: module "eap" returns updated for request 8 ) = 65 time(NULL) = 1159497421 write(1, " users: Matched entry Jason W"..., 56 users: Matched entry Jason Wittlin-Cohen at line 96 ) = 56 time(NULL) = 1159497421 write(1, " modcall[authorize]: module \"fi"..., 62 modcall[authorize]: module "files" returns ok for request 8 ) = 62 time(NULL) = 1159497421 write(1, "modcall: leaving group authorize"..., 65modcall: leaving group authorize (returns updated) for request 8 ) = 65 time(NULL) = 1159497421 write(1, " rad_check_password: Found Aut"..., 43 rad_check_password: Found Auth-Type EAP ) = 43 time(NULL) = 1159497421 write(1, "auth: type \"EAP\"\n", 17auth: type "EAP" ) = 17 time(NULL) = 1159497421 write(1, " Processing the authenticate se"..., 54 Processing the authenticate section of radiusd.conf ) = 54 time(NULL) = 1159497421 write(1, "modcall: entering group authenti"..., 51modcall: entering group authenticate for request 8 ) = 51 time(NULL) = 1159497421 write(1, " rlm_eap: Request found, releas"..., 49 rlm_eap: Request found, released from the list ) = 49 time(NULL) = 1159497421 write(1, " rlm_eap: EAP/tls\n", 19 rlm_eap: EAP/tls ) = 19 time(NULL) = 1159497421 write(1, " rlm_eap: processing type tls\n", 31 rlm_eap: processing type tls ) = 31 time(NULL) = 1159497421 write(1, " rlm_eap_tls: Authenticate\n", 28 rlm_eap_tls: Authenticate ) = 28 time(NULL) = 1159497421 write(1, " rlm_eap_tls: processing TLS\n", 30 rlm_eap_tls: processing TLS ) = 30 time(NULL) = 1159497421 write(1, "rlm_eap_tls: Received EAP-TLS AC"..., 42rlm_eap_tls: Received EAP-TLS ACK message ) = 42 time(NULL) = 1159497421 write(1, " rlm_eap_tls: ack handshake is "..., 41 rlm_eap_tls: ack handshake is finished ) = 41 time(NULL) = 1159497421 write(1, " eaptls_verify returned 3 \n", 28 eaptls_verify returned 3 ) = 28 time(NULL) = 1159497421 write(1, " eaptls_process returned 3 \n", 29 eaptls_process returned 3 ) = 29 time(NULL) = 1159497421 write(1, " rlm_eap: Freeing handler\n", 27 rlm_eap: Freeing handler ) = 27 time(NULL) = 1159497421 write(1, " modcall[authenticate]: module "..., 63 modcall[authenticate]: module "eap" returns ok for request 8 ) = 63 time(NULL) = 1159497421 write(1, "modcall: leaving group authentic"..., 63modcall: leaving group authenticate (returns ok) for request 8 ) = 63 time(NULL) = 1159497421 write(1, "Login OK: [Jason Wittlin-Cohen] "..., 75Login OK: [Jason Wittlin-Cohen] (from client WLAN port 8 cli 00095b93459e) ) = 75 write(1, "Sending Access-Accept of id 1 to"..., 55Sending Access-Accept of id 1 to 192.168.0.1 port 2054 ) = 55 write(1, "\tMS-MPPE-Recv-Key = 0xf4b929a28e"..., 87 MS-MPPE-Recv-Key = 0xf4b929a28edd544381b7df6e67e298627f682b0b4a01e9de61a113c3a14176d2 ) = 87 write(1, "\tMS-MPPE-Send-Key = 0xfed62e9031"..., 87 MS-MPPE-Send-Key = 0xfed62e90314b182ecaf572ce6ee6993aa39927c7610c1644204fef340f748862 ) = 87 write(1, "\tEAP-Message = 0x03080004\n", 26 EAP-Message = 0x03080004 ) = 26 write(1, "\tMessage-Authenticator = 0x00000"..., 60 Message-Authenticator = 0x00000000000000000000000000000000 ) = 60 write(1, "\tUser-Name = \"Jason Wittlin-Cohe"..., 35 User-Name = "Jason Wittlin-Cohen" ) = 35 sendto(3, "\2\1\0\265\265\351Rn\0015F\300\0\224Nr\354\340\36\316\32"..., 181, 0, {sa_family=AF_INET, sin_port=htons(2054), sin_addr=inet_addr("192.168.0.1")}, 16) = 181 time(NULL) = 1159497421 write(1, "Finished request 8\n", 19Finished request 8 ) = 19 time(NULL) = 1159497421 write(1, "Going to the next request\n", 26Going to the next request ) = 26 time(NULL) = 1159497421 write(1, "Waking up in 6 seconds...\n", 26Waking up in 6 seconds... ) = 26 select(5, [3 4], NULL, NULL, {6, 0}) = 0 (Timeout) time(NULL) = 1159497427 time(NULL) = 1159497427 write(1, "--- Walking the entire request l"..., 40--- Walking the entire request list --- ) = 40 time(NULL) = 1159497427 write(1, "Cleaning up request 8 ID 1 with "..., 51Cleaning up request 8 ID 1 with timestamp 451c86cd ) = 51 time(NULL) = 1159497427 write(1, "Nothing to do. Sleeping until w"..., 49Nothing to do. Sleeping until we see a request. ) = 49 select(5, [3 4], NULL, NULL, NULL) = ? ERESTARTNOHAND (To be restarted) --- SIGTERM (Terminated) @ 0 (0) --- sigreturn() = ? (mask now []) time(NULL) = 1159497505 time(NULL) = 1159497505 write(1, "--- Walking the entire request l"..., 40--- Walking the entire request list --- ) = 40 time(NULL) = 1159497505 write(1, "Nothing to do. Sleeping until w"..., 49Nothing to do. Sleeping until we see a request. ) = 49 time(NULL) = 1159497505 write(1, "Exiting...\n", 11Exiting... ) = 11 rt_sigaction(SIGTERM, {SIG_IGN}, {0x80593e0, [], 0}, 8) = 0 kill(-25503, SIGTERM) = -1 ESRCH (No such process) munmap(0xb7dce000, 9144) = 0 munmap(0xb7dad000, 8940) = 0 --- SIGTERM (Terminated) @ 0 (0) --- +++ killed by SIGKILL +++ I started the server with "strace radiusd -X" and authenticated a client. I then tried to kill the process with "kill 25503" which showed the output: ) = 49 time(NULL) = 1159497505 write(1, "Exiting...\n", 11Exiting... ) = 11 rt_sigaction(SIGTERM, {SIG_IGN}, {0x80593e0, [], 0}, 8) = 0 kill(-25503, SIGTERM) = -1 ESRCH (No such process) munmap(0xb7dce000, 9144) = 0 munmap(0xb7dad000, 8940) = 0 When I used "kill -9 25503" it ended successfully: --- SIGTERM (Terminated) @ 0 (0) --- +++ killed by SIGKILL +++ James Wakefield wrote:
Jason Wittlin-Cohen wrote:
Over the last few days I've been having a recurring problem. Whenever I start Freeradius either with radiusd in a terminal or as a service in Debian, I can not restart/kill radiusd properly if it's authenticated any clients. Restarting the service says it's successful but the radius log states that port 1812 is already in use. "top" shows 100% cpu usage after I attempt to restart radiusd. In addition, kill will not work. I need to use kill -9. No errors are thrown when I try to kill it in debug mode either. It just says exiting and sits there but doesn't die.
Howdy Jason,
Might you get any useful info by running radiusd with strace?
Cheers,
Jason Wittlin-Cohen <jasonwc@brandeis.edu> wrote:
Over the last few days I've been having a recurring problem. Whenever I start Freeradius either with radiusd in a terminal or as a service in Debian, I can not restart/kill radiusd properly if it's authenticated any clients. Restarting the service says it's successful but the radius log states that port 1812 is already in use. "top" shows 100% cpu usage
It looks like http://bugs.freeradius.org/show_bug.cgi?id=365 The solution is to not re-initialize the modules on HUP. It works in *most* cases, because the code handling the HUP tries to wait until all of the modules have stopped. But if your back-end DB's are slow, it doesn't have much choice but to proceed with handling the HUP. Most people don't see it because the modules respond quickly. I'd say the first step to a work-around is to make sure none of the modules you're using are blocking the server. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
Alan DeKok wrote:
Jason Wittlin-Cohen <jasonwc@brandeis.edu> wrote:
Over the last few days I've been having a recurring problem. Whenever I start Freeradius either with radiusd in a terminal or as a service in Debian, I can not restart/kill radiusd properly if it's authenticated any clients. Restarting the service says it's successful but the radius log states that port 1812 is already in use. "top" shows 100% cpu usage
It looks like http://bugs.freeradius.org/show_bug.cgi?id=365
The solution is to not re-initialize the modules on HUP.
It works in *most* cases, because the code handling the HUP tries to wait until all of the modules have stopped. But if your back-end DB's are slow, it doesn't have much choice but to proceed with handling the HUP.
Most people don't see it because the modules respond quickly. I'd say the first step to a work-around is to make sure none of the modules you're using are blocking the server.
Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
I have discovered the root of the problem. When I enable the "check_cert_cn = %{User-Name}" option in eap.conf and successfully authenticate 1 user , a restart or stop of the radiusd service leads to a zombie process which needs to be killed with "kill -9". If this option is disabled, as is the default setting, radiusd can be restarted normally without issue. This issue does not occur if either a) no users have attempted to authenticate, or b) users have authenticated but were rejected. Is this a known issue? Jason Wittlin-Cohen
Jason Wittlin-Cohen <jasonwc@brandeis.edu> wrote:
I have discovered the root of the problem. When I enable the "check_cert_cn = %{User-Name}" option in eap.conf and successfully authenticate 1 user , a restart or stop of the radiusd service leads to a zombie process which needs to be killed with "kill -9". If this option is disabled, as is the default setting, radiusd can be restarted normally without issue. This issue does not occur if either a) no users have attempted to authenticate, or b) users have authenticated but were rejected. Is this a known issue?
No. It's *very* weird. I'll try to take a look at it this week. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog
participants (3)
-
Alan DeKok -
James Wakefield -
Jason Wittlin-Cohen