Trouble with OCSP
Hello, We setup OCSP validation on our FreeRADIUS servers a while back and it had been working great until yesterday when we started seeing errors like: Dec 03 14:03:09 radiusd[109894]: (437) eap_tls: ERROR: (TLS) Alert write:fatal:internal error Dec 03 14:03:09 radiusd[109894]: (437) eap_tls: ERROR: (TLS) Server : Error in error Dec 03 14:03:09 radiusd[109894]: (437) eap_tls: ERROR: (TLS) Failed reading from OpenSSL Dec 03 14:03:09 radiusd[109894]: (437) Login incorrect (eap_tls: ocsp: Couldn't get OCSP response): [HOSTNAME] We discovered some errors on our Windows OCSP responders due to configuration certificates expiring and resolved that. However, even after that Windows machines are unable to authenticate. Running in debug mode we see:  Certificate chain - 1 cert(s) untrusted (TLS) untrusted certificate with depth [0] subject name /CN=staff01.ad.nwra.com (14) eap_tls: Starting OCSP Request (14) eap_tls: WARNING: (TLS) ocsp: No OCSP URL in certificate, falling back to configured URL (14) eap_tls: ocsp: Using responder URL "http://ocsp.ad.nwra.com:80/ocsp/" (14) eap_tls: ERROR: ocsp: Couldn't verify OCSP basic response (14) eap_tls: ERROR: (TLS) ocsp: Certificate has been expired/revoked (14) eap_tls: (TLS) send TLS 1.2 Alert, fatal internal_error (14) eap_tls: ERROR: (TLS) Alert write:fatal:internal error (14) eap_tls: ERROR: (TLS) Server : Error in error (14) eap_tls: ERROR: (TLS) Failed reading from OpenSSL (14) eap_tls: ERROR: (TLS) error:13800070:OCSP routines::root ca not trusted (14) eap_tls: ERROR: (TLS) error:0A000086:SSL routines::certificate verify failed (14) eap_tls: ERROR: (TLS) System call (I/O) error (-1) (14) eap_tls: ERROR: (TLS) EAP Receive handshake failed during operation (14) eap_tls: ERROR: [eaptls process] = fail (14) eap: ERROR: Failed continuing EAP TLS (13) session. EAP sub-module failed We have 3 certificates in the FreeRADIUS ca_file configuration - an AD CA cert that is expiring in about 18 months, a new AD CA cert that was issued in October, and an IPA CA cert - and that's been the case since the new AD CA cert was issued. That should cover all of the certificates issued internally, so I don't see why radiusd is complaining about an untrusted cert. Running openssl ocsp checks against the Windows ocsp responders work fine. I'm at a loss. Any ideas, things to check? ocsp config is simply enable = yes, and specifying the url fallback shown above. This is with freeradius-3.0.20-15.module_el8.10.0+3873+5b7fed0f.x86_64 freeradius-3.0.21-40.el9_4.x86_64 -- Orion Poplawski he/him/his - surely the least important thing about me IT Systems Manager 720-772-5637 NWRA, Boulder/CoRA Office FAX: 303-415-9702 3380 Mitchell Lane orion@nwra.com Boulder, CO 80301 https://www.nwra.com/
On 12/4/24 08:03, orion@nwra.com wrote:
Hello,
We setup OCSP validation on our FreeRADIUS servers a while back and it had been working great until yesterday when we started seeing errors like:
Dec 03 14:03:09 radiusd[109894]: (437) eap_tls: ERROR: (TLS) Alert write:fatal:internal error Dec 03 14:03:09 radiusd[109894]: (437) eap_tls: ERROR: (TLS) Server : Error in error Dec 03 14:03:09 radiusd[109894]: (437) eap_tls: ERROR: (TLS) Failed reading from OpenSSL Dec 03 14:03:09 radiusd[109894]: (437) Login incorrect (eap_tls: ocsp: Couldn't get OCSP response): [HOSTNAME]
We discovered some errors on our Windows OCSP responders due to configuration certificates expiring and resolved that.
However, even after that Windows machines are unable to authenticate. Running in debug mode we see:
 Certificate chain - 1 cert(s) untrusted (TLS) untrusted certificate with depth [0] subject name / CN=staff01.ad.nwra.com (14) eap_tls: Starting OCSP Request (14) eap_tls: WARNING: (TLS) ocsp: No OCSP URL in certificate, falling back to configured URL (14) eap_tls: ocsp: Using responder URL "http://ocsp.ad.nwra.com:80/ocsp/" (14) eap_tls: ERROR: ocsp: Couldn't verify OCSP basic response (14) eap_tls: ERROR: (TLS) ocsp: Certificate has been expired/revoked (14) eap_tls: (TLS) send TLS 1.2 Alert, fatal internal_error (14) eap_tls: ERROR: (TLS) Alert write:fatal:internal error (14) eap_tls: ERROR: (TLS) Server : Error in error (14) eap_tls: ERROR: (TLS) Failed reading from OpenSSL (14) eap_tls: ERROR: (TLS) error:13800070:OCSP routines::root ca not trusted (14) eap_tls: ERROR: (TLS) error:0A000086:SSL routines::certificate verify failed (14) eap_tls: ERROR: (TLS) System call (I/O) error (-1) (14) eap_tls: ERROR: (TLS) EAP Receive handshake failed during operation (14) eap_tls: ERROR: [eaptls process] = fail (14) eap: ERROR: Failed continuing EAP TLS (13) session. EAP sub-module failed
We have 3 certificates in the FreeRADIUS ca_file configuration - an AD CA cert that is expiring in about 18 months, a new AD CA cert that was issued in October, and an IPA CA cert - and that's been the case since the new AD CA cert was issued. That should cover all of the certificates issued internally, so I don't see why radiusd is complaining about an untrusted cert.
Running openssl ocsp checks against the Windows ocsp responders work fine.
I think it is just having trouble with ocsp checks of certs issued by the older AD CA cert. Here is a successful check of an IPA issued cert: Certificate chain - 1 cert(s) untrusted (TLS) untrusted certificate with depth [1] subject name /O=NWRA.COM/CN=Certificate Authority (TLS) untrusted certificate with depth [0] subject name /O=NWRA.COM/CN=FQDN (6) eap_tls: Starting OCSP Request (6) eap_tls: ocsp: Using responder URL "http://ipa-ca.nwra.com:80/ca/ocsp" This Update: Dec 4 15:50:19 2024 GMT (6) eap_tls: ocsp: Cert status: good (6) eap_tls: ocsp: Certificate is valid It also has the message about untrusted certs (although it also mentions the CA cert - so maybe the windows clients don't provide it?), so that does not seem particularly relevant. I guess the key then is: (14) eap_tls: ERROR: ocsp: Couldn't verify OCSP basic response But why? openssl doesn't complain: $ openssl ocsp -issuer /etc/pki/ca-trust/source/anchors/ad.nwra.com.crt -cert orionad.crt -url http://ocsp.ad.nwra.com/ocsp Response verify OK orionad.crt: good This Update: Dec 3 17:50:03 2024 GMT Next Update: Dec 5 06:10:03 2024 GMT -- Orion Poplawski he/him/his - surely the least important thing about me IT Systems Manager 720-772-5637 NWRA, Boulder/CoRA Office FAX: 303-415-9702 3380 Mitchell Lane orion@nwra.com Boulder, CO 80301 https://www.nwra.com/
On Dec 4, 2024, at 11:53 AM, Orion Poplawski <orion@nwra.com> wrote:
I think it is just having trouble with ocsp checks of certs issued by the older AD CA cert. Here is a successful check of an IPA issued cert: .. It also has the message about untrusted certs (although it also mentions the CA cert - so maybe the windows clients don't provide it?), so that does not seem particularly relevant. I guess the key then is:
(14) eap_tls: ERROR: ocsp: Couldn't verify OCSP basic response
But why? openssl doesn't complain:
We call an OpenSSL function to do the OCSP verification, and that function returns "failed". Why? OpenSSL magic. i.e. OpenSSL doesn't give FreeRADIUS any reason why. There's just a "failed" response. No error result, nothing useful which we can print.
$ openssl ocsp -issuer /etc/pki/ca-trust/source/anchors/ad.nwra.com.crt -cert orionad.crt -url http://ocsp.ad.nwra.com/ocsp Response verify OK orionad.crt: good This Update: Dec 3 17:50:03 2024 GMT Next Update: Dec 5 06:10:03 2024 GMT
You're passing the issuer here, though. OpenSSL needs the *entire* certificate chain for verification. If it only has an intermediate certificate, then it will fail. So did you add the issuer certificate to the FreeRADIUS configuration? i.e. put it into "ca_path". Alan DeKok.
On 12/4/24 10:40, Alan DeKok wrote:
On Dec 4, 2024, at 11:53 AM, Orion Poplawski <orion@nwra.com> wrote:
I think it is just having trouble with ocsp checks of certs issued by the older AD CA cert. Here is a successful check of an IPA issued cert: .. It also has the message about untrusted certs (although it also mentions the CA cert - so maybe the windows clients don't provide it?), so that does not seem particularly relevant. I guess the key then is:
(14) eap_tls: ERROR: ocsp: Couldn't verify OCSP basic response
But why? openssl doesn't complain:
We call an OpenSSL function to do the OCSP verification, and that function returns "failed". Why? OpenSSL magic.
i.e. OpenSSL doesn't give FreeRADIUS any reason why. There's just a "failed" response. No error result, nothing useful which we can print.
$ openssl ocsp -issuer /etc/pki/ca-trust/source/anchors/ad.nwra.com.crt -cert orionad.crt -url http://ocsp.ad.nwra.com/ocsp Response verify OK orionad.crt: good This Update: Dec 3 17:50:03 2024 GMT Next Update: Dec 5 06:10:03 2024 GMT
You're passing the issuer here, though. OpenSSL needs the *entire* certificate chain for verification. If it only has an intermediate certificate, then it will fail.
So did you add the issuer certificate to the FreeRADIUS configuration? i.e. put it into "ca_path".
I have: tls-config tls-common { verify_depth = 0 ca_path = "/etc/raddb/certs" pem_file_type = yes private_key_file = "/etc/pki/tls/private/FQDN.key" certificate_file = "/etc/pki/tls/certs/FQDN.crt" ca_file = "/etc/raddb/certs/nwra_ad_ipa_ca.crt" private_key_password = <<< secret >>> fragment_size = 1024 include_length = yes auto_chain = yes check_crl = no check_all_crl = no ca_path_reload_interval = 0 cipher_list = "DEFAULT" cipher_server_preference = no reject_unknown_intermediate_ca = no ecdh_curve = "prime256v1" tls_min_version = "1.2" cache { enable = no lifetime = 24 max_entries = 255 } verify { skip_if_ocsp_ok = no } ocsp { enable = yes override_cert_url = no url = "http://ocsp.ad.nwra.com/ocsp/" use_nonce = yes timeout = 0 softfail = no } } ca_file's nwra_ad_ipa_ca.crt contains all 3 of our CA certs in use. There are no intermediate certs. I did just now realize that ca_path would not have been working properly currently because openssl rehash requires all of the files there to only contain 1 cert. So I split the AD CA cert file into two, rehashed again, and tried again - but I'm still getting the same failure. lrwxrwxrwx. 1 root root 16 Dec 4 11:00 241bed0c.0 -> ad.nwra.com2.crt lrwxrwxrwx. 1 root root 15 Dec 4 11:00 241bed0c.1 -> ad.nwra.com.crt lrwxrwxrwx. 1 root root 10 Sep 1 2023 599be2cf.0 -> ipa_ca.crt (40) eap_tls: ERROR: ocsp: Couldn't verify OCSP basic response (40) eap_tls: ERROR: (TLS) ocsp: Certificate has been expired/revoked (40) eap_tls: (TLS) send TLS 1.2 Alert, fatal internal_error (40) eap_tls: ERROR: (TLS) Alert write:fatal:internal error (40) eap_tls: ERROR: (TLS) Server : Error in error (40) eap_tls: ERROR: (TLS) Failed reading from OpenSSL (40) eap_tls: ERROR: (TLS) error:13800070:OCSP routines::root ca not trusted (40) eap_tls: ERROR: (TLS) error:0A000086:SSL routines::certificate verify failed (40) eap_tls: ERROR: (TLS) System call (I/O) error (-1) (40) eap_tls: ERROR: (TLS) EAP Receive handshake failed during operation It seems like openssl might be thinking that the signing cert used for the ocsp response might be expired/revoked, but who knows. If it really was I would have expected the openssl ocsp command to fail as well. -- Orion Poplawski he/him/his - surely the least important thing about me IT Systems Manager 720-772-5637 NWRA, Boulder/CoRA Office FAX: 303-415-9702 3380 Mitchell Lane orion@nwra.com Boulder, CO 80301 https://www.nwra.com/
On 12/4/24 12:03, orion@nwra.com wrote:
On 12/4/24 10:40, Alan DeKok wrote:
On Dec 4, 2024, at 11:53 AM, Orion Poplawski <orion@nwra.com> wrote:
I think it is just having trouble with ocsp checks of certs issued by the older AD CA cert. Here is a successful check of an IPA issued cert: .. It also has the message about untrusted certs (although it also mentions the CA cert - so maybe the windows clients don't provide it?), so that does not seem particularly relevant. I guess the key then is:
(14) eap_tls: ERROR: ocsp: Couldn't verify OCSP basic response
But why? openssl doesn't complain:
We call an OpenSSL function to do the OCSP verification, and that function returns "failed". Why? OpenSSL magic.
i.e. OpenSSL doesn't give FreeRADIUS any reason why. There's just a "failed" response. No error result, nothing useful which we can print.
$ openssl ocsp -issuer /etc/pki/ca-trust/source/anchors/ ad.nwra.com.crt -cert orionad.crt -url http://ocsp.ad.nwra.com/ocsp Response verify OK orionad.crt: good This Update: Dec 3 17:50:03 2024 GMT Next Update: Dec 5 06:10:03 2024 GMT
You're passing the issuer here, though. OpenSSL needs the *entire* certificate chain for verification. If it only has an intermediate certificate, then it will fail.
So did you add the issuer certificate to the FreeRADIUS configuration? i.e. put it into "ca_path".
I have:
tls-config tls-common { verify_depth = 0 ca_path = "/etc/raddb/certs" pem_file_type = yes private_key_file = "/etc/pki/tls/private/FQDN.key" certificate_file = "/etc/pki/tls/certs/FQDN.crt" ca_file = "/etc/raddb/certs/nwra_ad_ipa_ca.crt" private_key_password = <<< secret >>> fragment_size = 1024 include_length = yes auto_chain = yes check_crl = no check_all_crl = no ca_path_reload_interval = 0 cipher_list = "DEFAULT" cipher_server_preference = no reject_unknown_intermediate_ca = no ecdh_curve = "prime256v1" tls_min_version = "1.2" cache { enable = no lifetime = 24 max_entries = 255 } verify { skip_if_ocsp_ok = no } ocsp { enable = yes override_cert_url = no url = "http://ocsp.ad.nwra.com/ocsp/" use_nonce = yes timeout = 0 softfail = no } }
ca_file's nwra_ad_ipa_ca.crt contains all 3 of our CA certs in use. There are no intermediate certs.
I did just now realize that ca_path would not have been working properly currently because openssl rehash requires all of the files there to only contain 1 cert. So I split the AD CA cert file into two, rehashed again, and tried again - but I'm still getting the same failure.
lrwxrwxrwx. 1 root root 16 Dec 4 11:00 241bed0c.0 -> ad.nwra.com2.crt lrwxrwxrwx. 1 root root 15 Dec 4 11:00 241bed0c.1 -> ad.nwra.com.crt lrwxrwxrwx. 1 root root 10 Sep 1 2023 599be2cf.0 -> ipa_ca.crt
I think I've figured out how to get a working config again by dropping ca_file and having a working ca_path - but it seems like there might be a bug in freeradius, or at least in the docs, let me know what you think. This seems to be the same issue as this: https://serverfault.com/questions/1090456/freeradius-with-mixed-cas What is the expected relationship between ca_file and ca_path? In the docs for ca_file there is this: If ca_file is not used, then the certificate_file MUST include not only the server certificate, but ALSO all of the CA certificates used to sign the server certificate. which led me to believe that it is needed since I didn't have the CA cert in the certficate_file. The comments in the config file says: # ALL of the CA's in this list will be trusted # to issue client certificates for authentication. # # In general, you should use self-signed # certificates for 802.1x (EAP) authentication. # In that case, this CA file should contain # *one* CA certificate. Which seems a little contradictory - why say "all of the CA's" if it is only expected to contain one? Why can't it contain more than one? It appears that if ca_file is specified, ca_path is not used. It doesn't seem like this is explicitly stated in the docs. Is this expected? Thanks. -- Orion Poplawski he/him/his - surely the least important thing about me IT Systems Manager 720-772-5637 NWRA, Boulder/CoRA Office FAX: 303-415-9702 3380 Mitchell Lane orion@nwra.com Boulder, CO 80301 https://www.nwra.com/
Orion Poplawski <orion@nwra.com>
It appears that if ca_file is specified, ca_path is not used. It doesn't seem like this is explicitly stated in the docs. Is this expected?
That's the general behavior of most applications that directly use OpenSSL libraries, at least the ones I have seen. It's usually file of path, one or the other, not both. The behavior like comes from OpenSSL API, so it would have seemed obvious to someone who uses that library often when they were writing the docs, but yes it might be useful to elaborate in the documentation, and maybe a link to some generic OpenSSL-hosted source that explains their basic PKI directory schemes. (There are some rather complex systems for managing certificate preferences on multi-user/multi-security-level systems that use a systemd-like directory labyrinth... I mean... structure of overrides and such, just be glad we don't use those :-) )
On 12/4/24 15:35, Brian Julin wrote:
Orion Poplawski <orion@nwra.com>
It appears that if ca_file is specified, ca_path is not used. It doesn't seem like this is explicitly stated in the docs. Is this expected?
That's the general behavior of most applications that directly use OpenSSL libraries, at least the ones I have seen. It's usually file of path, one or the other, not both.
The behavior like comes from OpenSSL API, so it would have seemed obvious to someone who uses that library often when they were writing the docs, but yes it might be useful to elaborate in the documentation, and maybe a link to some generic OpenSSL-hosted source that explains their basic PKI directory schemes.
Hmm, I'm not entirely sure this is true for the FreeRADIUS case - it calls: /* Load the CAs we trust */ if (conf->ca_file || conf->ca_path) if (!X509_STORE_load_locations(store, conf->ca_file, conf->ca_path)) { tls_error_log(NULL, "Error reading Trusted root CA list \"%s\"", conf->ca_file); X509_STORE_free(store); return NULL; } And according to https://docs.openssl.org/3.0/man3/X509_STORE_add_cert/ : X509_STORE_load_locations_ex() combines X509_STORE_load_file_ex() and X509_STORE_load_path() for a given file and/or directory path. It is permitted to specify just a file, just a directory, or both paths. -- Orion Poplawski he/him/his - surely the least important thing about me Manager of IT Systems 720-772-5637 NWRA, Boulder Office FAX: 303-415-9702 3380 Mitchell Lane orion@nwra.com Boulder, CO 80301 https://www.nwra.com/
Orion Poplawski <orion@nwra.com> wrote:
Hmm, I'm not entirely sure this is true for the FreeRADIUS case - it calls: if (!X509_STORE_load_locations(store, conf->ca_file, conf->ca_path)) { ... X509_STORE_load_locations_ex() combines X509_STORE_load_file_ex() and X509_STORE_load_path() for a given file and/or directory path. It is permitted to specify just a file, just a directory, or both paths.
Interesting. And if I googled to the right source repo, that does this: int X509_STORE_load_locations_ex(X509_STORE *ctx, const char *file, const char *path, OSSL_LIB_CTX *libctx, const char *propq) { if (file == NULL && path == NULL) return 0; if (file != NULL && !X509_STORE_load_file_ex(ctx, file, libctx, propq)) return 0; if (path != NULL && !X509_STORE_load_path(ctx, path)) return 0; return 1; } OpenSSL seems to be a "return 1 on success or 0 on failure" codebase, so that means if both are specified but one fails, the combo fails, a sensible decision given it is a security layer, I guess.
On 12/4/24 14:51, orion@nwra.com wrote:
On 12/4/24 12:03, orion@nwra.com wrote:
On 12/4/24 10:40, Alan DeKok wrote:
On Dec 4, 2024, at 11:53 AM, Orion Poplawski <orion@nwra.com> wrote:
I think it is just having trouble with ocsp checks of certs issued by the older AD CA cert. Here is a successful check of an IPA issued cert: .. It also has the message about untrusted certs (although it also mentions the CA cert - so maybe the windows clients don't provide it?), so that does not seem particularly relevant. I guess the key then is:
(14) eap_tls: ERROR: ocsp: Couldn't verify OCSP basic response
But why? openssl doesn't complain:
We call an OpenSSL function to do the OCSP verification, and that function returns "failed". Why? OpenSSL magic.
i.e. OpenSSL doesn't give FreeRADIUS any reason why. There's just a "failed" response. No error result, nothing useful which we can print.
$ openssl ocsp -issuer /etc/pki/ca-trust/source/anchors/ ad.nwra.com.crt - cert orionad.crt -url http://ocsp.ad.nwra.com/ocsp Response verify OK orionad.crt: good This Update: Dec 3 17:50:03 2024 GMT Next Update: Dec 5 06:10:03 2024 GMT
You're passing the issuer here, though. OpenSSL needs the *entire* certificate chain for verification. If it only has an intermediate certificate, then it will fail.
So did you add the issuer certificate to the FreeRADIUS configuration? i.e. put it into "ca_path".
I have:
tls-config tls-common { verify_depth = 0 ca_path = "/etc/raddb/certs" pem_file_type = yes private_key_file = "/etc/pki/tls/private/FQDN.key" certificate_file = "/etc/pki/tls/certs/FQDN.crt" ca_file = "/etc/raddb/certs/nwra_ad_ipa_ca.crt" private_key_password = <<< secret >>> fragment_size = 1024 include_length = yes auto_chain = yes check_crl = no check_all_crl = no ca_path_reload_interval = 0 cipher_list = "DEFAULT" cipher_server_preference = no reject_unknown_intermediate_ca = no ecdh_curve = "prime256v1" tls_min_version = "1.2" cache { enable = no lifetime = 24 max_entries = 255 } verify { skip_if_ocsp_ok = no } ocsp { enable = yes override_cert_url = no url = "http://ocsp.ad.nwra.com/ocsp/" use_nonce = yes timeout = 0 softfail = no } }
ca_file's nwra_ad_ipa_ca.crt contains all 3 of our CA certs in use. There are no intermediate certs.
I did just now realize that ca_path would not have been working properly currently because openssl rehash requires all of the files there to only contain 1 cert. So I split the AD CA cert file into two, rehashed again, and tried again - but I'm still getting the same failure.
lrwxrwxrwx. 1 root root 16 Dec 4 11:00 241bed0c.0 -> ad.nwra.com2.crt lrwxrwxrwx. 1 root root 15 Dec 4 11:00 241bed0c.1 -> ad.nwra.com.crt lrwxrwxrwx. 1 root root 10 Sep 1 2023 599be2cf.0 -> ipa_ca.crt
I think I've figured out how to get a working config again by dropping ca_file and having a working ca_path - but it seems like there might be a bug in freeradius, or at least in the docs, let me know what you think.
This seems to be the same issue as this: https://serverfault.com/questions/1090456/freeradius-with-mixed-cas
Well, while I *thought* I had this briefly working, when I went to really deploy this it turns out that it is not. It appears to fail to verify OCSP responses for older AD certs issued by the previous AD CA cert. After some more reading and experimenting, I've come to the conclusion that by default openssl requires the CA cert that issued the client cert be the same as the CA cert that issued the OCSP signing cert used to sign the ocsp response. However, with the Microsoft CA and OCSP responder - the signing certificate validity is quite short - defaults to 2 weeks it seems, though we made our own that is valid for 2 months. But this has resulted in our OCSP responders no longer having valid signing certificates for the old CA cert. I'm not sure how anyone is supposed to support this situation. openssl does allow you to explicitly mark a certificate as trusted for OCSP signing with X509_add1_trust_object(), but it appears that freeradius does not support doing this. Orion -- Orion Poplawski he/him/his - surely the least important thing about me Manager of IT Systems 720-772-5637 NWRA, Boulder Office FAX: 303-415-9702 3380 Mitchell Lane orion@nwra.com Boulder, CO 80301 https://www.nwra.com/
participants (3)
-
Alan DeKok -
Brian Julin -
Orion Poplawski