Backporting TLS fixes to Fedora and RHEL
Hello everyone, I'm the maintainer for FreeRADIUS in RHEL and Fedora. We have found an issue when using FreeRADIUS 3.0.21 and OpenSSL 3.0. Running eapol_test with the attached config (EAP-TTLS-TLS) fails with the following errors (logs attached): (9) eap_ttls: ERROR: Invalid ACK received: 256 (9) eap_ttls: ERROR: [eaptls verify] = invalid (9) eap_ttls: ERROR: [eaptls process] = invalid (9) eap: ERROR: Failed continuing EAP TTLS (21) session. EAP sub-module failed Tried updating to 3.0.23 and the issue seems to be fixed. However due to the updates policy we can't do a full upgrade, so we have to backport fixes to 3.0.21. I am having issues finding the commit(s) that fix this issue, so any help would be appreciated. I'm not sure this is related, but we are hitting an error with the same error message as this one but using MSCHAPv2. Here's the report: https://bugzilla.redhat.com/show_bug.cgi?id=2014525 This is still valid in the latest FreeRADIUS release (3.0.25). Thank you!
On 15/10/2021 16:16, Antonio Torres wrote:
Tried updating to 3.0.23 and the issue seems to be fixed. However due to the updates policy we can't do a full upgrade, so we have to backport fixes to 3.0.21. I am having issues finding the commit(s) that fix this issue, so any help would be appreciated.
Nice try. RedHat are *paid* to look after their distribution, and you're asking us to investigate an an issue for free, in an obsolete version. Not sure that's going to go down too well. If 3.0.23 works, then great, upgrade to that instead. Or even 3.0.25, rather than yet another obsolete version.
I'm not sure this is related, but we are hitting an error with the same error message as this one but using MSCHAPv2. Here's the report: https://bugzilla.redhat.com/show_bug.cgi?id=2014525 This is still valid in the latest FreeRADIUS release (3.0.25).
Try it with a non-bleeding edge version of OpenSSL? I don't know if anyone's even looked at that yet. -- Matthew
On Fri, Oct 15, 2021 at 5:33 PM Matthew Newton <mcn@freeradius.org> wrote:
On 15/10/2021 16:16, Antonio Torres wrote:
Tried updating to 3.0.23 and the issue seems to be fixed. However due to the updates policy we can't do a full upgrade, so we have to backport fixes to 3.0.21. I am having issues finding the commit(s) that fix this issue, so any help would be appreciated.
Nice try. RedHat are *paid* to look after their distribution, and you're asking us to investigate an an issue for free, in an obsolete version.
Not sure that's going to go down too well.
I am sorry my message gave that impression. I'm not asking for anyone to do my job, I'm just asking for some pointers to identify this fix, since the release notes don't mention commit hashes or PR numbers. Rest assured that we have dedicated a considerable amount of time to test and investigate before asking for help.
If 3.0.23 works, then great, upgrade to that instead. Or even 3.0.25, rather than yet another obsolete version.
I'm not sure this is related, but we are hitting an error with the same error message as this one but using MSCHAPv2. Here's the report: https://bugzilla.redhat.com/show_bug.cgi?id=2014525 This is still valid in the latest FreeRADIUS release (3.0.25).
Try it with a non-bleeding edge version of OpenSSL? I don't know if anyone's even looked at that yet. -- Matthew - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On Oct 15, 2021, at 11:44 AM, Antonio Torres <antorres@redhat.com> wrote:
I am sorry my message gave that impression. I'm not asking for anyone to do my job, I'm just asking for some pointers to identify this fix,
$ git log
since the release notes don't mention commit hashes or PR numbers. Rest assured that we have dedicated a considerable amount of time to test and investigate before asking for help.
And if I may ask... why can OpenSSL go through a MAJOR version upgrade, but FreeRADIUS isn't allowed a MINOR version upgrade? OpenSSL 3.0.0 has other bugs, too. Some things simply don't work. I am rather surprised that there's any expectation that the system could do a major version upgrade on OpenSSL, and have everything work. I would pretty much expect random breakages in random applications. It's a very bad idea. But I guess it's driven by "corporate policies", so you're stuck trying to debug a complex problem created by those policies. Alan DeKok.
On Oct 15, 2021, at 11:16 AM, Antonio Torres <antorres@redhat.com> wrote:
I'm the maintainer for FreeRADIUS in RHEL and Fedora. We have found an issue when using FreeRADIUS 3.0.21 and OpenSSL 3.0. Running eapol_test with the attached config (EAP-TTLS-TLS) fails with the following errors (logs attached):
(9) eap_ttls: ERROR: Invalid ACK received: 256
That's due to magic changes in the internals of OpenSSL 3.0.0.
(9) eap_ttls: ERROR: [eaptls verify] = invalid (9) eap_ttls: ERROR: [eaptls process] = invalid (9) eap: ERROR: Failed continuing EAP TTLS (21) session. EAP sub-module failed
Tried updating to 3.0.23 and the issue seems to be fixed. However due to the updates policy we can't do a full upgrade, so we have to backport fixes to 3.0.21. I am having issues finding the commit(s) that fix this issue, so any help would be appreciated.
I'll echo Matthew here.
I'm not sure this is related, but we are hitting an error with the same error message as this one but using MSCHAPv2. Here's the report: https://bugzilla.redhat.com/show_bug.cgi?id=2014525 This is still valid in the latest FreeRADIUS release (3.0.25).
We're happy to do bug fixes for our software. We're rather less happy to do work for free, to debug issues created by corporate policies. Policies which we have no control over. To be clear: RedHat makes rather a lot more money off of FreeRADIUS than I do. RedHat has shared precisely *zero* of that revenue with me. Ever. RedHat has in fact competed with me for business, and is actively trying to get customers away from me. At the same time, we get RedHat customers asking us to help them. They're usually running versions which are years out of date, due to "no upgrade" policies like the above. When told "just upgrade to a version WE support", the answer is "No, I'm paying RedHat for support!" Except RH isn't supporting them, and isn't fixing the bugs. All in all, we fix bugs, and we're happy to work with people. But make no mistake, the corporate approach is to leech off of my work, and then turn around and bill their customers for it. That's allowed by the GPL, but it doesn't make me inclined to fix issues created by the internal policies of a billion-dollar corporation. Alan DeKok.
On Fri, Oct 15, 2021 at 5:50 PM Alan DeKok <aland@deployingradius.com> wrote:
On Oct 15, 2021, at 11:16 AM, Antonio Torres <antorres@redhat.com> wrote:
I'm the maintainer for FreeRADIUS in RHEL and Fedora. We have found an issue when using FreeRADIUS 3.0.21 and OpenSSL 3.0. Running eapol_test with the attached config (EAP-TTLS-TLS) fails with the following errors (logs attached):
(9) eap_ttls: ERROR: Invalid ACK received: 256
That's due to magic changes in the internals of OpenSSL 3.0.0.
(9) eap_ttls: ERROR: [eaptls verify] = invalid (9) eap_ttls: ERROR: [eaptls process] = invalid (9) eap: ERROR: Failed continuing EAP TTLS (21) session. EAP sub-module failed
Tried updating to 3.0.23 and the issue seems to be fixed. However due to the updates policy we can't do a full upgrade, so we have to backport fixes to 3.0.21. I am having issues finding the commit(s) that fix this issue, so any help would be appreciated.
I'll echo Matthew here.
I'm not sure this is related, but we are hitting an error with the same error message as this one but using MSCHAPv2. Here's the report: https://bugzilla.redhat.com/show_bug.cgi?id=2014525 This is still valid in the latest FreeRADIUS release (3.0.25).
We're happy to do bug fixes for our software. We're rather less happy to do work for free, to debug issues created by corporate policies. Policies which we have no control over.
The bug report I have linked is for the latest FreeRADIUS release and the latest OpenSSL release. The updates policy for the distribution has nothing to do with it. I understand maybe a better approach would be to directly report this in the issue tracker instead of asking in the mailing list. As for the other issue, I haven't asked FreeRADIUS developers to debug or even investigate. I totally understand that the focus is in the latest release. My only ask was for help identifying a work that has already been done. Once again I am deeply sorry for this situation. Thank you for your work on FreeRADIUS.
To be clear: RedHat makes rather a lot more money off of FreeRADIUS than I do. RedHat has shared precisely *zero* of that revenue with me. Ever. RedHat has in fact competed with me for business, and is actively trying to get customers away from me.
At the same time, we get RedHat customers asking us to help them. They're usually running versions which are years out of date, due to "no upgrade" policies like the above. When told "just upgrade to a version WE support", the answer is "No, I'm paying RedHat for support!" Except RH isn't supporting them, and isn't fixing the bugs.
All in all, we fix bugs, and we're happy to work with people. But make no mistake, the corporate approach is to leech off of my work, and then turn around and bill their customers for it. That's allowed by the GPL, but it doesn't make me inclined to fix issues created by the internal policies of a billion-dollar corporation.
Alan DeKok.
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
On Oct 15, 2021, at 12:09 PM, Antonio Torres <antorres@redhat.com> wrote:
The bug report I have linked is for the latest FreeRADIUS release and the latest OpenSSL release.
Sure, we'll take a look.
The updates policy for the distribution has nothing to do with it. I understand maybe a better approach would be to directly report this in the issue tracker instead of asking in the mailing list.
Whatever works.
As for the other issue, I haven't asked FreeRADIUS developers to debug or even investigate. I totally understand that the focus is in the latest release. My only ask was for help identifying a work that has already been done.
Once again I am deeply sorry for this situation. Thank you for your work on FreeRADIUS.
We do our best. We know it's not your fault that you're in this situation. But sometimes it is useful to point out underlying issues. Aalan DeKok.
On Oct 15, 2021, at 10:16 AM, Antonio Torres <antorres@redhat.com> wrote:
Hello everyone,
I'm the maintainer for FreeRADIUS in RHEL and Fedora. We have found an issue when using FreeRADIUS 3.0.21 and OpenSSL 3.0. Running eapol_test with the attached config (EAP-TTLS-TLS) fails with the following errors (logs attached):
Note that eapol_test/hostapd/wpa_supplicant does not support OpenSSL 3.0.0. I needed to build it against OpenSSL 1.1.x in our CI to get our tests functional when I fixed FreeRADIUS master to work with OpenSSL 3.0.0. There was nothing from the work that I did that would suggest that OpenSSL 3.0.0 would not work with FreeRADIUS 3. Most of the fixes were swapping away from deprecated low level APIs to the EVP API. Have you tried building eapol_test/wpa_supplicant with an older version of OpenSSL? -Arran
participants (4)
-
Alan DeKok -
Antonio Torres -
Arran Cudbard-Bell -
Matthew Newton