Warning about OpenSSL 1.0.2
OpenSSL 1.0.2 changes the way it interacts with FreeRADIUS. None of this is documented by OpenSSL. The result is that instead of successful authentication, you get: (6) eap_ttls: ERROR: Invalid ACK received: 256 (6) eap_ttls: ERROR: [eaptls verify] = invalid (6) eap_ttls: ERROR: [eaptls process] = invalid The only solution is to apply the patch in commit b7b5493c61. It doesn't fix the underlying OpenSSL problem, but it makes FreeRADIUS ignore the broken API calls. This problem is serious enough that we may have to issue 3.0.11, and possibly 2.2.10, also. Sadly, this isn't the first time that OpenSSL broke FreeRADIUS, or other applications. Alan DeKok.
On 10 Oct 2015, at 08:57, Alan DeKok <aland@deployingradius.com> wrote:
OpenSSL 1.0.2 changes the way it interacts with FreeRADIUS. None of this is documented by OpenSSL. The result is that instead of successful authentication, you get:
(6) eap_ttls: ERROR: Invalid ACK received: 256 (6) eap_ttls: ERROR: [eaptls verify] = invalid (6) eap_ttls: ERROR: [eaptls process] = invalid
The only solution is to apply the patch in commit b7b5493c61. It doesn't fix the underlying OpenSSL problem, but it makes FreeRADIUS ignore the broken API calls.
This problem is serious enough that we may have to issue 3.0.11, and possibly 2.2.10, also.
Have to draw a line on 2.2.x this uncertainty undermines people making the case to move to v3.0.x. 1.0.2 is not included by default in any stable releases of FreeBSD, Ubuntu/Debian, Redhat/Centos, OSX. We experienced it because homebrew has moved to OpenSSL 1.0.2. In related news, FTP server seems to be broken. shinyhead:freeradius-server-fork arr2036$ brew install freeradius-server ==> Downloading ftp://ftp.freeradius.org/pub/freeradius/freeradius-server-3.0.9.tar.bz2 curl: (78) RETR response: 550 Trying a mirror... -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
If OpenSSL 1.0.2 isn't in old systems, we can leave 2.2.10 alone. For the ftp site, old releases are in the "old" directory. :) Sent from my iPhone
On Oct 10, 2015, at 12:54 PM, Arran Cudbard-Bell <a.cudbardb@freeradius.org> wrote:
On 10 Oct 2015, at 08:57, Alan DeKok <aland@deployingradius.com> wrote:
OpenSSL 1.0.2 changes the way it interacts with FreeRADIUS. None of this is documented by OpenSSL. The result is that instead of successful authentication, you get:
(6) eap_ttls: ERROR: Invalid ACK received: 256 (6) eap_ttls: ERROR: [eaptls verify] = invalid (6) eap_ttls: ERROR: [eaptls process] = invalid
The only solution is to apply the patch in commit b7b5493c61. It doesn't fix the underlying OpenSSL problem, but it makes FreeRADIUS ignore the broken API calls.
This problem is serious enough that we may have to issue 3.0.11, and possibly 2.2.10, also.
Have to draw a line on 2.2.x this uncertainty undermines people making the case to move to v3.0.x. 1.0.2 is not included by default in any stable releases of FreeBSD, Ubuntu/Debian, Redhat/Centos, OSX.
We experienced it because homebrew has moved to OpenSSL 1.0.2.
In related news, FTP server seems to be broken.
shinyhead:freeradius-server-fork arr2036$ brew install freeradius-server ==> Downloading ftp://ftp.freeradius.org/pub/freeradius/freeradius-server-3.0.9.tar.bz2
curl: (78) RETR response: 550 Trying a mirror...
-Arran
Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team
FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
- List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Would it really be that big a deal to release a 2.2.10 that solves this there? I would considering the proximity to the 2.2.9 release on a discretionary basis. Nick
On Oct 10, 2015, at 2:46 PM, Nick Lowe <nick.lowe@gmail.com> wrote:
Would it really be that big a deal to release a 2.2.10 that solves this there?
The problem is we want to encourage people to use version 3. Having version 2 live forever as a zombie release (not quite dead, not quite alive) is a bad idea. Give that this is a major problem, I'm inclined to do another release to fix the OpenSSL idiocy. But it's only under protest, and only because OpenSSL broke a fundamental API used by FreeRADIUS. Alan DeKok.
Hi Alan, Le 11/10/2015 14:55, Alan DeKok a écrit :
Would it really be that big a deal to release a 2.2.10 that solves this there? The problem is we want to encourage people to use version 3. Having version 2 live forever as a zombie release (not quite dead, not quite alive) is a bad idea.
Give that this is a major problem, I'm inclined to do another release to fix the OpenSSL idiocy. But it's only under protest, and only because OpenSSL broke a fundamental API used by FreeRADIUS.
Would you mind to wait until Ubuntu 16.04 release (LTS) before dropping support for 2.2.x ? I am using at this time Ubuntu 14.04, and am hoping to upgrade to FR 3.x when Ubuntu 16.04 is released (with the hope that FR 3.x will be included in standard ubunytu repository then). Thanks, Alain
Am 11.10.2015 um 20:34 schrieb Alain Péan:
Hi Alan,
Would you mind to wait until Ubuntu 16.04 release (LTS) before dropping support for 2.2.x ? I am using at this time Ubuntu 14.04, and am hoping to upgrade to FR 3.x when Ubuntu 16.04 is released (with the hope that FR 3.x will be included in standard ubunytu repository then).
The problem is that Ubuntu is using Debian's FreeRADIUS package (unless they decide to ship their own package for 16.04) and that is pretty much stuck at 2.2.x Though there was some activity at Debian, thanks to the people who have worked on improving the debian packaging files inside the Debian source files. -- Mathieu
On Oct 11, 2015, at 2:54 PM, Mathieu Simon (Lists) <matsimon.lists@simweb.ch> wrote:
The problem is that Ubuntu is using Debian's FreeRADIUS package (unless they decide to ship their own package for 16.04) and that is pretty much stuck at 2.2.x
That's something Debian needs to fix, unfortunately. The previous debian maintainer disappeared, and no one else has stepped up to maintain the package. But Debian has 2-3 *other* RADIUS servers, each of which is 10+ years old... and hasn't been *maintained* for 10+ years. Why? Just... why?
Though there was some activity at Debian, thanks to the people who have worked on improving the debian packaging files inside the Debian source files.
We still need a Debian maintainer for FreeRADIUS. And we need to convince Debian to stop shipping crappy software that is 10+ years old, when better software exists. Alan DeKok.
On Oct 11, 2015, at 2:34 PM, Alain Péan <alain.pean@lpn.cnrs.fr> wrote:
Would you mind to wait until Ubuntu 16.04 release (LTS) before dropping support for 2.2.x ?
Debian / Ubuntu are free to package software which is a decade old an unmaintained. They do it with multiple other RADIUS servers. Why be different for FreeRADIUS?
I am using at this time Ubuntu 14.04, and am hoping to upgrade to FR 3.x when Ubuntu 16.04 is released (with the hope that FR 3.x will be included in standard ubunytu repository then).
Perhaps hell will freeze over, and Debian will upgrade to a recent version of the server. Alan DeKok.
Le 11/10/2015 23:27, Alan DeKok a écrit :
Would you mind to wait until Ubuntu 16.04 release (LTS) before dropping support for 2.2.x ? Debian / Ubuntu are free to package software which is a decade old an unmaintained. They do it with multiple other RADIUS servers. Why be different for FreeRADIUS?
I am using at this time Ubuntu 14.04, and am hoping to upgrade to FR 3.x when Ubuntu 16.04 is released (with the hope that FR 3.x will be included in standard ubunytu repository then). Perhaps hell will freeze over, and Debian will upgrade to a recent version of the server.
I have also the option to switch to CentOS (RHEL) 7. It seems that freeradius is version 3.0.4 there : [root@centos-test ~]# yum info freeradius .... Nom : freeradius Architecture : x86_64 Version : 3.0.4 Révision : 6.el7 In general, bug corrections are backported to previous (stable in RHEL) versions. I don't know if it is the case there ? And openssl is version 1.0.1e : ... Nom : openssl Architecture : x86_64 Date : 1 Version : 1.0.1e Révision : 42.el7.9 Alain -- Administrateur Système/Réseau Laboratoire de Photonique et Nanostructures (LPN/CNRS - UPR20) Centre de Recherche Alcatel Data IV - Marcoussis route de Nozay - 91460 Marcoussis Tel : 01-69-63-61-34
On Oct 12, 2015, at 7:50 AM, Alain Péan <alain.pean@lpn.cnrs.fr> wrote:
I have also the option to switch to CentOS (RHEL) 7. It seems that freeradius is version 3.0.4 there :
Yeah... RedHat upgrades the server... sometimes.
In general, bug corrections are backported to previous (stable in RHEL) versions. I don't know if it is the case there ?
Ask RedHat. If you're running RHEL, you have a support contract with them. If they can't answer your questions (and if they don't bother upgrading to fix bugs), I suggest getting support from someone else. Alan DeKok.
Red Hat have an update available for their affected 6.7 release: https://rhn.redhat.com/errata/RHBA-2015-1829.html If you're running CentOS 6.7, you're likely out of luck until 6.8 becomes available: https://bugs.centos.org/view.php?id=9295 (That bug report has seemingly been ignored.) As this is making a lot of noise with Android 6.0 (Marshmallow), please consider releasing a 3.x and discretionary, under protest 2.x release sooner rather than later to close the loop of known possible issues: https://code.google.com/p/android/issues/detail?id=188867 https://code.google.com/p/android-developer-preview/issues/detail?id=2136 I agree that there aren't likely to be many sites using OpenSSL 1.0.2 at this point in time, worth nipping in the bud though.
Hi Nick, It seems to me that both rhel 6.7 and centos 6.7 have same last patched freeradius version: https://rhn.redhat.com/errata/RHBA-2015-1829.html https://lists.centos.org/pipermail/centos-announce/2015-September/021404.htm... freeradius.x86_64 2.2.6-6.el6_7 as well as same openssl openssl.x86_64 1.0.1e-42.el6 Regards, Paolo.
On 13 ott 2015, at 10:20, Nick Lowe <nick.lowe@gmail.com> wrote:
Red Hat have an update available for their affected 6.7 release:
https://rhn.redhat.com/errata/RHBA-2015-1829.html
If you're running CentOS 6.7, you're likely out of luck until 6.8 becomes available:
https://bugs.centos.org/view.php?id=9295
(That bug report has seemingly been ignored.)
As this is making a lot of noise with Android 6.0 (Marshmallow), please consider releasing a 3.x and discretionary, under protest 2.x release sooner rather than later to close the loop of known possible issues:
https://code.google.com/p/android/issues/detail?id=188867
https://code.google.com/p/android-developer-preview/issues/detail?id=2136
I agree that there aren't likely to be many sites using OpenSSL 1.0.2 at this point in time, worth nipping in the bud though. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
------------------------------------------------------------------------------------------------ Paolo Barbato Consorzio RFX corso Stati Uniti,4 35127 Padova - Italy Network Administrator phone: +39 049 8295097 fax: +39 049 8700718 ------------------------------------------------------------------------------------------------
On Tue, Oct 13, 2015 at 3:20 PM, Nick Lowe <nick.lowe@gmail.com> wrote:
Red Hat have an update available for their affected 6.7 release:
https://rhn.redhat.com/errata/RHBA-2015-1829.html
If you're running CentOS 6.7, you're likely out of luck until 6.8 becomes available:
https://bugs.centos.org/view.php?id=9295
(That bug report has seemingly been ignored.)
As this is making a lot of noise with Android 6.0 (Marshmallow), please consider releasing a 3.x and discretionary, under protest 2.x release sooner rather than later to close the loop of known possible issues:
https://code.google.com/p/android/issues/detail?id=188867
https://code.google.com/p/android-developer-preview/issues/detail?id=2136
I agree that there aren't likely to be many sites using OpenSSL 1.0.2 at this point in time, worth nipping in the bud though.
@Nick: aren't the links you posted a different issues from Alan's original post? Looking at the dates, as well as Arran's comment on https://code.google.com/p/android/issues/detail?id=188867#c63, the bug affecting clients using TLS-1.2 (including android Marshmallow) is fixed in 2.2.9 and 3.0.10. This should also the one addressed with RH's errata. The warning Allan posted was about servers that use openssl-1.0.2, like debian testing and ubuntu wily (which should be released this month). The fix is present in git, and will be present in the next 3.0.11 (whenever that is), but there probably won't be any 2.2.10 due to EOL policy. In which case the "fix" that admins can use is to ensure: - NOT use openssl-1.0.2. Not an issue if they already stick to LTS release anyway. OR - Build their own FR version from git, and later upgrade to 3.0.11 when that is released. I agree with Arran here. Admins wishing to use openssl-1.0.2 on their server should use the git version or 3.0.11+. -- Fajar
It's all mixed in together as they're all service affecting issues. Read down that Google thread, Arran commented about this very issue. Nick
On Tue, Oct 13, 2015 at 3:49 PM, Nick Lowe <nick.lowe@gmail.com> wrote:
It's all mixed in together as they're all service affecting issues.
Read down that Google thread, Arran commented about this very issue.
I did. The last of his comment (which I linked above) was "FreeRADIUS 2.2.9 and FreeRADIUS 3.0.10 contain fixes for the above issues" What I DON'T see is how it is related to Alan's warning. Of course it might be because I missed something. -- Fajar
It's related to the Google thread because there may be people affected there who have been bitten by THIS issue as the backend is using OpenSSL 1.0.2, not the others. We just don't know.
Sorry, I realise I've framed my email really poorly... I should be more careful. To clarify: The update that Red Hat and CentOS has put out solve connectivity issues -while- they're using OpenSSL 1.0.1. A 3.0.11 and potential 2.0.10 update would be needed for people who end up using OpenSSL 1.0.2. Nick
Hi Fajar, Le 13/10/2015 10:47, Fajar A. Nugraha a écrit :
The warning Allan posted was about servers that use openssl-1.0.2, like debian testing and ubuntu wily (which should be released this month). The fix is present in git, and will be present in the next 3.0.11 (whenever that is), but there probably won't be any 2.2.10 due to EOL policy. In which case the "fix" that admins can use is to ensure: - NOT use openssl-1.0.2. Not an issue if they already stick to LTS release anyway. OR - Build their own FR version from git, and later upgrade to 3.0.11 when that is released.
I agree with Arran here. Admins wishing to use openssl-1.0.2 on their server should use the git version or 3.0.11+.
Thanks for your detailed explanations. In fact, I don't want to bother with recompiling by hand any application that I use, and follow all of bugs and securty annoucements, and if one is compatible with the other. It's why I prefer to let this job to distro manager, and update my distro on a regular basis. Alain -- Administrateur Système/Réseau Laboratoire de Photonique et Nanostructures (LPN/CNRS - UPR20) Centre de Recherche Alcatel Data IV - Marcoussis route de Nozay - 91460 Marcoussis Tel : 01-69-63-61-34
Arran Cudbard-Bell wrote:
On 10 Oct 2015, at 08:57, Alan DeKok <aland@deployingradius.com> wrote:
OpenSSL 1.0.2 changes the way it interacts with FreeRADIUS. None of this is documented by OpenSSL. The result is that instead of successful authentication, you get:
(6) eap_ttls: ERROR: Invalid ACK received: 256 (6) eap_ttls: ERROR: [eaptls verify] = invalid (6) eap_ttls: ERROR: [eaptls process] = invalid
The only solution is to apply the patch in commit b7b5493c61. It doesn't fix the underlying OpenSSL problem, but it makes FreeRADIUS ignore the broken API calls.
This problem is serious enough that we may have to issue 3.0.11, and possibly 2.2.10, also.
Have to draw a line on 2.2.x this uncertainty undermines people making the case to move to v3.0.x. 1.0.2 is not included by default in any stable releases of FreeBSD, Ubuntu/Debian, Redhat/Centos, OSX.
We experienced it because homebrew has moved to OpenSSL 1.0.2.
Which exact version of OpenSSL 1.0.2? I'm asking because I'm running FreeRADIUS 3.0.10 (formerly 3.0.9) on openSUSE Tumbleweed (x86_64 and armv6l) with package openssl-1.0.2d-1.1 using EAP-TTLS/PAP without issue. Maybe you're hitting the HMAC ABI incompatibility? It was fixed in 1.0.2c: https://www.openssl.org/news/changelog.html#x2 Ciao, Michael.
On 10 Oct 2015, at 14:40, Michael Ströder <michael@stroeder.com> wrote:
Arran Cudbard-Bell wrote:
On 10 Oct 2015, at 08:57, Alan DeKok <aland@deployingradius.com> wrote:
OpenSSL 1.0.2 changes the way it interacts with FreeRADIUS. None of this is documented by OpenSSL. The result is that instead of successful authentication, you get:
(6) eap_ttls: ERROR: Invalid ACK received: 256 (6) eap_ttls: ERROR: [eaptls verify] = invalid (6) eap_ttls: ERROR: [eaptls process] = invalid
The only solution is to apply the patch in commit b7b5493c61. It doesn't fix the underlying OpenSSL problem, but it makes FreeRADIUS ignore the broken API calls.
This problem is serious enough that we may have to issue 3.0.11, and possibly 2.2.10, also.
Have to draw a line on 2.2.x this uncertainty undermines people making the case to move to v3.0.x. 1.0.2 is not included by default in any stable releases of FreeBSD, Ubuntu/Debian, Redhat/Centos, OSX.
We experienced it because homebrew has moved to OpenSSL 1.0.2.
Which exact version of OpenSSL 1.0.2?
I'm asking because I'm running FreeRADIUS 3.0.10 (formerly 3.0.9) on openSUSE Tumbleweed (x86_64 and armv6l) with package openssl-1.0.2d-1.1 using EAP-TTLS/PAP without issue.
TTLS PAP works because we don't ACK the last TLS fragment sent by the supplicant, we just fudge an EAP-Success. Or at least that's what it appears is happening. I'm not sure if this is the correct behaviour. It's on a todo list of EAP-TLS things to investigate, along with sending the correct TLS Alert record on authentication failure. If you try TTLS-MSCHAPv2 you'll see a failure. If you wanted to do something useful, you could try the different TLS based EAP methods and figure out which ones are definitely broken. The code path is common for all EAP-TLS based methods, so the assumption is that many will be broken by this.
Maybe you're hitting the HMAC ABI incompatibility? It was fixed in 1.0.2c: https://www.openssl.org/news/changelog.html#x2
This was with 1.0.2d, so no. -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
On Oct 10, 2015, at 2:40 PM, Michael Ströder <michael@stroeder.com> wrote:
Which exact version of OpenSSL 1.0.2?
I'm not sure... I haven't seen people posting which version of OpenSSL they're using.
I'm asking because I'm running FreeRADIUS 3.0.10 (formerly 3.0.9) on openSUSE Tumbleweed (x86_64 and armv6l) with package openssl-1.0.2d-1.1 using EAP-TTLS/PAP without issue.
Maybe you're hitting the HMAC ABI incompatibility? It was fixed in 1.0.2c: https://www.openssl.org/news/changelog.html#x2
I don't think so. But who knows... If OpenSSL 1.0.2d fixes the problem, then we can leave 2.2.9 alone, and not do 2.2.10. It's almost like the OpenSSL people just write code, and release it without doing any tests. Alan DeKok.
Le 11 oct. 2015 à 15:01, Alan DeKok a écrit :
On Oct 10, 2015, at 2:40 PM, Michael Ströder wrote:
Which exact version of OpenSSL 1.0.2?
I'm not sure... I haven’t seen people posting which version of OpenSSL they're using.
Hello Alan, I got the problem with FR 3.0.10/OpenSSL 1.0.2d, before Arran told me to pull the v3.0.x HEAD. HTH, Axel
On Oct 11, 2015, at 11:03 AM, Axel Luttgens <axel.luttgens@skynet.be> wrote:
I got the problem with FR 3.0.10/OpenSSL 1.0.2d, before Arran told me to pull the v3.0.x HEAD.
Well, that's that, then. I've tried wandering through the OpenSSL code to see when / why / where the change was made. The code is not just complicated, it's needlessly complicated. Endless layers of obfuscation. The FreeRADIUS code looks beautiful in comparison. :( Alan DeKok.
On Sun, Oct 11, 2015 at 7:03 PM, Alan DeKok <aland@deployingradius.com> wrote:
I've tried wandering through the OpenSSL code to see when / why / where the change was made. The code is not just complicated, it's needlessly complicated. Endless layers of obfuscation. The FreeRADIUS code looks beautiful in comparison. :(
Don't ask me how to find this, but this commit is the most likely reason: https://github.com/openssl/openssl/commit/bc200e691cd68870c2062d3c1e74280a59... ('SSL/TLS record tracing code (backport from HEAD). ') (or commit 36b5bb6f2f944d6fb9a458da76ffdfa9154c03c2 in the openssl master branch) The msg_callback() calls with hardcoded version 0 for SSL3_RT_HEADER in ssl/s3_pkt.c is the extension that you are likely seeing here. The other lovely ones that you have unlikely seen yet are the msg_callback() calls with write_p == 2 (instead of the documented 0/1). Though, I'd assume and hope that these write_p == 2 cases do not show up without a special OpenSSL build (OPENSSL_SSL_TRACE_CRYPTO enabled with enable-ssl-trace). Anyway, you may want to be ready for them just in case and return from the callback function is write_p == 2 is seen (or maybe more robustly: if write_p is not 0 or 1). - Jouni
On Oct 11, 2015, at 3:38 PM, Jouni Malinen <jkmalinen@gmail.com> wrote:
Don't ask me how to find this, but this commit is the most likely reason: https://github.com/openssl/openssl/commit/bc200e691cd68870c2062d3c1e74280a59... ('SSL/TLS record tracing code (backport from HEAD). ')
Yeah... that's probably it.
The msg_callback() calls with hardcoded version 0 for SSL3_RT_HEADER in ssl/s3_pkt.c is the extension that you are likely seeing here. The other lovely ones that you have unlikely seen yet are the msg_callback() calls with write_p == 2 (instead of the documented 0/1).
<sigh> They added tracing mechanisms... by changing their public API. That's *terrible*. It's like they have no comprehension that anyone *uses* their software.
Though, I'd assume and hope that these write_p == 2 cases do not show up without a special OpenSSL build (OPENSSL_SSL_TRACE_CRYPTO enabled with enable-ssl-trace). Anyway, you may want to be ready for them just in case and return from the callback function is write_p == 2 is seen (or maybe more robustly: if write_p is not 0 or 1).
Yes, I'll push a fix for that, too. Ugh. Alan DeKok.
On 11 Oct 2015, at 09:01, Alan DeKok <aland@deployingradius.com> wrote:
On Oct 10, 2015, at 2:40 PM, Michael Ströder <michael@stroeder.com> wrote:
Which exact version of OpenSSL 1.0.2?
I'm not sure... I haven't seen people posting which version of OpenSSL they're using.
As I said previously, tests were with OpenSSL 1.0.2d. That's likely to be the ones everyone is using, because, as I said previously, the main exposure to OpenSSL 1.0.2 is through homebrew, and people building it themselves.
I'm asking because I'm running FreeRADIUS 3.0.10 (formerly 3.0.9) on openSUSE Tumbleweed (x86_64 and armv6l) with package openssl-1.0.2d-1.1 using EAP-TTLS/PAP without issue.
Maybe you're hitting the HMAC ABI incompatibility? It was fixed in 1.0.2c: https://www.openssl.org/news/changelog.html#x2
I don't think so. But who knows...
As I said previously, OpenSSL 1.0.2d does not fix the problem. You might want to read what I wrote before contributing to the dialogue? -Arran Arran Cudbard-Bell <a.cudbardb@freeradius.org> FreeRADIUS development team FD31 3077 42EC 7FCD 32FE 5EE2 56CF 27F9 30A8 CAA2
participants (10)
-
Alain Péan -
Alan DeKok -
Arran Cudbard-Bell -
Axel Luttgens -
Fajar A. Nugraha -
Jouni Malinen -
Mathieu Simon (Lists) -
Michael Ströder -
Nick Lowe -
Paolo Barbato