Free Radius problem with sending large certificate chains, using EAP-TLS
Hi, We are running freeradius, version 1.1.7, on Fedora. We are testing WPA2/EAP-TLS authentication, with large certificate chains (just under 64K in PEM format). Some individual cert sizes in the chain approach 10K in DER format. If the chain is small enough to fit in a single TLS message, authentication works fine. But is the chain is greater than 16,384 bytes, eap-tls fails. Looking at a packet trace, freeradius does not send a message above 16.438 bytes. Instead of breaking it up into different records, it attempts to send it in one TLS record, with fragments that are too large. Wireshark trace of Radius traffic shows the following: TLS Server Hello, Certificate, Certificate Request {Malformed Packet} Frame: 1239, payload 16384-17407 (1024 bytes) [Fragment overlap: True] {Fragment too long: True] This frame pattern repeats with later fragments ... Per RFC's 2716 and 5216, it seems freeradius should brake a single TLS message (larger than one 16,384 byte record can support) into multiple TLS records. We could not find anything on this problem in the FAQ or user lists. Can someone tell us what we are doing wrong, or is this a bug which hasn't been reports, since this large cert chain is rare? We will update to the latest freeradius release. Thanks, Brian Smith RFC's 2716 and 5216 - EAP-TLS Authentication Protocol Section 2.1.5. Fragmentation A single TLS record may be up to 16384 octets in length, but a TLS message may span multiple TLS records, and a TLS certificate message may in principle be as long as 16 MB. The group of EAP-TLS messages sent in a single round may thus be larger than the MTU size or the maximum Remote Authentication Dail-In User Service (RADIUS) packet size of 4096 octets. As a result, an EAP-TLS implementation MUST provide its own support for fragmentation and reassembly. However, in order to ensure interoperability with existing implementations, TLS handshake messages SHOULD NOT be fragmented into multiple TLS records if they fit within a single TLS record. In order to protect against reassembly lockup and denial-of-service attacks, it may be desirable for an implementation to set a maximum size for one such group of TLS messages. Since a single certificate is rarely longer than a few thousand octets, and no other field is likely to be anywhere near as long, a reasonable choice of maximum acceptable message length might be 64 KB.
Hi, * Smith, Brian (ESEA IS&A) <brian.smith@honeywell.com> [Fri, 20 Feb 2009 11:15:01 -0700]:
We are running freeradius, version 1.1.7, on Fedora. We are testing WPA2/EAP-TLS authentication, with large certificate chains (just under 64K in PEM format). Some individual cert sizes in the chain approach 10K in DER format. If the chain is small enough to fit in a single TLS message, authentication works fine. But is the chain is greater than 16,384 bytes, eap-tls fails. Looking at a packet trace, freeradius does not send a message above 16.438 bytes. Instead of breaking it up into different records, it attempts to send it in one TLS record, with fragments that are too large.
Overlooking the possible FreeRADIUS bug, I'm pretty sure I remember chatting to Tom Rixom (of SecureW2 fame) and he was grumbling that some supplicants[1] would not accept standalone certificates above 4kB in size (it was something like that); as that's all the memory set aside in a buffer internally. You might find there are supplicants out there that are going to sulk when forced to accept such whopping payloads :) Cheers [1] in this case the grumble was pointed at Microsoft Windows CE -- Alexander Clouter .sigmonster says: Encyclopedia for sale by father. Son knows everything.
Hi Alexander, Thanks for your reply and yes, I expect you are right about some clients not supporting large certificates. Thanks for your help! Regards, Brian Smith Ph. 602-436-6691 Honeywell -----Original Message----- From: freeradius-users-bounces+brian.smith=honeywell.com@lists.freeradius.org [mailto:freeradius-users-bounces+brian.smith=honeywell.com@lists.freerad ius.org] On Behalf Of Alexander Clouter Sent: Friday, February 20, 2009 12:52 PM To: freeradius-users@lists.freeradius.org Subject: Re: Free Radius problem with sending large certificate chains,using EAP-TLS Hi, * Smith, Brian (ESEA IS&A) <brian.smith@honeywell.com> [Fri, 20 Feb 2009 11:15:01 -0700]:
We are running freeradius, version 1.1.7, on Fedora. We are testing WPA2/EAP-TLS authentication, with large certificate chains (just under 64K in PEM format). Some individual cert sizes in the chain approach 10K in DER format. If the chain is small enough to fit in a single
TLS
message, authentication works fine. But is the chain is greater than 16,384 bytes, eap-tls fails. Looking at a packet trace, freeradius does not send a message above 16.438 bytes. Instead of breaking it up into different records, it attempts to send it in one TLS record, with fragments that are too large.
Overlooking the possible FreeRADIUS bug, I'm pretty sure I remember chatting to Tom Rixom (of SecureW2 fame) and he was grumbling that some supplicants[1] would not accept standalone certificates above 4kB in size (it was something like that); as that's all the memory set aside in a buffer internally. You might find there are supplicants out there that are going to sulk when forced to accept such whopping payloads :) Cheers [1] in this case the grumble was pointed at Microsoft Windows CE -- Alexander Clouter .sigmonster says: Encyclopedia for sale by father. Son knows everything. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Smith, Brian (ESEA IS&A) wrote:
We are running freeradius, version 1.1.7, on Fedora. We are testing WPA2/EAP-TLS authentication, with large certificate chains (just under 64K in PEM format).
Ouch... that's big.
Some individual cert sizes in the chain approach 10K in DER format. If the chain is small enough to fit in a single TLS message, authentication works fine. But is the chain is greater than 16,384 bytes, eap-tls fails. Looking at a packet trace, freeradius does not send a message above 16.438 bytes. Instead of breaking it up into different records, it attempts to send it in one TLS record, with fragments that are too large.
Hmm... OK.
Per RFC’s 2716 and 5216, it seems freeradius should brake a single TLS message (larger than one 16,384 byte record can support) into multiple TLS records.
It's supposed to. It doesn't, however.
We could not find anything on this problem in the FAQ or user lists. Can someone tell us what we are doing wrong, or is this a bug which hasn’t been reports, since this large cert chain is rare? We will update to the latest freeradius release.
I think that this is the first time someone ran into this problem. The other issue is that 64k certificate chains may cause other problems. Both supplicants && access points have EAP packet counters. Aftert 30-50 packets in one EAP session, they simply drop the session as "taking too long". i.e. You might get FreeRADIUS to support 64K certificate chains, and then discover that none of the access points or PC's can support it. I don't think it's too hard to fix this, it just requires some additional code to deal with messages greater than 16K. Right now, all of the internal code assumes that the maximum message size is one TLS fragment. Alan DeKok.
Hi Alan, Thanks for the great reply. It makes perfect sense to me. Just be clear, FreeRadius will support a certificate/chain length up to the TLS record limit of 16384 bytes (minus some overhead). And, you don't know of anyone that has every tried to test beyond this, which tells me in practice, it's not done.... Also, you point out that very likely AP's and STA' might not support multiple records, though the RFC says they should. Also telling me, this is not normally done. Two quick questions for you. - What do you think the market penetration of FreeRadius (or commercial clones) to authenticate wireless WPA2 clients is, verses commercial products? - Do you know of any other Radius Server that does support multiple TLS records for a single message? - What is the largest certificate chain you have seen used with FreeRadius? Thanks Again! Brian Smith Ph. 602-436-6691 Honeywell -----Original Message----- From: freeradius-users-bounces+brian.smith=honeywell.com@lists.freeradius.org [mailto:freeradius-users-bounces+brian.smith=honeywell.com@lists.freerad ius.org] On Behalf Of Alan DeKok Sent: Saturday, February 21, 2009 1:37 AM To: FreeRadius users mailing list Subject: Re: Free Radius problem with sending large certificate chains, usingEAP-TLS Smith, Brian (ESEA IS&A) wrote:
We are running freeradius, version 1.1.7, on Fedora. We are testing WPA2/EAP-TLS authentication, with large certificate chains (just under 64K in PEM format).
Ouch... that's big.
Some individual cert sizes in the chain approach 10K in DER format. If the chain is small enough to fit in a single TLS message, authentication works fine. But is the chain is greater than 16,384 bytes, eap-tls fails. Looking at a packet trace, freeradius does not send a message above 16.438 bytes. Instead of breaking it up into different records, it attempts to send it in one TLS record, with fragments that are too large.
Hmm... OK.
Per RFC's 2716 and 5216, it seems freeradius should brake a single TLS message (larger than one 16,384 byte record can support) into multiple TLS records.
It's supposed to. It doesn't, however.
We could not find anything on this problem in the FAQ or user lists. Can someone tell us what we are doing wrong, or is this a bug which hasn't been reports, since this large cert chain is rare? We will update to the latest freeradius release.
I think that this is the first time someone ran into this problem. The other issue is that 64k certificate chains may cause other problems. Both supplicants && access points have EAP packet counters. Aftert 30-50 packets in one EAP session, they simply drop the session as "taking too long". i.e. You might get FreeRADIUS to support 64K certificate chains, and then discover that none of the access points or PC's can support it. I don't think it's too hard to fix this, it just requires some additional code to deal with messages greater than 16K. Right now, all of the internal code assumes that the maximum message size is one TLS fragment. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Smith, Brian (ESEA IS&A) wrote:
Hi Alan, Thanks for the great reply. It makes perfect sense to me. Just be clear, FreeRadius will support a certificate/chain length up to the TLS record limit of 16384 bytes (minus some overhead). And, you don't know of anyone that has every tried to test beyond this, which tells me in practice, it's not done....
Yes.
Also, you point out that very likely AP's and STA' might not support multiple records, though the RFC says they should. Also telling me, this is not normally done.
No... they *do* support multiple round trips. But they have an upper limit on "too many" round trips. For example, WPA supplicant (the most widely used one) has a default limit of 50. This means it's *highly* unlikely that it will work with 64K certificate chains.
Two quick questions for you.
- What do you think the market penetration of FreeRadius (or commercial clones) to authenticate wireless WPA2 clients is, verses commercial products?
It's the most widely used RADIUS server on the planet. Most large telcos on Europe are either using it, or switching to it.
- Do you know of any other Radius Server that does support multiple TLS records for a single message?
No idea, sorry. And if you're thinking of buying one that does, I can pretty much guarantee you it'll be cheaper and faster to fix FreeRADIUS.
- What is the largest certificate chain you have seen used with FreeRadius?
I don't know. People don't usually report that kind of statistics. Alan DeKok.
On Tue, Feb 24, 2009 at 9:20 AM, Alan DeKok <aland@deployingradius.com> wrote:
No... they *do* support multiple round trips. But they have an upper limit on "too many" round trips. For example, WPA supplicant (the most widely used one) has a default limit of 50. This means it's *highly* unlikely that it will work with 64K certificate chains.
The main (well, more or less, the only) reason for that limit on number of round trips is to work around issues where the EAP peer and server ended up in an infinite loop ACKing their messages. I would prefer to change that to be based on whether any real progress has been made during the last round trip or two, i.e., to remove the hard limit and allow as many round trips as it takes to get through the authentication (or whatever else one adds into EAP, e.g., TNC). It would be nicer to support the whatever maximum length is described for EAP-TLS or TNC, but not at the cost of bringing back interop issues that may result in infinite authentication loops. Anyway, the only case I remember of someone discussing the round trip limit as a too strict limit was for TNC, not for certificate sizes. If someone is really using huge certificates (or well, long enough chain to make the total size of the TLS message long) in real world, I would like to make sure it can be done. I just haven't come up with a real use case so far. - Jouni
Jouni Malinen wrote:
The main (well, more or less, the only) reason for that limit on number of round trips is to work around issues where the EAP peer and server ended up in an infinite loop ACKing their messages. I would prefer to change that to be based on whether any real progress has been made during the last round trip or two, i.e., to remove the hard limit and allow as many round trips as it takes to get through the authentication (or whatever else one adds into EAP, e.g., TNC). It would be nicer to support the whatever maximum length is described for EAP-TLS or TNC, but not at the cost of bringing back interop issues that may result in infinite authentication loops.
Defining "progress" per EAP type may be difficult.
Anyway, the only case I remember of someone discussing the round trip limit as a too strict limit was for TNC, not for certificate sizes. If someone is really using huge certificates (or well, long enough chain to make the total size of the TLS message long) in real world, I would like to make sure it can be done. I just haven't come up with a real use case so far.
Yes, I recall those discussions related to TNC and NEA a while ago.
From what I see in the standards now, there is no reason for *bulk* transfer of data over EAP. The TNC standards require pretty small data transfers.
And even if wpa_supplicant is changed, it will be difficult to change the millions of AP's out there. Alan DeKok.
On Tue, Feb 24, 2009 at 10:36 AM, Alan DeKok <aland@deployingradius.com> wrote:
Defining "progress" per EAP type may be difficult.
Indeed and that is why the hardcoded limit of round trips ended up being there in the first place.. ;-) Anyway, the most common issue case I've seen is where EAP server and peer end up sending TLS ACK messages in a loop and that would be easy to catch. Anyway, if this were to change at some point, I would assume there ends up being the default round trip limit and then some EAP type specific improvements to optimize that for the methods that need support for longer exchanges.
Yes, I recall those discussions related to TNC and NEA a while ago. From what I see in the standards now, there is no reason for *bulk* transfer of data over EAP. The TNC standards require pretty small data transfers.
Sure, no bulk data should be supported, but even TNC requires IF-TNCCS messages of up to 100 kilobytes in length which goes beyond the 50*1400 bytes or so (depending on max frame length) limit that is currently hardcoded in wpa_supplicant.
And even if wpa_supplicant is changed, it will be difficult to change the millions of AP's out there.
Well, I would hope that most APs don't have such limits on the EAP/EAPOL; this is supposed to be transparent data they are just proxying through.. Anyway, yes, if they do have a hard limit, there is not much that can be done to make this work with such a NAS. - Jouni
Hi Jouni, Thanks for your reply. I understand your concern on wasting time when in a failure condition. I agree it would be ideal for the code to continue transfers, based on progress. We will try to validate the use case before taking this further. Regards, Brian Smith Ph. 602-436-6691 Honeywell -----Original Message----- From: freeradius-users-bounces+brian.smith=honeywell.com@lists.freeradius.org [mailto:freeradius-users-bounces+brian.smith=honeywell.com@lists.freeradius.org] On Behalf Of Jouni Malinen Sent: Tuesday, February 24, 2009 1:20 AM To: FreeRadius users mailing list Subject: Re: Free Radius problem with sending large certificate chains, usingEAP-TLS On Tue, Feb 24, 2009 at 9:20 AM, Alan DeKok <aland@deployingradius.com> wrote:
No... they *do* support multiple round trips. But they have an upper limit on "too many" round trips. For example, WPA supplicant (the most widely used one) has a default limit of 50. This means it's *highly* unlikely that it will work with 64K certificate chains.
The main (well, more or less, the only) reason for that limit on number of round trips is to work around issues where the EAP peer and server ended up in an infinite loop ACKing their messages. I would prefer to change that to be based on whether any real progress has been made during the last round trip or two, i.e., to remove the hard limit and allow as many round trips as it takes to get through the authentication (or whatever else one adds into EAP, e.g., TNC). It would be nicer to support the whatever maximum length is described for EAP-TLS or TNC, but not at the cost of bringing back interop issues that may result in infinite authentication loops. Anyway, the only case I remember of someone discussing the round trip limit as a too strict limit was for TNC, not for certificate sizes. If someone is really using huge certificates (or well, long enough chain to make the total size of the TLS message long) in real world, I would like to make sure it can be done. I just haven't come up with a real use case so far. - Jouni - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Hi Alan, Again, thanks for your great reply. If we wanted to pursue this capability, what would be the process to get FreeRadius to support large chains? Regards, Brian Smith Ph. 602-436-6691 Honeywell -----Original Message----- From: freeradius-users-bounces+brian.smith=honeywell.com@lists.freeradius.org [mailto:freeradius-users-bounces+brian.smith=honeywell.com@lists.freerad ius.org] On Behalf Of Alan DeKok Sent: Tuesday, February 24, 2009 12:20 AM To: FreeRadius users mailing list Subject: Re: Free Radius problem with sending large certificate chains,usingEAP-TLS Smith, Brian (ESEA IS&A) wrote:
Hi Alan, Thanks for the great reply. It makes perfect sense to me. Just be clear, FreeRadius will support a certificate/chain length up to the TLS record limit of 16384 bytes (minus some overhead). And, you don't know of anyone that has every tried to test beyond this, which tells me in practice, it's not done....
Yes.
Also, you point out that very likely AP's and STA' might not support multiple records, though the RFC says they should. Also telling me, this is not normally done.
No... they *do* support multiple round trips. But they have an upper limit on "too many" round trips. For example, WPA supplicant (the most widely used one) has a default limit of 50. This means it's *highly* unlikely that it will work with 64K certificate chains.
Two quick questions for you.
- What do you think the market penetration of FreeRadius (or commercial clones) to authenticate wireless WPA2 clients is, verses commercial products?
It's the most widely used RADIUS server on the planet. Most large telcos on Europe are either using it, or switching to it.
- Do you know of any other Radius Server that does support multiple TLS records for a single message?
No idea, sorry. And if you're thinking of buying one that does, I can pretty much guarantee you it'll be cheaper and faster to fix FreeRADIUS.
- What is the largest certificate chain you have seen used with FreeRadius?
I don't know. People don't usually report that kind of statistics. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
participants (4)
-
Alan DeKok -
Alexander Clouter -
Jouni Malinen -
Smith, Brian (ESEA IS&A)