FR 2.1.8 Issue - Unjustified(?) Access-Rejects.
Hi, We migrated to 2.1.8 (from 2.1.7) last week while things were quiet, as the users have re-appeared after the holiday we've started to receive a few reports from users stating that they have been getting lots of prompts for credentials. Two users in particular, both Ubuntu users, have been having big problems. I had a look at the radpostauth table and sure enough at least 50% of their auth attempts have resulted in access-rejects. I rolled back to 2.1.7 this afternoon and the problem has gone away. Should anyone be interested I have attached an radmin output for one of the users, it shows some accepts and lots of rejects. Is this likely to be a configuration error (no changes were made to the 2.1.7 config), or a bug? Thanks, Jezz Palmer.
Palmer J.D.F. wrote:
We migrated to 2.1.8 (from 2.1.7) last week while things were quiet, as the users have re-appeared after the holiday we've started to receive a few reports from users stating that they have been getting lots of prompts for credentials.
The log says: ... WARNING: No information in cached session! This means that the session wasn't cached, and they are trying to resume a session that never was started. The change in 2.1.8 is there to work around a bug in OpenSSL. The only other alternative is that they *are* resuming a valid session, but (a) after the session has timed out, or (b) where no User-Name was cached from the inner tunnel session.
Is this likely to be a configuration error (no changes were made to the 2.1.7 config), or a bug?
Try increasing the size of the cache. Try ensuring that there is always a User-Name in the inner tunnel. This user name is cached, and is checked on session resumption. Alan DeKok.
Hi,
Is this likely to be a configuration error (no changes were made to the 2.1.7 config), or a bug?
Try increasing the size of the cache. Try ensuring that there is always a User-Name in the inner tunnel. This user name is cached, and is checked on session resumption.
How does this work together with anonymous outer ids? I.e. if outer User-Name = anon@foo.bar and the inner User-Name is stefan@foo.bar, then the cache contains a session for stefan@foo.bar On session resumption, there is no inner tunnel exchange, there's a packet User-Name = anon@foo.bar and an EAP-Message with SSL magic (but no inner User-Name)... So how does FreeRADIUS know what to look up in the cache? Or am I missing something here? Greetings, Stefan -- Stefan WINTER Ingenieur de Recherche Fondation RESTENA - Réseau Téléinformatique de l'Education Nationale et de la Recherche 6, rue Richard Coudenhove-Kalergi L-1359 Luxembourg Tel: +352 424409 1 Fax: +352 422473
Stefan Winter wrote:
How does this work together with anonymous outer ids? I.e. if outer User-Name = anon@foo.bar and the inner User-Name is stefan@foo.bar, then the cache contains a session for stefan@foo.bar
Yes.
On session resumption, there is no inner tunnel exchange, there's a packet User-Name = anon@foo.bar and an EAP-Message with SSL magic (but no inner User-Name)... So how does FreeRADIUS know what to look up in the cache? Or am I missing something here?
There's an SSL identifier associated with the session: supplicant: I have SSL id 0x282674736733673 server: OK, it's in my cache. (Modulo various crypto operations to keep it secure) The server uses the Id to find the cache entry, and the cached User-Name. Alan DeKok.
Stefan Winter <stefan.winter@restena.lu> wrote:
Is this likely to be a configuration error (no changes were made to the 2.1.7 config), or a bug?
Try increasing the size of the cache. Try ensuring that there is always a User-Name in the inner tunnel. This user name is cached, and is checked on session resumption.
How does this work together with anonymous outer ids? I.e. if outer User-Name = anon@foo.bar and the inner User-Name is stefan@foo.bar, then the cache contains a session for stefan@foo.bar
On session resumption, there is no inner tunnel exchange, there's a packet User-Name = anon@foo.bar and an EAP-Message with SSL magic (but no inner User-Name)... So how does FreeRADIUS know what to look up in the cache? Or am I missing something here?
You get the inner-tunnel to return in the reply packet the inner User-Name (you probably are doing this already to fixup your accounting packets properly) and it's that reply response which is cached by the session-resumption cache thingy mcwhatsit. Works rather nicely here. It's a minor ballache with load-balancers and overlapping 'eduroam' domains mind you...but that is a non-trivially[1] solved problem and something I can live with as it rarely crops up. Cheers [1] you need to share the SSL session cache between your different FreeRADIUS boxen, the support for that is not in OpenSSL yet if I remember correctly (or was it FreeRADIUS). This would be done with some file that could probably be NFS shared or something or other with locking safely enough -- Alexander Clouter .sigmonster says: How come only your friends step on your new white sneakers?
[1] you need to share the SSL session cache between your different FreeRADIUS boxen, the support for that is not in OpenSSL yet if I remember correctly (or was it FreeRADIUS). This would be done
Shared SSL session caches are definitely supported in OpenSSL, and have been for a while IIRC; see "distcache" for info. Whether it's compiled into your SSL library, and whether there are caveats that mean it won't work with FreeRadius... not sure. Apache lists (commented out) config items like: #SSLSessionCache dc:UNIX:/var/cache/mod_ssl/distcache ...in Fedora at least. Whilst testing the SoH/NAP stuff, I saw some oddities with SSL session resumption. I wasn't sure if it was something I did (i.e. broke inside the PEAP code) or not, but the server seemed to be allowing resumption even when it was disabled i.e. with the default FR config. Random info: PEAP/SoH in fact *does* send traffic inside the tunnel on session resumption - the spec has the SoH exchanged even when resumed, adding a round trip, but it doesn't re-run the inner mschap auth. Weird.
Shared SSL session caches are definitely supported in OpenSSL, and have been for a while IIRC; see "distcache" for info. Whether it's compiled
No wait, I'm talking crap. Distcache is a layer *on top of* OpenSSL. You have to write for the distcache API. I had assumed it was a plugin, but no - that would be too easy.
Random info: PEAP/SoH in fact *does* send traffic inside the tunnel on session resumption - the spec has the SoH exchanged even when resumed, adding a round trip, but it doesn't re-run the inner mschap auth. Weird.
The authentication state hasn't changed if the session can be re-established. The authorisation state however, may have, which is why the SoH is sent on every authentication attempt. If you didn't send the SoH on resumption, the neat 'spot check' and periodic re-checking that you can do using various triggers for re-authentication would go away. -Arran
Alan DeKok <aland@deployingradius.com> writes:
Palmer J.D.F. wrote:
We migrated to 2.1.8 (from 2.1.7) last week while things were quiet, as the users have re-appeared after the holiday we've started to receive a few reports from users stating that they have been getting lots of prompts for credentials.
The log says:
... WARNING: No information in cached session!
This means that the session wasn't cached, and they are trying to resume a session that never was started. The change in 2.1.8 is there to work around a bug in OpenSSL.
The only other alternative is that they *are* resuming a valid session, but (a) after the session has timed out, or (b) where no User-Name was cached from the inner tunnel session.
Don't know if it's relevant, but I briefly tried to enable caching on my home network after installing 2.1.8 and could not make it work. It wasn't important to me, so I just disabled it. Haven't reported it earlier as I suspected (and still does...) that I was doing something wrong. But here are the log messages anyway, in the event that they are symptoms of a real problem: I got this after a sucessful first authentication: Tue Jan 5 19:00:21 2010 : Info: [ttls] Got tunneled Access-Accept Tue Jan 5 19:00:21 2010 : Info: [ttls] Saving response in the cache Tue Jan 5 19:00:21 2010 : Info: [ttls] WARNING: No information to cache: session caching will be disabled for this session. Then, as I guess is expected based on the above, on reauth (please ignore the timestamps - these are not matched samples): Tue Jan 5 18:18:15 2010 : Info: [eap] Request found, released from the list Tue Jan 5 18:18:15 2010 : Info: [eap] EAP/ttls Tue Jan 5 18:18:15 2010 : Info: [eap] processing type ttls Tue Jan 5 18:18:15 2010 : Info: [ttls] Authenticate Tue Jan 5 18:18:15 2010 : Info: [ttls] processing EAP-TLS Tue Jan 5 18:18:15 2010 : Info: [ttls] eaptls_verify returned 7 Tue Jan 5 18:18:15 2010 : Info: [ttls] Done initial handshake Tue Jan 5 18:18:15 2010 : Info: [ttls] TLS_accept: SSLv3 read finished A Tue Jan 5 18:18:15 2010 : Info: [ttls] (other): SSL negotiation finished successfully Tue Jan 5 18:18:15 2010 : Info: [ttls] eaptls_process returned 3 Tue Jan 5 18:18:15 2010 : Info: [ttls] Skipping Phase2 due to session resumption Tue Jan 5 18:18:15 2010 : Info: [ttls] WARNING: No information in cached session! Tue Jan 5 18:18:15 2010 : Info: [eap] Freeing handler Tue Jan 5 18:18:15 2010 : Info: ++[eap] returns reject Bjørn
Thanks for the reply Alan.
This means that the session wasn't cached, and they are trying to resume a session that never was started. The change in 2.1.8 is there to work around a bug in OpenSSL.
Ok
The only other alternative is that they *are* resuming a valid session, but (a) after the session has timed out, or (b) where no User-Name was cached from the inner tunnel session.
b) is possible due to the point below about cache size.
Try increasing the size of the cache. Try ensuring that there is always a User-Name in the inner tunnel. This user name is cached, and is checked on session resumption.
I reinstated 2.1.8 this morning after having set the cache size to infinity (was the default 255) but the problem still exists. Caching is enabled in eap.conf, but does fastreauth need to be enabled in experimental.conf? It is currently disabled. Whether this has any bearing on it I'm not sure, but this seems to be affecting users that use wpa_supplicant more, though Windows users have also reported the problem. Thanks, Jezz.
Palmer J.D.F. wrote:
I reinstated 2.1.8 this morning after having set the cache size to infinity (was the default 255) but the problem still exists. Caching is enabled in eap.conf, but does fastreauth need to be enabled in experimental.conf? It is currently disabled.
You are not using experimental.conf, so changing values there won't affect anything.
Whether this has any bearing on it I'm not sure, but this seems to be affecting users that use wpa_supplicant more, though Windows users have also reported the problem.
The sessions *also* have a timeout. Read eap.conf. Alan DeKok.
Whether this has any bearing on it I'm not sure, but this seems to be affecting users that use wpa_supplicant more, though Windows users have also reported the problem.
The sessions *also* have a timeout. Read eap.conf.
Do you mean under the cache directive? If so that is set to 48 hours, yet failure occurred within 30 minutes. Our wireless controllers have a 1800 second client session timeout, forcing a reauth every 30 mins. Below is an excerpt of the radpostauth table for an affected user, it shows the user re-authing every 30 mins though the night. I migrated back to 2.1.8 (from 2.1.7) at 08:41 this morning. The client reauthed successfully once after the reload at 08:48, but then fails at the first 30min session timeout after that. 3843824 554694@swansea.ac.uk Access-Accept 2010-01-13 02:17:26 3845505 554694@swansea.ac.uk Access-Accept 2010-01-13 02:47:26 3846902 554694@swansea.ac.uk Access-Accept 2010-01-13 03:17:26 3848181 554694@swansea.ac.uk Access-Accept 2010-01-13 03:47:26 3849298 554694@swansea.ac.uk Access-Accept 2010-01-13 04:17:26 3850365 554694@swansea.ac.uk Access-Accept 2010-01-13 04:47:26 3851351 554694@swansea.ac.uk Access-Accept 2010-01-13 05:17:51 3852249 554694@swansea.ac.uk Access-Accept 2010-01-13 05:47:51 3853197 554694@swansea.ac.uk Access-Accept 2010-01-13 06:17:51 3854151 554694@swansea.ac.uk Access-Accept 2010-01-13 06:47:51 3855154 554694@swansea.ac.uk Access-Accept 2010-01-13 07:17:51 3856266 554694@swansea.ac.uk Access-Accept 2010-01-13 07:47:51 3857449 554694@swansea.ac.uk Access-Accept 2010-01-13 08:18:16 3858680 554694@swansea.ac.uk Access-Accept 2010-01-13 08:48:16 3860025 554694@swansea.ac.uk Access-Reject 2010-01-13 09:18:16 3860091 554694@swansea.ac.uk Access-Reject 2010-01-13 09:19:37 3860144 554694@swansea.ac.uk Access-Reject 2010-01-13 09:20:58 3860165 554694@swansea.ac.uk Access-Reject 2010-01-13 09:21:23 Thanks, Jezz.
Hi Alan, I've just been perusing the release notes for 2.1.9 and I see a bug fix... " Set EAP-Session-Resumed = Yes, not "No" when session is resumed. " Can you confirm if this is relating to the problem I reported in the conversation below? Many thanks, Jezz.
-----Original Message----- From: freeradius-users- bounces+j.d.f.palmer=swansea.ac.uk@lists.freeradius.org [mailto:freeradius-users- bounces+j.d.f.palmer=swansea.ac.uk@lists.freeradius.org] On Behalf Of Alan DeKok Sent: 12 January 2010 11:33 To: FreeRadius users mailing list Subject: Re: FR 2.1.8 Issue - Unjustified(?) Access-Rejects.
Palmer J.D.F. wrote:
We migrated to 2.1.8 (from 2.1.7) last week while things were quiet, as the users have re-appeared after the holiday we've started to receive a few reports from users stating that they have been getting lots of prompts for credentials.
The log says:
... WARNING: No information in cached session!
This means that the session wasn't cached, and they are trying to resume a session that never was started. The change in 2.1.8 is there to work around a bug in OpenSSL.
The only other alternative is that they *are* resuming a valid session, but (a) after the session has timed out, or (b) where no User-Name was cached from the inner tunnel session.
Is this likely to be a configuration error (no changes were made to the 2.1.7 config), or a bug?
Try increasing the size of the cache. Try ensuring that there is always a User-Name in the inner tunnel. This user name is cached, and is checked on session resumption.
Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Palmer J.D.F. wrote:
Hi Alan,
I've just been perusing the release notes for 2.1.9 and I see a bug fix... " Set EAP-Session-Resumed = Yes, not "No" when session is resumed. "
Can you confirm if this is relating to the problem I reported in the conversation below?
No, it's not. The fix for that issue will be in 2.1.10. If you need it now, see http://git.freeradius.org, and grab the "v2.1.x" branch. Alan DeKok.
No, it's not.
The fix for that issue will be in 2.1.10. If you need it now, see http://git.freeradius.org, and grab the "v2.1.x" branch.
Ok thanks Alan. I'll most likely wait until 2.1.10. Do you have any details on the bug? Thanks, Jezz.
Palmer J.D.F. wrote:
Ok thanks Alan. I'll most likely wait until 2.1.10. Do you have any details on the bug?
FreeRADIUS requires IDs to uniquely identify each SSL session. At some point, OpenSSL changed their code to *not* generate or store IDs. So... many of the assumptions of the server broke. Alan DeKok.
participants (7)
-
Alan DeKok -
Alexander Clouter -
Arran Cudbard-Bell -
Bjørn Mork -
Palmer J.D.F. -
Phil Mayers -
Stefan Winter