Hello, now this is unpleasant: Apparently, many OSes set the DF bit on every datagram they send, if Path MTU discovery is turned on. Even on UDP packets. This breaks your RADIUS communication if datagram > MTU. I have reason to believe that this has led to numerous problems with EAP-TLS in eduroam, for example. It is of course not in principle a FreeRADIUS problem, but an OS one (speaking here of Linux in particular), but other software like BIND takes precautions against that and I think it might be good to do the same in FreeRADIUS as well (i.e. explicitly disable IP_DONTFRAG socket option). To illustrate the problem (FR 2.1.6 on a host called radius-1): 09:54:55.234018 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 291) tld1.eduroam.lu.56581 > radius-1.restena.lu.radius: [udp sum ok] RADIUS, length: 263 09:54:55.234315 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 314) radius-1.restena.lu.tdp-suite > eomund.restena.lu.radius: [udp sum ok] RADIUS, length: 286 09:54:55.241858 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto UDP (17), length 1123) eomund.restena.lu.radius > radius-1.restena.lu.tdp-suite: [udp sum ok] RADIUS, length: 1095 09:54:55.242170 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 1118) radius-1.restena.lu.radius > tld1.eduroam.lu.56581: [udp sum ok] RADIUS, length: 1090 "Flags [DF]" means pain. One workaround is to disable Path MTU discovery on the IP layer of the host (/proc/sys/net/ipv4/ip_no_pmtu_disc), but obviously, that's a bad workaround since it will ruin Path MTU discovery for every TCP service on the same host. Luckily, IPv6 doesn't seem to be affected since MTU discovery works differently there (AFAIK). The BIND sources have a small workaround in their socket code to get around that: #if defined(IP_MTU_DISCOVER) && defined(IP_PMTUDISC_DONT) /* * Turn off Path MTU discovery on IPv4/UDP sockets. */ if (sock->pf == AF_INET) { int action = IP_PMTUDISC_DONT; (void)setsockopt(sock->fd, IPPROTO_IP, IP_MTU_DISCOVER, &action, sizeof(action)); } #endif #if defined(IP_DONTFRAG) /* * Turn off Path MTU discovery on IPv4/UDP sockets. */ if (sock->pf == AF_INET) { int off = 0; (void)setsockopt(sock->fd, IPPROTO_IP, IP_DONTFRAG, &off, sizeof(off)); } #endif May I suggest to do a similar thing in FreeRADIUS? The same happens on Radiator BTW, and I will post a similar message to these guys as well. Greetings, Stefan Winter -- Stefan WINTER Ingenieur de Recherche Fondation RESTENA - Réseau Téléinformatique de l'Education Nationale et de la Recherche 6, rue Richard Coudenhove-Kalergi L-1359 Luxembourg Tel: +352 424409 1 Fax: +352 422473
Stefan Winter wrote:
One workaround is to disable Path MTU discovery on the IP layer of the host (/proc/sys/net/ipv4/ip_no_pmtu_disc), but obviously, that's a bad workaround since it will ruin Path MTU discovery for every TCP service on the same host. Luckily, IPv6 doesn't seem to be affected since MTU discovery works differently there (AFAIK).
OK.
The BIND sources have a small workaround in their socket code to get around that:
That seems reasonable (which is a surprising thing to say about something that Bind does...)
May I suggest to do a similar thing in FreeRADIUS?
Done && committed to git. Please test. :) Alan DeKok.
Hi,
May I suggest to do a similar thing in FreeRADIUS?
Done && committed to git. Please test. :)
libtool: compile: gcc -g -O2 -D_REENTRANT -D_POSIX_PTHREAD_SEMANTICS -Wall -D_GNU_SOURCE -g -Wshadow -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -Wnested-externs -W -Wredundant-decls -Wundef -D_LIBRADIUS -I/root/freeradius-server-GIT-pre-217/src -c packet.c -fPIC -DPIC -o .libs/packet.o packet.c: In function ‘fr_socket’: packet.c:238: error: ‘sock’ undeclared (first use in this function) packet.c:238: error: (Each undeclared identifier is reported only once packet.c:238: error: for each function it appears in.) packet.c:239: error: ‘action’ undeclared (first use in this function) gmake[4]: *** [packet.lo] Error 1 gmake[4]: Leaving directory `/root/freeradius-server-GIT-pre-217/src/lib' gmake[3]: *** [common] Error 2 gmake[3]: Leaving directory `/root/freeradius-server-GIT-pre-217/src' gmake[2]: *** [all] Error 2 gmake[2]: Leaving directory `/root/freeradius-server-GIT-pre-217/src' gmake[1]: *** [common] Error 2 gmake[1]: Leaving directory `/root/freeradius-server-GIT-pre-217' make: *** [all] Error 2 That should be sockfd instead of sock->fd in 238, right? And the first #if clause misses int action = IP_PMTUDISC_DONT; Then it compiles alright, but doesn't solve the problem: 11:51:25.253733 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 221) radius-1.restena.lu.tdp-suite > eomund.restena.lu.radius: [udp sum ok] RADIUS, length: 193 Greetings, Stefan Winter -- Stefan WINTER Ingenieur de Recherche Fondation RESTENA - Réseau Téléinformatique de l'Education Nationale et de la Recherche 6, rue Richard Coudenhove-Kalergi L-1359 Luxembourg Tel: +352 424409 1 Fax: +352 422473
Stefan Winter wrote:
That should be sockfd instead of sock->fd in 238, right?
Yes.
And the first #if clause misses
int action = IP_PMTUDISC_DONT;
OK... fixed that, too.
Then it compiles alright, but doesn't solve the problem:
11:51:25.253733 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 221) radius-1.restena.lu.tdp-suite > eomund.restena.lu.radius: [udp sum ok] RADIUS, length: 193
Maybe the setsockopt() has to be done after the bind()... Alan DeKok.
Hi,
Then it compiles alright, but doesn't solve the problem:
11:51:25.253733 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 221) radius-1.restena.lu.tdp-suite > eomund.restena.lu.radius: [udp sum ok] RADIUS, length: 193
Maybe the setsockopt() has to be done after the bind()...
You could always check bind's code: in their distribution tarball, it's in lib/isc/unix/socket.c There's a plethora of platform-specific #if directives that makes the code really hard to read. But from what it looks like, binding happens in a separate function and will be called separately after the socket is opened with the PMTU sockopts. But you may want to verify that yourself :-) Stefan -- Stefan WINTER Ingenieur de Recherche Fondation RESTENA - Réseau Téléinformatique de l'Education Nationale et de la Recherche 6, rue Richard Coudenhove-Kalergi L-1359 Luxembourg Tel: +352 424409 1 Fax: +352 422473
Hi,
11:51:25.253733 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 221) radius-1.restena.lu.tdp-suite > eomund.restena.lu.radius: [udp sum ok] RADIUS, length: 193
Maybe the setsockopt() has to be done after the bind()...
I played around a bit with this. Turned out that no matter where I placed the code in this function, it didn't have any effect. Then I grep'ed the code for other socket() calls and found the only other occurence in main/listen.c: listen_bind() I replicated the code there and now all packets leave the server with the DF bit cleared (works for both replies and self-initiated messages). Greetings, Stefan Winter -- Stefan WINTER Ingenieur de Recherche Fondation RESTENA - Réseau Téléinformatique de l'Education Nationale et de la Recherche 6, rue Richard Coudenhove-Kalergi L-1359 Luxembourg Tel: +352 424409 1 Fax: +352 422473
Hi,
Hello,
now this is unpleasant: Apparently, many OSes set the DF bit on every datagram they send, if Path MTU discovery is turned on. Even on UDP packets. This breaks your RADIUS communication if datagram > MTU. I have reason to believe that this has led to numerous problems with EAP-TLS in eduroam, for example.
many of these problems are created by firewalls configured to throw away fragmented UDP (eg Solaris by default will discard fragmented packets). obviously UDP fragments are quite right and can be DoS material... but if you ensure only your RADIUS proxies to national proxies can actually have fragmented UDP then it fixes things nicely.
on the same host. Luckily, IPv6 doesn't seem to be affected since MTU discovery works differently there (AFAIK).
it does :-) alan
Hi,
many of these problems are created by firewalls configured to throw away fragmented UDP (eg Solaris by default will discard fragmented packets).
Yes, that's the problem we faced and eliminated about a year ago in eduroam. The issue at hand is that your firewall will likely never encounter a fragment because the originating hosts prevents fragmenting in the first place.
obviously UDP fragments are quite right and can be DoS material... but if you ensure only your RADIUS proxies to national proxies can actually have fragmented UDP then it fixes things nicely.
In a protocol like RADIUS UDP fragments can occur alright, and preventing them from the OS layer just breaks RADIUS. Many many people in eduroam who use EAP-TLS can sing a song of failed authentications at random places, when it worked alright at home. Admitted, this enables DoS onto 1812/UDP with non-first fragments, but too bad. Fragment support is *required* for proper RADIUS operation.
on the same host. Luckily, IPv6 doesn't seem to be affected since MTU discovery works differently there (AFAIK).
it does :-)
Thanks :-) Greetings, Stefan -- Stefan WINTER Ingenieur de Recherche Fondation RESTENA - Réseau Téléinformatique de l'Education Nationale et de la Recherche 6, rue Richard Coudenhove-Kalergi L-1359 Luxembourg Tel: +352 424409 1 Fax: +352 422473
participants (3)
-
Alan Buxey -
Alan DeKok -
Stefan Winter