TLS failover behaviour and a backtrace if want it.
Hi, Using 3.0.19 on Ubuntu from the Ubuntu networkradius repo. I've been doing some testing on tls proxy connections. sites-enabled/tls below. home_server rsh-haproxy-rp1 { ipaddr = 192.168.110.46 port = 2083 type = auth secret = radsec proto = tcp status_check = none tls { private_key_file = /etc/freeradius/certs/privkey.pem certificate_file = /etc/freeradius/certs/fullchain.pem ca_file = /etc/ssl/certs/ca-certificates.crt dh_file = ${certdir}/dh random_file = /dev/urandom fragment_size = 8192 ca_path = ${cadir} cipher_list = "DEFAULT" } limit { idle_timeout = 0 } } home_server prh-haproxy-rp1 { ipaddr = 192.168.12.200 port = 2083 type = auth secret = radsec proto = tcp status_check = none tls { private_key_file = /etc/freeradius/certs/privkey.pem certificate_file = /etc/freeradius/certs/fullchain.pem ca_file = /etc/ssl/certs/ca-certificates.crt dh_file = ${certdir}/dh random_file = /dev/urandom fragment_size = 8192 ca_path = ${cadir} cipher_list = "DEFAULT" } } home_server_pool some_radius_servers { type = fail-over home_server = rsh-haproxy-rp1 home_server = prh-haproxy-rp1 } realm DEFAULT { auth_pool = some_radius_servers } Firstly, I must admit I expected failover to be "within same request", but it takes a repeat request from the client should a tls server be unavailable. I guess this is just me misunderstanding the failover for tls, I assumed same as something like a redundant {} section. If the server isn't listening on 2083, i.e. service stopped: .. (1) Starting proxy to home server 192.168.110.46 port 2083 (1) server default { (1) } Failed opening new proxy socket 'proxy (0.0.0.0, 0) -> home_server (192.168.110.46, 2083)' : Failed connecting socket: Connection refused (1) Failed to insert request into the proxy list (1) There was no response configured: rejecting request .. If the client repeats the request, it tries the next server ok, but I'd be a little concerned some might not after a direct reject reply. Could someone please confirm this is by design? Also, I'm noticing a crash if the home server pool is depleted to zero, i.e. all servers are down. It's unlikely to happen, but you may be interested in coding these out. .. (2) } # authorize = updated (2) ERROR: Failed to find live home server: Cancelling proxy (2) WARNING: No home server selected (2) Clearing existing &reply: attributes (2) Found Post-Proxy-Type Fail-Authentication Thread 3 "freeradius" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fffe552e700 (LWP 169613)] 0x0000555555587c8b in ?? () (gdb) bt #0 0x0000555555587c8b in ?? () #1 0x000055555558e61a in ?? () #2 0x0000555555586a75 in ?? () #3 0x00007ffff6bde6db in start_thread (arg=0x7fffe552e700) at pthread_create.c:463 #4 0x00007ffff644b88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thanks Andy ******************************************************************************************************************** This message may contain confidential information. If you are not the intended recipient please inform the sender that you have received the message in error before deleting it. Please do not disclose, copy or distribute information in this e-mail or take any action in relation to its contents. To do so is strictly prohibited and may be unlawful. Thank you for your co-operation. NHSmail is the secure email and directory service available for all NHS staff in England and Scotland. NHSmail is approved for exchanging patient data and other sensitive information with NHSmail and other accredited email services. For more information and to find out how you can switch, https://portal.nhs.net/help/joiningnhsmail
On Nov 18, 2019, at 8:44 AM, FRANKS, Andy (SHREWSBURY AND TELFORD HOSPITAL NHS TRUST) via Freeradius-Users <freeradius-users@lists.freeradius.org> wrote:
Firstly, I must admit I expected failover to be "within same request", but it takes a repeat request from the client should a tls server be unavailable. I guess this is just me misunderstanding the failover for tls, I assumed same as something like a redundant {} section.
This is documented. Read proxy.conf: # # In 2.0, the server is always "synchronous", and setting # "synchronous = no" is impossible. This simplifies the # server and increases the stability of the network. # However, it means that the server (i.e. proxy) NEVER # originates packets. It proxies packets ONLY when it receives # a packet or a re-transmission from the NAS. If the NAS never # re-transmits, the proxy never re-transmits, either. This can # affect fail-over, where a packet does *not* fail over to a # second home server.. because the NAS never retransmits the # packet.
If the server isn't listening on 2083, i.e. service stopped:
.. (1) Starting proxy to home server 192.168.110.46 port 2083 (1) server default { (1) } Failed opening new proxy socket 'proxy (0.0.0.0, 0) -> home_server (192.168.110.46, 2083)' : Failed connecting socket: Connection refused (1) Failed to insert request into the proxy list (1) There was no response configured: rejecting request ..
If the client repeats the request, it tries the next server ok, but I'd be a little concerned some might not after a direct reject reply.
You can configure the proxy to do something else if the home server is down. Again, read proxy.conf. This is all documented. It tells you how to do failover between multiple home servers. It tells you how to have a "fallback" if the home servers are down.
Could someone please confirm this is by design?
It's not only by design, it's extensively documented. if you're editing proxy.conf, it helps to read the comments located there. The idea is that *you* can control what happens when a home server is down. The only caveat is that you actually have to tell the server what to do.
Also, I'm noticing a crash if the home server pool is depleted to zero, i.e. all servers are down. It's unlikely to happen, but you may be interested in coding these out.
If you're not running 3.0.20, upgrade. Packages are available at http://packages.networkradius.com If you're running a very old version of the server, then this crash has likely already been fixed. Alan DeKok.
Hi Alan, You're right of course, apologies I overlooked that. Running 3.0.19, pulled this morning from the networkradius repo but I'll try again later today. Thanks Andy -----Original Message----- From: Alan DeKok <aland@deployingradius.com> Sent: 18 November 2019 13:54 To: FreeRadius users mailing list <freeradius-users@lists.freeradius.org> Cc: FRANKS, Andy (SHREWSBURY AND TELFORD HOSPITAL NHS TRUST) <andy.franks1@nhs.net> Subject: Re: TLS failover behaviour and a backtrace if want it. On Nov 18, 2019, at 8:44 AM, FRANKS, Andy (SHREWSBURY AND TELFORD HOSPITAL NHS TRUST) via Freeradius-Users <freeradius-users@lists.freeradius.org> wrote:
Firstly, I must admit I expected failover to be "within same request", but it takes a repeat request from the client should a tls server be unavailable. I guess this is just me misunderstanding the failover for tls, I assumed same as something like a redundant {} section.
This is documented. Read proxy.conf: # # In 2.0, the server is always "synchronous", and setting # "synchronous = no" is impossible. This simplifies the # server and increases the stability of the network. # However, it means that the server (i.e. proxy) NEVER # originates packets. It proxies packets ONLY when it receives # a packet or a re-transmission from the NAS. If the NAS never # re-transmits, the proxy never re-transmits, either. This can # affect fail-over, where a packet does *not* fail over to a # second home server.. because the NAS never retransmits the # packet.
If the server isn't listening on 2083, i.e. service stopped:
.. (1) Starting proxy to home server 192.168.110.46 port 2083 (1) server default { (1) } Failed opening new proxy socket 'proxy (0.0.0.0, 0) -> home_server (192.168.110.46, 2083)' : Failed connecting socket: Connection refused (1) Failed to insert request into the proxy list (1) There was no response configured: rejecting request ..
If the client repeats the request, it tries the next server ok, but I'd be a little concerned some might not after a direct reject reply.
You can configure the proxy to do something else if the home server is down. Again, read proxy.conf. This is all documented. It tells you how to do failover between multiple home servers. It tells you how to have a "fallback" if the home servers are down.
Could someone please confirm this is by design?
It's not only by design, it's extensively documented. if you're editing proxy.conf, it helps to read the comments located there. The idea is that *you* can control what happens when a home server is down. The only caveat is that you actually have to tell the server what to do.
Also, I'm noticing a crash if the home server pool is depleted to zero, i.e. all servers are down. It's unlikely to happen, but you may be interested in coding these out.
If you're not running 3.0.20, upgrade. Packages are available at http://packages.networkradius.com If you're running a very old version of the server, then this crash has likely already been fixed. Alan DeKok. ******************************************************************************************************************** This message may contain confidential information. If you are not the intended recipient please inform the sender that you have received the message in error before deleting it. Please do not disclose, copy or distribute information in this e-mail or take any action in relation to its contents. To do so is strictly prohibited and may be unlawful. Thank you for your co-operation. NHSmail is the secure email and directory service available for all NHS staff in England and Scotland. NHSmail is approved for exchanging patient data and other sensitive information with NHSmail and other accredited email services. For more information and to find out how you can switch, https://portal.nhs.net/help/joiningnhsmail
On Nov 18, 2019, at 9:56 AM, FRANKS, Andy (SHREWSBURY AND TELFORD HOSPITAL NHS TRUST) via Freeradius-Users <freeradius-users@lists.freeradius.org> wrote:
Hi Alan, You're right of course, apologies I overlooked that.
Running 3.0.19, pulled this morning from the networkradius repo but I'll try again later today.
If it still dies, please report a backtrace with symbols. See doc/bugs for details Alan DeKok.
Thanks Alan, Still occurs in 3.0.20, so I'll build from source asap (as the network radius Ubuntu package binary seems to have no symbols) Kind regards -----Original Message----- From: Alan DeKok <aland@deployingradius.com> Sent: 18 November 2019 15:50 To: FreeRadius users mailing list <freeradius-users@lists.freeradius.org> Cc: FRANKS, Andy (SHREWSBURY AND TELFORD HOSPITAL NHS TRUST) <andy.franks1@nhs.net> Subject: Re: TLS failover behaviour and a backtrace if want it.
On Nov 18, 2019, at 9:56 AM, FRANKS, Andy (SHREWSBURY AND TELFORD HOSPITAL NHS TRUST) via Freeradius-Users <freeradius-users@lists.freeradius.org> wrote:
Hi Alan, You're right of course, apologies I overlooked that.
Running 3.0.19, pulled this morning from the networkradius repo but I'll try again later today.
If it still dies, please report a backtrace with symbols. See doc/bugs for details Alan DeKok. ******************************************************************************************************************** This message may contain confidential information. If you are not the intended recipient please inform the sender that you have received the message in error before deleting it. Please do not disclose, copy or distribute information in this e-mail or take any action in relation to its contents. To do so is strictly prohibited and may be unlawful. Thank you for your co-operation. NHSmail is the secure email and directory service available for all NHS staff in England and Scotland. NHSmail is approved for exchanging patient data and other sensitive information with NHSmail and other accredited email services. For more information and to find out how you can switch, https://portal.nhs.net/help/joiningnhsmail
On Tue, 2019-11-19 at 14:44 +0000, FRANKS, Andy (SHREWSBURY AND TELFORD HOSPITAL NHS TRUST) via Freeradius-Users wrote:
Still occurs in 3.0.20, so I'll build from source asap (as the network radius Ubuntu package binary seems to have no symbols)
Install the freeradius-dbg package? -- Matthew
That worked fine. Please say if you need the full pre-crash output, I've chopped it to what I thought was needed. The crash happens when you get to the "no more servers available", be that the only 1 server in the pool that's down, or 2 that are both down etc. Hopefully unlikely in production anyway! Using radsec exclusively for forwarding, just in case it's relevant, no tried with udp/tcp. By the way, not worth another mail I don't think; I noticed something else tiny in the output that might need sorting, a typo really: In the warning message about not setting the tls version to 1.2 for radsec, it says : Please set: min_tls_version = "1.2" But then I think configuration line seems to be tls_min_version not min_tls_version listen { .. tls { .. tls_max_version = "" tls_min_version = "1.0" .. ? Anyway the crash, hope it's more use this time.. GDB output: (1) ERROR: Failed to find live home server: Cancelling proxy (1) WARNING: No home server selected (1) Clearing existing &reply: attributes (1) Found Post-Proxy-Type Fail-Authentication Thread 4 "freeradius" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fffea203700 (LWP 22404)] process_proxy_reply (request=request@entry=0x555555bdc400, reply=reply@entry=0x0) at src/main/process.c:2483 (gdb) bt #0 process_proxy_reply (request=request@entry=0x555555bdc400, reply=reply@entry=0x0) at src/main/process.c:2483 #1 0x000055555558e61a in request_running (request=0x555555bdc400, action=<optimized out>) at src/main/process.c:1648 #2 0x0000555555586a75 in request_handler_thread (arg=0x555555bcd310) at src/main/threads.c:826 #3 0x00007ffff6bde6db in start_thread (arg=0x7fffea203700) at pthread_create.c:463 #4 0x00007ffff644b88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Kind Regards Andy -----Original Message----- From: Matthew Newton <mcn@freeradius.org> Sent: 19 November 2019 17:38 To: FreeRadius users mailing list <freeradius-users@lists.freeradius.org> Cc: FRANKS, Andy (SHREWSBURY AND TELFORD HOSPITAL NHS TRUST) <andy.franks1@nhs.net> Subject: Re: TLS failover behaviour and a backtrace if want it. On Tue, 2019-11-19 at 14:44 +0000, FRANKS, Andy (SHREWSBURY AND TELFORD HOSPITAL NHS TRUST) via Freeradius-Users wrote:
Still occurs in 3.0.20, so I'll build from source asap (as the network radius Ubuntu package binary seems to have no symbols)
Install the freeradius-dbg package? -- Matthew ******************************************************************************************************************** This message may contain confidential information. If you are not the intended recipient please inform the sender that you have received the message in error before deleting it. Please do not disclose, copy or distribute information in this e-mail or take any action in relation to its contents. To do so is strictly prohibited and may be unlawful. Thank you for your co-operation. NHSmail is the secure email and directory service available for all NHS staff in England and Scotland. NHSmail is approved for exchanging patient data and other sensitive information with NHSmail and other accredited email services. For more information and to find out how you can switch, https://portal.nhs.net/help/joiningnhsmail
On Nov 20, 2019, at 8:21 AM, FRANKS, Andy (SHREWSBURY AND TELFORD HOSPITAL NHS TRUST) via Freeradius-Users <freeradius-users@lists.freeradius.org> wrote:
That worked fine.
Good to hear.
Please say if you need the full pre-crash output, I've chopped it to what I thought was needed.
No, that's OK.
The crash happens when you get to the "no more servers available", be that the only 1 server in the pool that's down, or 2 that are both down etc. Hopefully unlikely in production anyway!
I happens. *Anything* can happen with RADIUS :(
Using radsec exclusively for forwarding, just in case it's relevant, no tried with udp/tcp.
By the way, not worth another mail I don't think; I noticed something else tiny in the output that might need sorting, a typo really:
In the warning message about not setting the tls version to 1.2 for radsec, it says :
Please set: min_tls_version = "1.2"
? The code is: $ git grep 'Please set' src/main/tls.c: WARN("Please set: tls_min_version = \"1.2\""); So that seems OK.
Anyway the crash, hope it's more use this time..
Yes. I've pushed a fix. Alan DeKok.
Thank you Alan, super quick! Kind Regards Andy -----Original Message----- From: Alan DeKok <aland@deployingradius.com> Sent: 20 November 2019 13:33 To: FreeRadius users mailing list <freeradius-users@lists.freeradius.org> Cc: Matthew Newton <mcn@freeradius.org>; FRANKS, Andy (SHREWSBURY AND TELFORD HOSPITAL NHS TRUST) <andy.franks1@nhs.net> Subject: Re: TLS failover behaviour and a backtrace if want it. On Nov 20, 2019, at 8:21 AM, FRANKS, Andy (SHREWSBURY AND TELFORD HOSPITAL NHS TRUST) via Freeradius-Users <freeradius-users@lists.freeradius.org> wrote:
That worked fine.
Good to hear.
Please say if you need the full pre-crash output, I've chopped it to what I thought was needed.
No, that's OK.
The crash happens when you get to the "no more servers available", be that the only 1 server in the pool that's down, or 2 that are both down etc. Hopefully unlikely in production anyway!
I happens. *Anything* can happen with RADIUS :(
Using radsec exclusively for forwarding, just in case it's relevant, no tried with udp/tcp.
By the way, not worth another mail I don't think; I noticed something else tiny in the output that might need sorting, a typo really:
In the warning message about not setting the tls version to 1.2 for radsec, it says :
Please set: min_tls_version = "1.2"
? The code is: $ git grep 'Please set' src/main/tls.c: WARN("Please set: tls_min_version = \"1.2\""); So that seems OK.
Anyway the crash, hope it's more use this time..
Yes. I've pushed a fix. Alan DeKok. ******************************************************************************************************************** This message may contain confidential information. If you are not the intended recipient please inform the sender that you have received the message in error before deleting it. Please do not disclose, copy or distribute information in this e-mail or take any action in relation to its contents. To do so is strictly prohibited and may be unlawful. Thank you for your co-operation. NHSmail is the secure email and directory service available for all NHS staff in England and Scotland. NHSmail is approved for exchanging patient data and other sensitive information with NHSmail and other accredited email services. For more information and to find out how you can switch, https://portal.nhs.net/help/joiningnhsmail
participants (3)
-
Alan DeKok -
FRANKS, Andy (SHREWSBURY AND TELFORD HOSPITAL NHS TRUST) -
Matthew Newton