Configurations for proxying radius requests to home-server using TCP

[Alan DeKok]

On Aug 26, 2021, at 2:48 PM, Abhilash Y G <ygabhi at gmail.com> wrote:
> 
> Proxying to homeserver using a TCP connection is causing issue.
> I am seeing the response from the homeserver, it seems like freeradius
> running on proxy server is not honoring the response.
> I have attached the debug logs collected on the proxying server
> (debug_proxy_server.txt), homeservers (debug_homeserver.txt).
> 
> I had to trim the logs to focus on issue logs due to limitation in sending
> attachments more than 500KB.

  It's generally a good idea to trim the logs so that only the relevant information is sent to the list.  If you send megabytes of irrelevant logs, it's likely that people will just ignore the message.

  Plus, the debug log shows the server receiving many packets at the same time.  This makes it more difficult to see what's going on.

  One useful skill for debugging is knowing how to simplify the problem.  Don't try to debug issues in production, with tons of packets going by.  It will be impossible.  Instead, set up one or more test systems.  You can then send ONE request, and follow it through the proxy chain.

> Please advise if any configuration changes are needed on the proxying
> server to handle the requests without timeouts.

  That is not a useful way to phrase things.  As you've been told, FreeRADIUS replies to all packets it gets, UNLESS you've done something to break the configuration.

  From the debug log on the proxy:

(28) Received Access-Request Id 27 from 172.29.93.13:2000 to 0.0.0.0:2083 length 687
(28)   User-Name = "03111800000899580"
(28)   NAS-Identifier = "wifi"
...

  This is proxied:

(28) Proxying request to home server 206.XX.XXX.70 port 1234 timeout 3.000000
(28) Sent Access-Request Id 120 from 172.19.0.2:37787 to 206.XX.XXX.70:1234 length 691
(28)   User-Name = "03111800000899580"
(28)   NAS-Identifier = "wifi"


  Which then gets to the home server:

(13) Received Access-Request Id 120 from 206.XX.XXX.117:37787 to 0.0.0.0:1234 length 691
(13)   User-Name = "03111800000899580"
(13)   NAS-Identifier = "wifi"

  The IDs match.  But the IP addresses are different.  So there's a NAT, or another proxy server involved.

  That request is proxied in turn to another server:

(13) Proxying request to home server 206.XX.XXX.71 port 1817 timeout 3.000000
(13) Sent Access-Request Id 54 from 0.0.0.0:44020 to 206.XX.XXX.71:1817 length 696
(13)   User-Name = "03111800000899580"

  Which replies:

(13) Received Access-Accept Id 54 from 206.XX.XXX.71:1817 to 172.18.0.2:44020 length 640
(13)   Class = 0x4469616d657465722f6175732d653035342d726d722e636f72702e776179706f72742e6e65743b313632393739323030323b343531363b3033313131383030303030383939353840776c616e2e6d6e633138302e6d63633331312e336770706e6574776f726b2e6f72673b6175732d653035342d726d722e636f72702e776179706f72742e6e6574

...

  And sends the Access-Challenge back to the proxy:

(13) Sent Access-Accept Id 120 from 0.0.0.0:1234 to 206.XX.XXX.117:37787 length 0
(13)   Class = 0x4469616d657465722f6175732d653035342d726d722e636f72702e776179706f72742e6e65743b313632393739323030323b343531363b3033313131383030303030383939353840776c616e2e6d6e633138302e6d63633331312e336770706e6574776f726b2e6f72673b6175732d653035342d726d722e636f72702e776179706f72742e6e6574

...

  And then nothing comes back to the proxy.

  But there are other weird things in the trace.  On the "home server", it says:

(5)   State = 0x4469616d657465722f68706e623032766f6c746568762e6570632e6d6e633138302e6d63633331312e336770706e6574776f726b2e6f72672f6570632e6d6e633138302e6d63633331312e336770706e6574776f726b2e6f72672f6175732d653035342d726d722e636f72702e776179706f72742e6e65743b313632393739323030323b343531363b3033313131383030303030383939353840776c616e2e6d6e633138302e6d63633331312e336770706e6574776f726b2e6f72673b6175732d653035342d726d722e636f72702e776179706f72742e6e6574

  What the heck is that?  Oh wait, it's just ASCII:

Diameter/hpnb02voltehv.epc.mnc180.mcc311.3gppnetwork.org/epc.mnc180.mcc311.3gppnetwork.org/aus-e054-rmr.corp.wayport.net;1629792002;4516;0311180000089958 at wlan.mnc180.mcc311.3gppnetwork.org;aus-e054-rmr.corp.wayport.net

  So no, you are NOT using FreeRADIUS everywhere.  FreeRADIUS DOES NOT produce that kind of State attribute.  So you're using some kind of RADIUS to Diameter gateway.

  There's also this:

(5)   Proxy-State = 0x000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfc

  Again, that is ridiculous, and not created by FreeRADIUS.

  But 

  So to summarize:

* the debug output shows that many packets get replies, but some don't get replies.
* There's not one proxy. 
* There's two
* the home server isn't a home server.  It's a proxy to the "real" home server 
* Almost all of the packets are going back and forth just fine.  i.e. ALL RADIUS servers reply
* the ACTUAL home server is some kind of horrific RADIUS to Diameter gateway.

  What's happening is that there's some kind of NAT gateway / firewall which is dropping the TCP traffic.  It's probably either blocking the TCP connections after a bit, or it's tearing them down.

  Don't put firewalls between your RADIUS servers.  Don't have firewalls drop RADIUS traffic.  It's bad.

  PLEASE set up a test system.  Trying to debug issues in production is just too difficult.  You get megabytes of logs, and no idea where to look for problems.

  There's nothing wrong with FreeRADIUS.  Your network is broken.  No amount of poking FreeRADIUS will fix the network.  Go fix the network.

  Alan DeKok.