DHCP relay - wrong interface for relayed packet
Hi Folks, This may be a Matthew specific question - I'm testing out the patch he kindly posted to me on uknof at the beginning of June to enable relaying to a different UDP port. Early disclaimer: I'm testing this on a Cumulus Linux box - whilst they are Debian underneath, there are some 'Cumulus-isms' on the network side that potentially might be in play here. I am using freeradius_head_v3-2-x (downloaded 12/06/2022) with these patches applied: https://github.com/mcnewton/freeradius-server/commit/23f1856611722e30c997969... For testing, I have a Cumulus box with two interfaces that we care about: vlan4091 which has a test client connected to it, and vlan545 which is the connection back to the rest of the network. Our test DHCP server (217.198.54.101) is in this "rest of the network" - IP routing all works as expected, and you can ping the DHCP server from the vlan4091 interface IP address (217.198.52.2) on the Cumulus router, and from clients on that VLAN - and vice versa. When I send in a test DHCP request from a client connected to vlan4091, FR receives it and and re-sends it out to 217.198.54.101 on port 1067. However, the DHCP server never sees that packet. After some rummaging with strace and tcpdump, I've determined that FR is transmitting the relayed packet out of vlan4091 (I saw suspect looking ARPs, and put in a static ARP entry after which I see the frame transmitted on vlan4091) and not following the default route up out of vlan545. Looking in proto_dhcp/dhcpd.c at dhcprelay_process_client_request, there's nothing in there to suggest that the outgoing interface is set and routing is overridden: But it looks like some parts are copied over from the socket handling the client communication; is there something here that could cause what I'm seeing? My other thought that if this was a general problem it would break FR DHCP relaying in a lot of people's use cases - and there aren't lots of people grumbling about this, so there must be something wacky going on with my setup but I'm slightly at a loss to see what's happening. Any pointers greatfully received. Paul. # radiusd -X FreeRADIUS Version 3.2.1 Copyright (C) 1999-2021 The FreeRADIUS server project and contributors There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE You may redistribute copies of FreeRADIUS under the terms of the GNU General Public License For more information about these matters, see the file named COPYRIGHT Starting - reading configuration files ... including dictionary file /usr/local/share/freeradius/dictionary including dictionary file /usr/local/share/freeradius/dictionary.dhcp including dictionary file /usr/local/share/freeradius/dictionary.vqp including dictionary file /usr/local/etc/raddb/dictionary including configuration file /usr/local/etc/raddb/radiusd.conf including configuration file /usr/local/etc/raddb/proxy.conf including configuration file /usr/local/etc/raddb/clients.conf including files in directory /usr/local/etc/raddb/mods-enabled/ including configuration file /usr/local/etc/raddb/mods-enabled/attr_filter including configuration file /usr/local/etc/raddb/mods-enabled/date including configuration file /usr/local/etc/raddb/mods-enabled/preprocess including configuration file /usr/local/etc/raddb/mods-enabled/exec including configuration file /usr/local/etc/raddb/mods-enabled/detail including configuration file /usr/local/etc/raddb/mods-enabled/echo including configuration file /usr/local/etc/raddb/mods-enabled/always including configuration file /usr/local/etc/raddb/mods-enabled/expr including configuration file /usr/local/etc/raddb/mods-enabled/dhcp including configuration file /usr/local/etc/raddb/mods-enabled/digest including configuration file /usr/local/etc/raddb/mods-enabled/linelog including files in directory /usr/local/etc/raddb/policy.d/ including configuration file /usr/local/etc/raddb/policy.d/moonshot-targeted-ids including configuration file /usr/local/etc/raddb/policy.d/debug including configuration file /usr/local/etc/raddb/policy.d/operator-name including configuration file /usr/local/etc/raddb/policy.d/canonicalization including configuration file /usr/local/etc/raddb/policy.d/control including configuration file /usr/local/etc/raddb/policy.d/cui including configuration file /usr/local/etc/raddb/policy.d/abfab-tr including configuration file /usr/local/etc/raddb/policy.d/filter including configuration file /usr/local/etc/raddb/policy.d/dhcp including configuration file /usr/local/etc/raddb/policy.d/accounting including configuration file /usr/local/etc/raddb/policy.d/eap including configuration file /usr/local/etc/raddb/policy.d/rfc7542 including files in directory /usr/local/etc/raddb/sites-enabled/ including configuration file /usr/local/etc/raddb/sites-enabled/test_dhcp_relay main { security { allow_core_dumps = no } name = "radiusd" prefix = "/usr/local" localstatedir = "/usr/local/var" logdir = "/usr/local/var/log/radius" run_dir = "/usr/local/var/run/radiusd" } main { name = "radiusd" prefix = "/usr/local" localstatedir = "/usr/local/var" sbindir = "/usr/local/sbin" logdir = "/usr/local/var/log/radius" run_dir = "/usr/local/var/run/radiusd" libdir = "/usr/local/lib" radacctdir = "/usr/local/var/log/radius/radacct" hostname_lookups = no max_request_time = 30 cleanup_delay = 5 max_requests = 16384 postauth_client_lost = no pidfile = "/usr/local/var/run/radiusd/radiusd.pid" checkrad = "/usr/local/sbin/checkrad" debug_level = 0 proxy_requests = yes log { stripped_names = no auth = no auth_badpass = no auth_goodpass = no colourise = yes msg_denied = "You are already logged in - access denied" } resources { } security { max_attributes = 200 reject_delay = 1.000000 status_server = yes allow_vulnerable_openssl = "no" } } radiusd: #### Loading Realms and Home Servers #### proxy server { retry_delay = 5 retry_count = 3 default_fallback = no dead_time = 120 wake_all_if_all_dead = no } home_server localhost { ipaddr = 127.0.0.1 port = 1812 type = "auth" secret = <<< secret >>> response_window = 20.000000 response_timeouts = 1 max_outstanding = 65536 zombie_period = 40 status_check = "status-server" ping_interval = 30 check_interval = 30 check_timeout = 4 num_answers_to_alive = 3 revive_interval = 120 limit { max_connections = 16 max_requests = 0 lifetime = 0 idle_timeout = 0 } coa { irt = 2 mrt = 16 mrc = 5 mrd = 30 } } home_server_pool my_auth_failover { type = fail-over home_server = localhost } realm example.com { auth_pool = my_auth_failover } realm LOCAL { } radiusd: #### Loading Clients #### client localhost { ipaddr = 127.0.0.1 require_message_authenticator = no secret = <<< secret >>> nas_type = "other" proto = "*" limit { max_connections = 16 lifetime = 0 idle_timeout = 30 } } client localhost_ipv6 { ipv6addr = ::1 require_message_authenticator = no secret = <<< secret >>> limit { max_connections = 16 lifetime = 0 idle_timeout = 30 } } Debugger not attached radiusd: #### Instantiating modules #### modules { # Loaded module rlm_attr_filter # Loading module "attr_filter.post-proxy" from file /usr/local/etc/raddb/mods-enabled/attr_filter attr_filter attr_filter.post-proxy { filename = "/usr/local/etc/raddb/mods-config/attr_filter/post-proxy" key = "%{Realm}" relaxed = no } # Loading module "attr_filter.pre-proxy" from file /usr/local/etc/raddb/mods-enabled/attr_filter attr_filter attr_filter.pre-proxy { filename = "/usr/local/etc/raddb/mods-config/attr_filter/pre-proxy" key = "%{Realm}" relaxed = no } # Loading module "attr_filter.access_reject" from file /usr/local/etc/raddb/mods-enabled/attr_filter attr_filter attr_filter.access_reject { filename = "/usr/local/etc/raddb/mods-config/attr_filter/access_reject" key = "%{User-Name}" relaxed = no } # Loading module "attr_filter.access_challenge" from file /usr/local/etc/raddb/mods-enabled/attr_filter attr_filter attr_filter.access_challenge { filename = "/usr/local/etc/raddb/mods-config/attr_filter/access_challenge" key = "%{User-Name}" relaxed = no } # Loading module "attr_filter.accounting_response" from file /usr/local/etc/raddb/mods-enabled/attr_filter attr_filter attr_filter.accounting_response { filename = "/usr/local/etc/raddb/mods-config/attr_filter/accounting_response" key = "%{User-Name}" relaxed = no } # Loading module "attr_filter.coa" from file /usr/local/etc/raddb/mods-enabled/attr_filter attr_filter attr_filter.coa { filename = "/usr/local/etc/raddb/mods-config/attr_filter/coa" key = "%{User-Name}" relaxed = no } # Loaded module rlm_date # Loading module "date" from file /usr/local/etc/raddb/mods-enabled/date date { format = "%b %e %Y %H:%M:%S %Z" utc = no } # Loading module "wispr2date" from file /usr/local/etc/raddb/mods-enabled/date date wispr2date { format = "%Y-%m-%dT%H:%M:%S" utc = no } # Loaded module rlm_preprocess # Loading module "preprocess" from file /usr/local/etc/raddb/mods-enabled/preprocess preprocess { huntgroups = "/usr/local/etc/raddb/mods-config/preprocess/huntgroups" hints = "/usr/local/etc/raddb/mods-config/preprocess/hints" with_ascend_hack = no ascend_channels_per_line = 23 with_ntdomain_hack = no with_specialix_jetstream_hack = no with_cisco_vsa_hack = no with_alvarion_vsa_hack = no } # Loaded module rlm_exec # Loading module "exec" from file /usr/local/etc/raddb/mods-enabled/exec exec { wait = no input_pairs = "request" shell_escape = yes timeout = 10 } # Loaded module rlm_detail # Loading module "detail" from file /usr/local/etc/raddb/mods-enabled/detail detail { filename = "/usr/local/var/log/radius/radacct/%{%{Packet-Src-IP-Address}:-%{Packet-Src-IPv6-Address}}/detail-%Y%m%d" header = "%t" permissions = 384 locking = no escape_filenames = no log_packet_header = no } # Loading module "echo" from file /usr/local/etc/raddb/mods-enabled/echo exec echo { wait = yes program = "/bin/echo %{User-Name}" input_pairs = "request" output_pairs = "reply" shell_escape = yes } # Loaded module rlm_always # Loading module "reject" from file /usr/local/etc/raddb/mods-enabled/always always reject { rcode = "reject" simulcount = 0 mpp = no } # Loading module "fail" from file /usr/local/etc/raddb/mods-enabled/always always fail { rcode = "fail" simulcount = 0 mpp = no } # Loading module "ok" from file /usr/local/etc/raddb/mods-enabled/always always ok { rcode = "ok" simulcount = 0 mpp = no } # Loading module "handled" from file /usr/local/etc/raddb/mods-enabled/always always handled { rcode = "handled" simulcount = 0 mpp = no } # Loading module "invalid" from file /usr/local/etc/raddb/mods-enabled/always always invalid { rcode = "invalid" simulcount = 0 mpp = no } # Loading module "userlock" from file /usr/local/etc/raddb/mods-enabled/always always userlock { rcode = "userlock" simulcount = 0 mpp = no } # Loading module "notfound" from file /usr/local/etc/raddb/mods-enabled/always always notfound { rcode = "notfound" simulcount = 0 mpp = no } # Loading module "noop" from file /usr/local/etc/raddb/mods-enabled/always always noop { rcode = "noop" simulcount = 0 mpp = no } # Loading module "updated" from file /usr/local/etc/raddb/mods-enabled/always always updated { rcode = "updated" simulcount = 0 mpp = no } # Loaded module rlm_expr # Loading module "expr" from file /usr/local/etc/raddb/mods-enabled/expr expr { safe_characters = "@abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789.-_: /äéöüàâæçèéêëîïôœùûüaÿÄÉÖÜßÀÂÆÇÈÉÊËÎÏÔŒÙÛÜŸ" } # Loaded module rlm_dhcp # Loading module "dhcp" from file /usr/local/etc/raddb/mods-enabled/dhcp # Loaded module rlm_digest # Loading module "digest" from file /usr/local/etc/raddb/mods-enabled/digest # Loaded module rlm_linelog # Loading module "linelog" from file /usr/local/etc/raddb/mods-enabled/linelog linelog { filename = "/usr/local/var/log/radius/linelog" escape_filenames = no syslog_severity = "info" permissions = 384 format = "This is a log message for %{User-Name}" reference = "messages.%{%{reply:Packet-Type}:-default}" } # Loading module "log_accounting" from file /usr/local/etc/raddb/mods-enabled/linelog linelog log_accounting { filename = "/usr/local/var/log/radius/linelog-accounting" escape_filenames = no syslog_severity = "info" permissions = 384 format = "" reference = "Accounting-Request.%{%{Acct-Status-Type}:-unknown}" } instantiate { } # Instantiating module "attr_filter.post-proxy" from file /usr/local/etc/raddb/mods-enabled/attr_filter reading pairlist file /usr/local/etc/raddb/mods-config/attr_filter/post-proxy # Instantiating module "attr_filter.pre-proxy" from file /usr/local/etc/raddb/mods-enabled/attr_filter reading pairlist file /usr/local/etc/raddb/mods-config/attr_filter/pre-proxy # Instantiating module "attr_filter.access_reject" from file /usr/local/etc/raddb/mods-enabled/attr_filter reading pairlist file /usr/local/etc/raddb/mods-config/attr_filter/access_reject # Instantiating module "attr_filter.access_challenge" from file /usr/local/etc/raddb/mods-enabled/attr_filter reading pairlist file /usr/local/etc/raddb/mods-config/attr_filter/access_challenge # Instantiating module "attr_filter.accounting_response" from file /usr/local/etc/raddb/mods-enabled/attr_filter reading pairlist file /usr/local/etc/raddb/mods-config/attr_filter/accounting_response # Instantiating module "attr_filter.coa" from file /usr/local/etc/raddb/mods-enabled/attr_filter reading pairlist file /usr/local/etc/raddb/mods-config/attr_filter/coa # Instantiating module "preprocess" from file /usr/local/etc/raddb/mods-enabled/preprocess reading pairlist file /usr/local/etc/raddb/mods-config/preprocess/huntgroups reading pairlist file /usr/local/etc/raddb/mods-config/preprocess/hints # Instantiating module "detail" from file /usr/local/etc/raddb/mods-enabled/detail # Instantiating module "reject" from file /usr/local/etc/raddb/mods-enabled/always # Instantiating module "fail" from file /usr/local/etc/raddb/mods-enabled/always # Instantiating module "ok" from file /usr/local/etc/raddb/mods-enabled/always # Instantiating module "handled" from file /usr/local/etc/raddb/mods-enabled/always # Instantiating module "invalid" from file /usr/local/etc/raddb/mods-enabled/always # Instantiating module "userlock" from file /usr/local/etc/raddb/mods-enabled/always # Instantiating module "notfound" from file /usr/local/etc/raddb/mods-enabled/always # Instantiating module "noop" from file /usr/local/etc/raddb/mods-enabled/always # Instantiating module "updated" from file /usr/local/etc/raddb/mods-enabled/always # Instantiating module "linelog" from file /usr/local/etc/raddb/mods-enabled/linelog # Instantiating module "log_accounting" from file /usr/local/etc/raddb/mods-enabled/linelog } # modules radiusd: #### Loading Virtual Servers #### server { # from file /usr/local/etc/raddb/radiusd.conf } # server server dhcp.vlan4091 { # from file /usr/local/etc/raddb/sites-enabled/test_dhcp_relay # Loading dhcp DHCP-Discover {...} Compiling dhcp DHCP-Discover for attr DHCP-Message-Type # Loading dhcp DHCP-Request {...} Compiling dhcp DHCP-Request for attr DHCP-Message-Type } # server dhcp.vlan4091 radiusd: #### Opening IP addresses and Ports #### listen { type = "dhcp" ipaddr = * port = 67 } Listening on dhcp interface vlan4091 address * port 67 bound to server dhcp.vlan4091 Ready to process requests Received DHCP-Discover of Id 68879701 from 0.0.0.0:68 to 255.255.255.255:67 DHCP-Opcode = Client-Message DHCP-Hardware-Type = Ethernet DHCP-Hardware-Address-Length = 6 DHCP-Hop-Count = 0 DHCP-Transaction-Id = 1753716481 DHCP-Number-of-Seconds = 0 DHCP-Flags = 0 DHCP-Client-IP-Address = 0.0.0.0 DHCP-Your-IP-Address = 0.0.0.0 DHCP-Server-IP-Address = 0.0.0.0 DHCP-Gateway-IP-Address = 0.0.0.0 DHCP-Client-Hardware-Address = 90:e2:ba:86:3c:24 DHCP-Hostname = "test-optiplex" DHCP-Message-Type = DHCP-Discover DHCP-Parameter-Request-List = DHCP-Subnet-Mask DHCP-Parameter-Request-List = DHCP-Broadcast-Address DHCP-Parameter-Request-List = DHCP-Time-Offset DHCP-Parameter-Request-List = DHCP-Classless-Static-Route DHCP-Parameter-Request-List = DHCP-Router-Address DHCP-Parameter-Request-List = DHCP-Domain-Name DHCP-Parameter-Request-List = DHCP-Domain-Name-Server DHCP-Parameter-Request-List = DHCP-Hostname DHCP-Parameter-Request-List = DHCP-Domain-Search DHCP-Parameter-Request-List = DHCP-Interface-MTU-Size DHCP-Client-Identifier = 0x0190e2ba863c24 DHCP-Relay-Circuit-Id = 0x4c41423031206574682031372f31312f312f312f31 DHCP-Relay-Remote-Id = 0x6c616230315f6c66633030303030323233 DHCP-Network-Subnet = 0.0.0.0/32 (0) Received code 1025 Id 1753716481 from 0.0.0.0:68 to 255.255.255.255:67 length 548 (0) DHCP-Opcode = Client-Message (0) DHCP-Hardware-Type = Ethernet (0) DHCP-Hardware-Address-Length = 6 (0) DHCP-Hop-Count = 0 (0) DHCP-Transaction-Id = 1753716481 (0) DHCP-Number-of-Seconds = 0 (0) DHCP-Flags = 0 (0) DHCP-Client-IP-Address = 0.0.0.0 (0) DHCP-Your-IP-Address = 0.0.0.0 (0) DHCP-Server-IP-Address = 0.0.0.0 (0) DHCP-Gateway-IP-Address = 0.0.0.0 (0) DHCP-Client-Hardware-Address = 90:e2:ba:86:3c:24 (0) DHCP-Hostname = "test-optiplex" (0) DHCP-Message-Type = DHCP-Discover (0) DHCP-Parameter-Request-List = DHCP-Subnet-Mask (0) DHCP-Parameter-Request-List = DHCP-Broadcast-Address (0) DHCP-Parameter-Request-List = DHCP-Time-Offset (0) DHCP-Parameter-Request-List = DHCP-Classless-Static-Route (0) DHCP-Parameter-Request-List = DHCP-Router-Address (0) DHCP-Parameter-Request-List = DHCP-Domain-Name (0) DHCP-Parameter-Request-List = DHCP-Domain-Name-Server (0) DHCP-Parameter-Request-List = DHCP-Hostname (0) DHCP-Parameter-Request-List = DHCP-Domain-Search (0) DHCP-Parameter-Request-List = DHCP-Interface-MTU-Size (0) DHCP-Client-Identifier = 0x0190e2ba863c24 (0) DHCP-Relay-Circuit-Id = 0x4c41423031206574682031372f31312f312f312f31 (0) DHCP-Relay-Remote-Id = 0x6c616230315f6c66633030303030323233 (0) DHCP-Network-Subnet = 0.0.0.0/32 Trying sub-section dhcp DHCP-Discover {...} (0) dhcp DHCP-Discover { (0) update config { (0) &DHCP-Relay-To-IP-Address := 217.198.54.101 (0) &DHCP-Relay-To-Port := 1067 (0) } # update config = noop (0) update request { (0) &DHCP-Gateway-IP-Address := 217.198.52.2 (0) } # update request = noop (0) [ok] = ok (0) } # dhcp DHCP-Discover = ok DHCP-Opcode = Client-Message DHCP-Hardware-Type = Ethernet DHCP-Hardware-Address-Length = 6 DHCP-Hop-Count = 1 DHCP-Transaction-Id = 1753716481 DHCP-Number-of-Seconds = 0 DHCP-Flags = 0 DHCP-Client-IP-Address = 0.0.0.0 DHCP-Your-IP-Address = 0.0.0.0 DHCP-Server-IP-Address = 0.0.0.0 DHCP-Gateway-IP-Address = 217.198.52.2 DHCP-Client-Hardware-Address = 90:e2:ba:86:3c:24 DHCP-Server-Host-Name = "" DHCP-Boot-Filename = "" DHCP-Hostname = "test-optiplex" DHCP-Parameter-Request-List = DHCP-Subnet-Mask DHCP-Parameter-Request-List = DHCP-Broadcast-Address DHCP-Parameter-Request-List = DHCP-Time-Offset DHCP-Parameter-Request-List = DHCP-Classless-Static-Route DHCP-Parameter-Request-List = DHCP-Router-Address DHCP-Parameter-Request-List = DHCP-Domain-Name DHCP-Parameter-Request-List = DHCP-Domain-Name-Server DHCP-Parameter-Request-List = DHCP-Hostname DHCP-Parameter-Request-List = DHCP-Domain-Search DHCP-Parameter-Request-List = DHCP-Interface-MTU-Size DHCP-Client-Identifier = 0x0190e2ba863c24 DHCP-Relay-Circuit-Id = 0x4c41423031206574682031372f31312f312f312f31 DHCP-Relay-Remote-Id = 0x6c616230315f6c66633030303030323233 dhcprelay_process_client_request: paul: Here 1 Sending DHCP-Discover Id 68879701 from 0.0.0.0:67 to 217.198.54.101:1067 (0) Not sending reply to client. (0) Finished request (0) Cleaning up request packet ID 1753716481 with timestamp +5 due to done Ready to process requests Whilst this is going on, a tcpdump on vlan 4091 shows: # tcpdump -n -v -i vlan4091 -e -s 1500 tcpdump: listening on vlan4091, link-type EN10MB (Ethernet), capture size 1500 bytes 11:09:43.498585 90:e2:ba:86:3c:24 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 590: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 576) 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 90:e2:ba:86:3c:24, length 548, xid 0x7ca972e9, Flags [none] Client-Ethernet-Address 90:e2:ba:86:3c:24 Vendor-rfc1048 Extensions Magic Cookie 0x63825363 Hostname Option 12, length 13: "test-optiplex" DHCP-Message Option 53, length 1: Discover Parameter-Request Option 55, length 10: Subnet-Mask, BR, Time-Zone, Classless-Static-Route Default-Gateway, Domain-Name, Domain-Name-Server, Hostname Option 119, MTU Client-ID Option 61, length 7: ether 90:e2:ba:86:3c:24 Agent-Information Option 82, length 42: Circuit-ID SubOption 1, length 21: LAB01 eth 17/11/1/1/1 Remote-ID SubOption 2, length 17: lab01_lfc00000223 11:09:43.501284 04:3f:72:50:0a:94 > 02:aa:bb:cc:dd:ee, ethertype IPv4 (0x0800), length 367: (tos 0x0, ttl 64, id 16145, offset 0, flags [none], proto UDP (17), length 353) 217.198.52.2.67 > 217.198.54.101.1067: BOOTP/DHCP, Request from 90:e2:ba:86:3c:24, length 325, hops 1, xid 0x7ca972e9, Flags [none] Gateway-IP 217.198.52.2 Client-Ethernet-Address 90:e2:ba:86:3c:24 Vendor-rfc1048 Extensions Magic Cookie 0x63825363 DHCP-Message Option 53, length 1: Discover Hostname Option 12, length 13: "test-optiplex" Parameter-Request Option 55, length 10: Subnet-Mask, BR, Time-Zone, Classless-Static-Route Default-Gateway, Domain-Name, Domain-Name-Server, Hostname Option 119, MTU Client-ID Option 61, length 7: ether 90:e2:ba:86:3c:24 Agent-Information Option 82, length 42: Circuit-ID SubOption 1, length 21: LAB01 eth 17/11/1/1/1 Remote-ID SubOption 2, length 17: lab01_lfc00000223 ^C The MAC 02:aa:bb:cc:dd:ee comes from the static ARP entry on interface vlan4091 for 217.198.54.101
On Jul 7, 2022, at 7:24 AM, Paul Thornton <paul@prt.org> wrote:
This may be a Matthew specific question - I'm testing out the patch he kindly posted to me on uknof at the beginning of June to enable relaying to a different UDP port.
That shouldn't affect the selection of IP address tho.
For testing, I have a Cumulus box with two interfaces that we care about: vlan4091 which has a test client connected to it, and vlan545 which is the connection back to the rest of the network. Our test DHCP server (217.198.54.101) is in this "rest of the network" - IP routing all works as expected, and you can ping the DHCP server from the vlan4091 interface IP address (217.198.52.2) on the Cumulus router, and from clients on that VLAN - and vice versa.
When I send in a test DHCP request from a client connected to vlan4091, FR receives it and and re-sends it out to 217.198.54.101 on port 1067. However, the DHCP server never sees that packet.
After some rummaging with strace and tcpdump, I've determined that FR is transmitting the relayed packet out of vlan4091 (I saw suspect looking ARPs, and put in a static ARP entry after which I see the frame transmitted on vlan4091) and not following the default route up out of vlan545.
The DHCP socket generally binds to a particular interface. So that may be it.
Looking in proto_dhcp/dhcpd.c at dhcprelay_process_client_request, there's nothing in there to suggest that the outgoing interface is set and routing is overridden: But it looks like some parts are copied over from the socket handling the client communication; is there something here that could cause what I'm seeing?
If the src/dst IP/port are correct, then generally routing issues are the responsibility of the routing layer. Unless the socket is bound to a particular interface. In which case the packet *must* get sent out that interface.
My other thought that if this was a general problem it would break FR DHCP relaying in a lot of people's use cases - and there aren't lots of people grumbling about this, so there must be something wacky going on with my setup but I'm slightly at a loss to see what's happening. Any pointers greatfully received.
The current DHCP relay behavior is a bit of a hack, TBH. It should arguably have a different socket for outgoing packets. The current behaviour works in most situations, but sometimes doesn't. :(
Listening on dhcp interface vlan4091 address * port 67 bound to server dhcp.vlan4091
Yup. Packets using that socket are being sent via vlan4091, because that socket is bound to interface vlan4091. The solution is to create a new outbound socket, specifically for DHCP relaying. It can be bound to "*" for the IP, and then the normal DHCP port. There are a few ways to do this... one might be to simply create a "wildcard" DHCP listener, which doesn't have an "interface" configured. Then somehow update proto_dhcp/dhcpd.c to choose this listener for relayed packets. It's likely not hard, it just requires a bit of spelunking in the code. Alan DeKok.
Hi Alan, On 07/07/2022 12:51, Alan DeKok wrote:
Yup. Packets using that socket are being sent via vlan4091, because that socket is bound to interface vlan4091.
That makes perfect sense. Thanks for the steer, I did wonder if it was something like that. I presume that to listen for DHCP packets, we need to operate on a per-interface basis to be able to receive broadcasts; so operating with one socket - not bound to any specific interface - and listening on that isn't going to be an option.
The solution is to create a new outbound socket, specifically for DHCP relaying. It can be bound to "*" for the IP, and then the normal DHCP port.
There are a few ways to do this... one might be to simply create a "wildcard" DHCP listener, which doesn't have an "interface" configured. Then somehow update proto_dhcp/dhcpd.c to choose this listener for relayed packets.
It's likely not hard, it just requires a bit of spelunking in the code.
My C isn't up to fixing this properly but I'm going to attempt a quick hack proof-of-concept to see if it does work. Thanks, Paul.
On Jul 7, 2022, at 8:37 AM, Paul Thornton <paul@prt.org> wrote:
I presume that to listen for DHCP packets, we need to operate on a per-interface basis to be able to receive broadcasts; so operating with one socket - not bound to any specific interface - and listening on that isn't going to be an option.
Yes.
My C isn't up to fixing this properly but I'm going to attempt a quick hack proof-of-concept to see if it does work.
Great, thanks. Alan DeKok.
On 07/07/2022 13:43, Alan DeKok wrote:
On Jul 7, 2022, at 8:37 AM, Paul Thornton <paul@prt.org> wrote:
My C isn't up to fixing this properly but I'm going to attempt a quick hack proof-of-concept to see if it does work.
Great, thanks.
I did some basic (and ugly) work: Cloned fr_dhcp_send, added in a "create new socket, hard-code destination IP and port, and sendto() that socket instead" bit of code, and made dhcprelay_process_client_request call my modified version. This does solve my problem - the packet now follows the correct default route towards the upstream DHCP server :-). That then exposed another issue - more relating to the original patch to change the destination port relayed to. Of course, when I send to port 1067, the upstream DHCP server sends the reply on 1067 - which the relay isn't listening on. So the OFFER goes into the void. I think the proper fix for that is to ensure that the relay is listening on any alternative port; but my knowledge of the inner workings of the DHCP server is lacking to work out how to do that here (what is responsible for setting up the socket that dhcprelay_process_server_reply then handles data from?). As another slightly hacky fix, is there a way, on the upstream DHCP server (also FR), to force the destination port to 67 - aside from another blunt code hack - I couldn't see anything in the dictionary that would allow this. Paul.
On 07/07/2022 17:37, Paul Thornton wrote:
That then exposed another issue - more relating to the original patch to change the destination port relayed to. Of course, when I send to port 1067, the upstream DHCP server sends the reply on 1067 - which the relay isn't listening on. So the OFFER goes into the void.
Ah - I noticed this the first time, but wasn't sure if it actually mattered (any "normal" protocol would flip the src/dest ports, right? but we are talking DHCP here...)
I think the proper fix for that is to ensure that the relay is listening on any alternative port; but my knowledge of the inner workings of the
What happens if you just add a listen{} section on port 1067? -- Matthew
On 07/07/2022 17:43, Matthew Newton wrote:
On 07/07/2022 17:37, Paul Thornton wrote:
That then exposed another issue - more relating to the original patch to change the destination port relayed to. Of course, when I send to port 1067, the upstream DHCP server sends the reply on 1067 - which the relay isn't listening on. So the OFFER goes into the void.
Ah - I noticed this the first time, but wasn't sure if it actually mattered (any "normal" protocol would flip the src/dest ports, right? but we are talking DHCP here...)
I think the proper fix for that is to ensure that the relay is listening on any alternative port; but my knowledge of the inner workings of the
What happens if you just add a listen{} section on port 1067?
So close - but no cigar. The relay now gets the OFFER and passes it on - to the incorrect destination port. Sending DHCP-Offer Id 29d33634 from 0.0.0.0:1067 to 255.255.255.255:1067 Of course, another horrible hack might be "If I'm processing an OFFER or an ACK, and it comes in on the alternative port, please send it back to the originl client on port 68". Or is it meant to be 67. As you say, we're talking DHCP... Paul.
On Jul 7, 2022, at 1:07 PM, Paul Thornton <paul@prt.org> wrote:
So close - but no cigar. The relay now gets the OFFER and passes it on - to the incorrect destination port.
Sending DHCP-Offer Id 29d33634 from 0.0.0.0:1067 to 255.255.255.255:1067
Arguably the server *should* send packets to port 67, no matter what the source port.
Of course, another horrible hack might be "If I'm processing an OFFER or an ACK, and it comes in on the alternative port, please send it back to the originl client on port 68". Or is it meant to be 67. As you say, we're talking DHCP...
You can make it use port 67 by the following: update reply { Packet-Dst-Port := 67 } It will then send the packets to port 67 instead of 1067. We should arguably also support RFC 8357, which defines a "Remember Relay-Port" option. If the option exists, the reply should be sent to the UDP source port of the packet. FreeRADIUS has always done that, because hard-coding port 67 is stupid. Alan DEKok.
Hi Alan, Matthew (and anyone else following this), I've been doing some more digging with this over the past couple of days, but haven't had much success. I think I now understand the problem though. On 07/07/2022 20:26, Alan DeKok wrote:
You can make it use port 67 by the following:
update reply { Packet-Dst-Port := 67 }
It will then send the packets to port 67 instead of 1067.
Firstly, updating the reply port doesn't seem to make any difference. The debug shows that config being treated as a no-op: Received DHCP-Offer of Id 90779350 from 217.198.54.101:1067 to 217.198.52.2:1067 DHCP-Opcode = Server-Message DHCP-Hardware-Type = Ethernet DHCP-Hardware-Address-Length = 6 DHCP-Hop-Count = 1 DHCP-Transaction-Id = 2423755600 DHCP-Number-of-Seconds = 0 DHCP-Flags = 0 DHCP-Client-IP-Address = 0.0.0.0 DHCP-Your-IP-Address = 217.198.52.10 DHCP-Server-IP-Address = 0.0.0.0 DHCP-Gateway-IP-Address = 217.198.52.2 DHCP-Client-Hardware-Address = 90:e2:ba:86:3c:24 DHCP-Message-Type = DHCP-Offer DHCP-Subnet-Mask = 255.255.255.0 DHCP-Router-Address = 217.198.52.1 DHCP-Domain-Name-Server = 8.8.8.8 DHCP-Domain-Name-Server = 8.8.4.4 DHCP-Hostname = "lfc223" DHCP-IP-Address-Lease-Time = 1209823 DHCP-DHCP-Server-Identifier = 217.198.54.101 DHCP-Client-Identifier = 0x0190e2ba863c24 DHCP-Network-Subnet = 217.198.52.2/32 (4) Received code 1026 Id 2423755600 from 217.198.54.101:1067 to 217.198.52.2:1067 length 300 (4) DHCP-Opcode = Server-Message (4) DHCP-Hardware-Type = Ethernet (4) DHCP-Hardware-Address-Length = 6 (4) DHCP-Hop-Count = 1 (4) DHCP-Transaction-Id = 2423755600 (4) DHCP-Number-of-Seconds = 0 (4) DHCP-Flags = 0 (4) DHCP-Client-IP-Address = 0.0.0.0 (4) DHCP-Your-IP-Address = 217.198.52.10 (4) DHCP-Server-IP-Address = 0.0.0.0 (4) DHCP-Gateway-IP-Address = 217.198.52.2 (4) DHCP-Client-Hardware-Address = 90:e2:ba:86:3c:24 (4) DHCP-Message-Type = DHCP-Offer (4) DHCP-Subnet-Mask = 255.255.255.0 (4) DHCP-Router-Address = 217.198.52.1 (4) DHCP-Domain-Name-Server = 8.8.8.8 (4) DHCP-Domain-Name-Server = 8.8.4.4 (4) DHCP-Hostname = "lfc223" (4) DHCP-IP-Address-Lease-Time = 1209823 (4) DHCP-DHCP-Server-Identifier = 217.198.54.101 (4) DHCP-Client-Identifier = 0x0190e2ba863c24 (4) DHCP-Network-Subnet = 217.198.52.2/32 Trying sub-section dhcp DHCP-Offer {...} (4) dhcp DHCP-Offer { (4) update reply { (4) Packet-Dst-Port := 67 (4) } # update reply = noop (4) [ok] = ok (4) } # dhcp DHCP-Offer = ok (4) DHCP: response will be broadcast DHCP-Opcode = Server-Message DHCP-Hardware-Type = Ethernet DHCP-Hardware-Address-Length = 6 DHCP-Hop-Count = 1 DHCP-Transaction-Id = 2423755600 DHCP-Number-of-Seconds = 0 DHCP-Flags = 0 DHCP-Client-IP-Address = 0.0.0.0 DHCP-Your-IP-Address = 217.198.52.10 DHCP-Server-IP-Address = 0.0.0.0 DHCP-Gateway-IP-Address = 217.198.52.2 DHCP-Client-Hardware-Address = 90:e2:ba:86:3c:24 DHCP-Server-Host-Name = "" DHCP-Boot-Filename = "" DHCP-Subnet-Mask = 255.255.255.0 DHCP-Router-Address = 217.198.52.1 DHCP-Domain-Name-Server = 8.8.8.8 DHCP-Domain-Name-Server = 8.8.4.4 DHCP-Hostname = "lfc223" DHCP-IP-Address-Lease-Time = 1209823 DHCP-DHCP-Server-Identifier = 217.198.54.101 DHCP-Client-Identifier = 0x0190e2ba863c24 Sending DHCP-Offer Id 90779350 from 0.0.0.0:1067 to 255.255.255.255:1067 (4) Not sending reply to client. (4) Finished request (4) Cleaning up request packet ID 2423755600 with timestamp +89 due to done Ready to process requests The second part of the problem is that the offer that gets broadcasted on port 1067 goes out of the wrong interface - because the listen section for port 1067 doesn't have an interface configured by design as we're using it to receive packets from the upstream server via normal unicast routing. A quick tcpdump shows that broadcast going out of the default route interface (which is, I suppose, one stage better than the kernel just throwing it away with a shrug). My high level thoughts on "fixing" this is: If we want to DHCP relay to another port then there needs to be a listener for that other port. If we are relaying to a server not on the same LAN (likely) then this port cannot be bound to a particular interface. In fr_dhcp_send (or its callers) there needs to be a way to switch to this listener's socket to relay DISCOVER and REQUEST towards the server; and there needs to be a way to switch from this socket back to the original interface-bound socket to relay OFFER and ACK/NAK towards the client. Does this sound logical or have I missed something glaringly obvious? I've already tested the upstream with a quick hack to just create a new socket and send the packet. The downstream replies need some state saving whilst relaying DISCOVER or REQUEST to know what socket it was received on, so that when relaying the corresponding OFFER or ACK/NAK it can use that socket to transmit on. Any clues as to how I can go about testing the second half of this - any pointers to how I should find out the socket to send on etc.?
We should arguably also support RFC 8357, which defines a "Remember Relay-Port" option. If the option exists, the reply should be sent to the UDP source port of the packet.
FreeRADIUS has always done that, because hard-coding port 67 is stupid.
Indeed. Paul.
On Jul 10, 2022, at 6:16 AM, Paul Thornton <paul@prt.org> wrote:ackets to port 67 instead of 1067
Firstly, updating the reply port doesn't seem to make any difference. The debug shows that config being treated as a no-op:
OK. The rest of the server respects that, so that's likely an easy fix.
The second part of the problem is that the offer that gets broadcasted on port 1067 goes out of the wrong interface - because the listen section for port 1067 doesn't have an interface configured by design as we're using it to receive packets from the upstream server via normal unicast routing. A quick tcpdump shows that broadcast going out of the default route interface (which is, I suppose, one stage better than the kernel just throwing it away with a shrug).
My high level thoughts on "fixing" this is:
If we want to DHCP relay to another port then there needs to be a listener for that other port. If we are relaying to a server not on the same LAN (likely) then this port cannot be bound to a particular interface.
Yes.
In fr_dhcp_send (or its callers) there needs to be a way to switch to this listener's socket to relay DISCOVER and REQUEST towards the server;
Yes.
and there needs to be a way to switch from this socket back to the original interface-bound socket to relay OFFER and ACK/NAK towards the client.
Well, there needs to be a way for packets received on that socket to find the original socket to use.
Does this sound logical or have I missed something glaringly obvious?
It seems OK.
I've already tested the upstream with a quick hack to just create a new socket and send the packet. The downstream replies need some state saving whilst relaying DISCOVER or REQUEST to know what socket it was received on, so that when relaying the corresponding OFFER or ACK/NAK it can use that socket to transmit on.
You don't need to create the new socket on every packet. Just create it when the server starts, and leave it open.
Any clues as to how I can go about testing the second half of this - any pointers to how I should find out the socket to send on etc.?
It will involve looking at yiaddr from the received packet, and then using that to find the matching socket. i.e. "which socket is listening on a network that matches this IP". That's a bit more involved. listener_find_byipaddr() does the basic work, but there may be multiple subnets on the same network. That will require a bit more work. Alan DeKok.
participants (3)
-
Alan DeKok -
Matthew Newton -
Paul Thornton