UDP IPv6 return traffic coming from different IP address

Docker Engine General
1

I’ve found an odd behavior with an UDP ports over IPv6 in docker. I’m not entirely sure where to start, so I’ll share my observations, a test setup to recreate the issue.

Any help is appreciated, thank you.

  • I’m running a pihole image, but I believe that this issue is likely not specific to this image.
  • The docker host has multiple IPv6 addresses. This is very common in IPv6. One static well-known, but also a privacy address.
  • When observing with tcpdump on the host NIC, I see:
    • DNS queries (being served by the container) show src=Laptop, dst=Address1.
    • DNS responses (being generated by the container) show src=Address2, dst:Laptop.
    • The reply source does not match the request destination.
  • Using host networking fixes the problem.

My expectation is that DNS responses would come back from the address where the query originated. I suspect something is going on with iptables/nftables where the session state isn’t getting tracked quite right.

Currently my work-around is to use host networking, which is OK since this device is single-purpose.

Is anyone else experiencing this? Is this a bug? has it been reported / where would I report such a thing?

Any other information I need to provide.

The closest thing I’ve found:

Test Setup

docker-compose.yml:

version: "3"

services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    #network_mode: host
    ports:
      - "53:53/tcp"
      - "53:53/udp"
      - "67:67/udp"
      - "80:80/tcp"
      - "443:443/tcp"
    environment:
      TZ: 'UTC'
    volumes:
       - './etc-pihole/:/etc/pihole/'
       - './etc-dnsmasq.d/:/etc/dnsmasq.d/'
    dns:
      - 127.0.0.1
      - 1.1.1.1
      - 8.8.8.8
#    cap_add:
#      - NET_ADMIN
    restart: unless-stopped

Testing Commands

Docker Host

cat /etc/debian_version
uname -a
docker --version
cat /etc/network/interfaces
docker-compose up -d
ip -6 addr
tcpdump -ni eth0 -s0 -w pihole.pcap 'udp port 53 and ip6'
[CTRL-C after test is done]
tcpdump -rn pihole.pcap

Laptop:

dig google.com @REDACTED_PREFIX::2

Output

Docker Host

root@pihole:~/pihole# cat /etc/debian_version
10.11

root@pihole:~/pihole# uname -a
Linux pihole 5.4.157-1-pve #1 SMP PVE 5.4.157-1 (Mon, 29 Nov 2021 12:01:44 +0100) x86_64 GNU/Linux

root@pihole:~/pihole# docker --version
Docker version 20.10.12, build e91ed57

root@pihole:~/pihole# cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
	address 192.168.42.2/24
	gateway 192.168.42.1

iface eth0 inet6 static
	address REDACTED_PREFIX::2/64

	autoconf 1
	accept_ra 2

root@pihole:~/pihole# docker-compose up -d
Recreating pihole ... done

root@pihole:~/pihole# ip -6 addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 REDACTED_PREFIX:e44d:17ff:fee2:fe14/64 scope global dynamic mngtmpaddr
       valid_lft 4294967218sec preferred_lft 4294967218sec
    inet6 REDACTED_PREFIX::2/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::e44d:17ff:fee2:fe14/64 scope link
       valid_lft forever preferred_lft forever
3: br-1b31b053eca8: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 state DOWN
    inet6 fe80::42:77ff:fe99:a4cf/64 scope link
       valid_lft forever preferred_lft forever

root@pihole:~/pihole# tcpdump -nr pihole.pcap
14:06:30.193288 IP6 REDACTED_PREFIX:f1fb:58f:633f:beed.61003 > REDACTED_PREFIX::2.53: 57950+ [1au] A? google.com. (39)
14:06:30.193680 IP6 REDACTED_PREFIX:e44d:17ff:fee2:fe14.53 > REDACTED_PREFIX:f1fb:58f:633f:beed.61003: 57950 1/0/1 A 142.251.32.14 (55)
14:06:35.208945 IP6 REDACTED_PREFIX:f1fb:58f:633f:beed.61003 > REDACTED_PREFIX::2.53: 57950+ [1au] A? google.com. (39)
14:06:35.209213 IP6 REDACTED_PREFIX:e44d:17ff:fee2:fe14.53 > REDACTED_PREFIX:f1fb:58f:633f:beed.61003: 57950 1/0/1 A 142.251.32.14 (55)
14:06:40.196996 IP6 REDACTED_PREFIX:f1fb:58f:633f:beed.61003 > REDACTED_PREFIX::2.53: 57950+ [1au] A? google.com. (39)
14:06:40.197190 IP6 REDACTED_PREFIX:e44d:17ff:fee2:fe14.53 > REDACTED_PREFIX:f1fb:58f:633f:beed.61003: 57950 1/0/1 A 142.251.32.14 (55)
14:06:48.995986 IP6 REDACTED_PREFIX:f1fb:58f:633f:beed.60422 > REDACTED_PREFIX::2.53: 10923+ [1au] AAAA? google.com. (39)
14:06:48.996965 IP6 REDACTED_PREFIX:e44d:17ff:fee2:fe14.53 > REDACTED_PREFIX:f1fb:58f:633f:beed.60422: 10923 1/0/1 AAAA 2607:f8b0:4009:817::200e (67)
14:06:53.999137 IP6 REDACTED_PREFIX:f1fb:58f:633f:beed.60422 > REDACTED_PREFIX::2.53: 10923+ [1au] AAAA? google.com. (39)
14:06:53.999412 IP6 REDACTED_PREFIX:e44d:17ff:fee2:fe14.53 > REDACTED_PREFIX:f1fb:58f:633f:beed.60422: 10923 1/0/1 AAAA 2607:f8b0:4009:817::200e (67)
14:06:58.996236 IP6 REDACTED_PREFIX:f1fb:58f:633f:beed.60422 > REDACTED_PREFIX::2.53: 10923+ [1au] AAAA? google.com. (39)
14:06:58.996510 IP6 REDACTED_PREFIX:e44d:17ff:fee2:fe14.53 > REDACTED_PREFIX:f1fb:58f:633f:beed.60422: 10923 1/0/1 AAAA 2607:f8b0:4009:817::200e (67)
14:06:59.519148 IP6 REDACTED_PREFIX:f1fb:58f:633f:beed.57473 > REDACTED_PREFIX::2.53: 9811+ A? fonts.gstatic.com. (35)
14:06:59.519172 IP6 REDACTED_PREFIX:f1fb:58f:633f:beed.60230 > REDACTED_PREFIX::2.53: 63866+ AAAA? fonts.gstatic.com. (35)
14:06:59.542746 IP6 REDACTED_PREFIX:e44d:17ff:fee2:fe14.53 > REDACTED_PREFIX:f1fb:58f:633f:beed.57473: 9811 2/0/0 CNAME gstaticadssl.l.google.com., A 172.217.4.35 (87)
14:06:59.544465 IP6 REDACTED_PREFIX:e44d:17ff:fee2:fe14.53 > REDACTED_PREFIX:f1fb:58f:633f:beed.60230: 63866 2/0/0 CNAME gstaticadssl.l.google.com., AAAA 2607:f8b0:4009:806::2003 (99)
14:07:00.084832 IP6 REDACTED_PREFIX:f1fb:58f:633f:beed.59014 > REDACTED_PREFIX::2.53: 27305+ A? www.gstatic.com. (33)
14:07:00.085266 IP6 REDACTED_PREFIX:f1fb:58f:633f:beed.57307 > REDACTED_PREFIX::2.53: 33187+ AAAA? www.gstatic.com. (33)
14:07:00.091946 IP6 REDACTED_PREFIX:f1fb:58f:633f:beed.57937 > REDACTED_PREFIX::2.53: 42501+ A? lh3.googleusercontent.com. (43)
14:07:00.092277 IP6 REDACTED_PREFIX:f1fb:58f:633f:beed.57961 > REDACTED_PREFIX::2.53: 39273+ AAAA? lh3.googleusercontent.com. (43)
14:07:00.104260 IP6 REDACTED_PREFIX:e44d:17ff:fee2:fe14.53 > REDACTED_PREFIX:f1fb:58f:633f:beed.57937: 42501 2/0/0 CNAME googlehosted.l.googleusercontent.com., A 142.250.191.225 (88)
14:07:00.105042 IP6 REDACTED_PREFIX:e44d:17ff:fee2:fe14.53 > REDACTED_PREFIX:f1fb:58f:633f:beed.57961: 39273 2/0/0 CNAME googlehosted.l.googleusercontent.com., AAAA 2607:f8b0:4009:808::2001 (100)
14:07:00.107953 IP6 REDACTED_PREFIX:e44d:17ff:fee2:fe14.53 > REDACTED_PREFIX:f1fb:58f:633f:beed.59014: 27305 1/0/0 A 142.250.190.99 (49)
14:07:00.107973 IP6 REDACTED_PREFIX:e44d:17ff:fee2:fe14.53 > REDACTED_PREFIX:f1fb:58f:633f:beed.57307: 33187 1/0/0 AAAA 2607:f8b0:4009:819::2003 (61)
14:07:00.512608 IP6 REDACTED_PREFIX:f1fb:58f:633f:beed.57473 > REDACTED_PREFIX::2.53: 9811+ A? fonts.gstatic.com. (35)
14:07:00.512839 IP6 REDACTED_PREFIX:e44d:17ff:fee2:fe14.53 > REDACTED_PREFIX:f1fb:58f:633f:beed.57473: 9811 2/0/0 CNAME gstaticadssl.l.google.com., A 172.217.4.35 (90)
14:07:00.546750 IP6 REDACTED_PREFIX:f1fb:58f:633f:beed.60230 > REDACTED_PREFIX::2.53: 63866+ AAAA? fonts.gstatic.com. (35)
14:07:00.546950 IP6 REDACTED_PREFIX:e44d:17ff:fee2:fe14.53 > REDACTED_PREFIX:f1fb:58f:633f:beed.60230: 63866 2/0/0 CNAME gstaticadssl.l.google.com., AAAA 2607:f8b0:4009:806::2003 (102)
14:07:01.135683 IP6 REDACTED_PREFIX:f1fb:58f:633f:beed.59014 > REDACTED_PREFIX::2.53: 27305+ A? www.gstatic.com. (33)
14:07:01.135685 IP6 REDACTED_PREFIX:f1fb:58f:633f:beed.57307 > REDACTED_PREFIX::2.53: 33187+ AAAA? www.gstatic.com. (33)
14:07:01.135918 IP6 REDACTED_PREFIX:f1fb:58f:633f:beed.57937 > REDACTED_PREFIX::2.53: 42501+ A? lh3.googleusercontent.com. (43)
14:07:01.135919 IP6 REDACTED_PREFIX:f1fb:58f:633f:beed.57961 > REDACTED_PREFIX::2.53: 39273+ AAAA? lh3.googleusercontent.com. (43)
14:07:01.136016 IP6 REDACTED_PREFIX:e44d:17ff:fee2:fe14.53 > REDACTED_PREFIX:f1fb:58f:633f:beed.57307: 33187 1/0/0 AAAA 2607:f8b0:4009:819::2003 (61)
14:07:01.136036 IP6 REDACTED_PREFIX:e44d:17ff:fee2:fe14.53 > REDACTED_PREFIX:f1fb:58f:633f:beed.59014: 27305 1/0/0 A 142.250.190.99 (49)
14:07:01.136060 IP6 REDACTED_PREFIX:e44d:17ff:fee2:fe14.53 > REDACTED_PREFIX:f1fb:58f:633f:beed.57937: 42501 2/0/0 CNAME googlehosted.l.googleusercontent.com., A 142.250.191.225 (109)
14:07:01.136099 IP6 REDACTED_PREFIX:e44d:17ff:fee2:fe14.53 > REDACTED_PREFIX:f1fb:58f:633f:beed.57961: 39273 2/0/0 CNAME googlehosted.l.googleusercontent.com., AAAA 2607:f8b0:4009:808::2001 (121)

Laptop:

% dig google.com @REDACTED_PREFIX::2

; <<>> DiG 9.10.6 <<>> google.com @REDACTED_PREFIX::2
;; global options: +cmd
;; connection timed out; no servers could be reached