Trying to get the IPVLAN L3 mode driver working in my test Docker environment. Striking out pretty hard, and I have no idea why. (appolgies, YAML seems to be messing w/the reddit editor)
TL;DR: My Ubuntu docker host has known-good working DNS configuration, and if I do an nslookup (without specifying a server) inside a bridge network container it works fine, but if I do the same thing from within a container configured with the IPVLAN L3 driver the DNS query fails, but all other network traffic inside that container works fine and itās reachable/routable from other subnets.
Environment info:
Ubuntu 22.04 inside a ESXi 7 VM
Docker Host IP: 192.168.200.12
My internal DNS server IP: 192.168.100.2
IPVLAN L3 subnet: 192.168.201.0/24
Docker v25.0.5
Portainer v2.19.4 (I'm using Stacks inside of Portainer for container builds)
network:
ethernets:
ens160:
addresses:
- 192.168.200.12/24
nameservers:
addresses:
- 192.168.100.2
- 2.2.2.2
search:
- home.myfqdn.com
routes:
- to: default
via: 192.168.200.1
version: 2
I created an IPVLAN L3 network on the Docker host:
docker network create -d IPVLAN_L3_201 \
--subnet=192.168.201.0/24 \
-o ipvlan_mode=l3 ens160
I have a Ubiquiti UDM-Pro w/a static route for 192.168.201.0/24 pointing at 192.168.200.12:
If I SSH to the docker host and test networking, specifically NSLOOKUP, everything works.
Docker Host test results - Ping, nslookup against system default, nslookup w/manualy specified DNS server
areaman@dockerhost:~$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=52.7 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=57 time=43.2 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=57 time=38.6 ms
^C
--- 8.8.8.8 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 38.635/44.828/52.666/5.844 ms
areaman@dockerhost:~$ nslookup google.com Server:
127.0.0.53 Address: 127.0.0.53#53
Non-authoritative answer:
Name: google.com Address: 142.250.191.110
Name: google.com Address: 2607:f8b0:4009:803::200e
areaman@dockerhost~$ nslookup google.com 192.168.100.2
Server: 192.168.100.2
Address: 192.168.100.2#53
Non-authoritative answer:
Name: google.com Address: 142.250.191.110
Name: google.com Address: 2607:f8b0:4009:803::200e
No issues here. I see both DNS queries hitting my DNS server w/o issue. DNS works as Iād expect.
So, now letās go create a container using the IPVLAN L3 driver.
nginx container YAML:
version: "3"
services:
nginx_test:
container_name: nginx_test
image: linuxserver/nginx:latest
networks:
IPVLAN_L3_201:
ipv4_address: 192.168.201.118
networks:
IPVLAN_L3_201:
external: true
Container starts up just fine, and if I hop inside of it to run the same tests as we did against the host:
Ping, nslookup against system default, nslookup w/manualy specified DNS server
root@20e97f08c7db:/# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8):
56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=113 time=39.784 ms
64 bytes from 8.8.8.8: seq=1 ttl=113 time=42.886 ms
64 bytes from 8.8.8.8: seq=2 ttl=113 time=44.664 ms
^C
--- 8.8.8.8 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 39.784/42.444/44.664 ms
root@20e97f08c7db:/# nslookup google.com
Server: 127.0.0.11
Address: 127.0.0.11:53
** server can't find google.com: SERVFAIL
** server can't find google.com: SERVFAIL
root@20e97f08c7db:/# nslookup google.com 192.168.100.2
Server: 192.168.100.2
Address: 192.168.100.2:53
Non-authoritative answer: Name: google.com Address: 142.250.191.110
Non-authoritative answer: Name: google.com Address: 2607:f8b0:4009:803::200e
You can see SERVFAIL messages when the nslookup command is run against the system default DNS. This, to me at least, represents the driver failing to redirect the DNS query to the system DNS. Is that correct?
Also, to make sure I wasnāt crazy, I tested pinging into the container from another subnet that required routing to reach the docker VM (192.168.100.0/24 subnet). This works fine, I get a reply like Iād expect.
So, to see if this was specific to the ipvlan l3 driver mode, I created another container using the same image but using the default bridge networking:
version: "3"
services:
nginx_test:
container_name: nginx_bridge
image: linuxserver/nginx:latest
This container also starts up just fine, and if I hop inside of it to run the same tests we did inside the IPVLAN L3 dirver container (previous tests) it works fine.
Once again weāre testing by doing a ping to 8.8.8.8, an nslookup against system default, nslookup w/manualy specified DNS server
root@90b63fdd7260:/# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=56 time=43.751 ms
64 bytes from 8.8.8.8: seq=1 ttl=56 time=42.334 ms
64 bytes from 8.8.8.8: seq=2 ttl=56 time=39.053 ms
^C
--- 8.8.8.8 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 39.053/41.712/43.751 ms
root@90b63fdd7260:/# nslookup google.com*
Server: 127.0.0.11
Address: 127.0.0.11:53
Non-authoritative answer:
Name: google.com
Address: 142.250.191.110
Non-authoritative answer:
Name: google.com
Address: 2607:f8b0:4009:80b::200e
root@90b63fdd7260:/# nslookup google.com 192.168.100.2
Server: 192.168.100.2
Address: 192.168.100.2:53
Non-authoritative answer:
Name: google.com
Address: 142.250.191.110
Non-authoritative answer:
Name: google.com
Address: 2607:f8b0:4009:80b::200e
So, as expected, you can see the non-authoritative answer coming back from my internal DNS server. I can also see that query hit my internal DNS, so I know itās not using some other DNS server on the internet. The ping and external nslookup (with a specified server) also work fine.
Iām baffled. I didnāt see anything in the documentation for the IPVLAN L3 driver that explains this or hints at something I missed configuring.
What am I missing here? I know the upstream networking/routing/dns is fine.
I really donāt want to abandon the IPVLAN L3 driver (unless we know of a bug or something). I have other things going on in my network that make each container being route-able hugely useful and dramatically simplified some other aspects of my lab.
Any insight is appreciated, thanks!