I’m setting up a macvlan network in docker swarm, but when I attempt to ping a container from another container not in the same host, the ARP reply is sent, but it is not received on the other container.
I have two hosts, worker1 and worker2, which is provisioned using Vultr, I’m using the private network interface (ens7) to communicate between the hosts on the subnet 10.99.0.0/16. There is no gateway set up.
Worker 1 private ip (ens7): 10.99.0.11
Worker 2 private ip (ens7): 10.99.0.12
Ping from worker 1 to worker 2 and vice-versa works.
On worker 1, I run docker network create --config-only --subnet 10.99.0.0/16 -o parent=ens7 --ip-range 10.99.1.30/24 macvlan
On worker2, I run docker network create --config-only --subnet 10.99.0.0/16 -o parent=ens7 --ip-range 10.99.2.30/24 macvlan
On a manager node, I docker network create --config-from macvlan --scope swarm --driver macvlan mynet
I run dummy containers on worker 1 and 2, attaching it to the mynet macvlan network via a docker stack deploy with the mynet entry set as external: true
On worker 1, the container gets an ip address 10.99.1.1.
On worker 2, the container gets an ip address 10.99.2.1
On worker 1, pinging the address 10.99.1.1 does not work (this is expected). Likewise, on worker 2, pinging the address 10.99.2.1 does not work (again this is expected).
However, entering the container via docker exec -it CONTAINER_NAME bash on worker1 and pinging 10.99.2.1 (other container on worker 2) does not work. Likewise pinging 10.99.1.1 (container on worker 1) from the container on worker 2 does not work.
Also, on worker 1, pinging 10.99.2.1 (container on worker 2) does not work (I assume this should work?), and similarly, on worker 2, pinging 10.99.1.1 (container on worker 1) does not work.
If I run tcpdump -n -i ens7 on worker 1 and worker 2, and from inside the container on worker 1, I ping 10.99.2.1, I notice on worker 1 the ARP request is sent Request who-has 10.99.2.1 tell 10.99.1.1 and on worker 2, it sees the same ARP request, and makes a reply with Reply 10.99.2.1 is at MAC_ADDRESS, but the ARP response is not being received back on worker 1 to the container.
Now, what’s weird is that I tried IPVLAN in mode l2 with the same exact setup (same subnets, no gateway, etc), and I am able to ping from container to container. The only difference I can find between macvlan and ipvlan is that ipvlan does not assign a unique mac address to each container, whereas macvlan does. In other words, using ipvlan uses the same mac address as the host’s mac address (which is specified by the parent option).
I also attempted to disable the firewall on ubuntu 16.04, with ufw disable on both hosts, but it still doesn’t work.