MacVLAN swarm arp response not received from container to container on different hosts

I’m setting up a macvlan network in docker swarm, but when I attempt to ping a container from another container not in the same host, the ARP reply is sent, but it is not received on the other container.

I have two hosts, worker1 and worker2, which is provisioned using Vultr, I’m using the private network interface (ens7) to communicate between the hosts on the subnet There is no gateway set up.

Worker 1 private ip (ens7):
Worker 2 private ip (ens7):

Ping from worker 1 to worker 2 and vice-versa works.

On worker 1, I run docker network create --config-only --subnet -o parent=ens7 --ip-range macvlan
On worker2, I run docker network create --config-only --subnet -o parent=ens7 --ip-range macvlan

On a manager node, I docker network create --config-from macvlan --scope swarm --driver macvlan mynet

I run dummy containers on worker 1 and 2, attaching it to the mynet macvlan network via a docker stack deploy with the mynet entry set as external: true

On worker 1, the container gets an ip address
On worker 2, the container gets an ip address

On worker 1, pinging the address does not work (this is expected). Likewise, on worker 2, pinging the address does not work (again this is expected).
However, entering the container via docker exec -it CONTAINER_NAME bash on worker1 and pinging (other container on worker 2) does not work. Likewise pinging (container on worker 1) from the container on worker 2 does not work.
Also, on worker 1, pinging (container on worker 2) does not work (I assume this should work?), and similarly, on worker 2, pinging (container on worker 1) does not work.

If I run tcpdump -n -i ens7 on worker 1 and worker 2, and from inside the container on worker 1, I ping, I notice on worker 1 the ARP request is sent Request who-has tell and on worker 2, it sees the same ARP request, and makes a reply with Reply is at MAC_ADDRESS, but the ARP response is not being received back on worker 1 to the container.

Now, what’s weird is that I tried IPVLAN in mode l2 with the same exact setup (same subnets, no gateway, etc), and I am able to ping from container to container. The only difference I can find between macvlan and ipvlan is that ipvlan does not assign a unique mac address to each container, whereas macvlan does. In other words, using ipvlan uses the same mac address as the host’s mac address (which is specified by the parent option).

I also attempted to disable the firewall on ubuntu 16.04, with ufw disable on both hosts, but it still doesn’t work.

Can you pls check if promiscuous mode is set on ens7 on both hosts. I have seen that unless promiscuous mode is set, macvlan communication across host does not work.

I think you’re right, I’ve been reading that macvlan requires promiscuous mode to be enabled. Unfortunately, despite trying to enable it manually with ip link set dev ens7 promisc on, it still doesn’t work. I think this has to do with Vultr not supporting promiscuous mode on their private network interfaces.

Thanks for the help though.