Docker host sends reply via wrong interface when macvlan and custom bridge are used

General Discussions
,
1

Setup

I am a big fan of the docker network type macvlan. This is the only network I am using on my docker host to run containers and give them an IP in different subnets (vlans). The docker host is running inside a proxmox LXC with 2 virtual interfaces, eth0 and eth1.

eth0 is untagged 100 (the management subnet) and has a static IP assigned to it, so I can SSH to it.
eth1 has access to all the VLANs.

image
image1413×244 37.6 KB

The problem

Till now this setup works really great. 1 docker container gets his own IP and is reachable in the subnet I have put it in.
However, I recently stumbled on a piece of software that requires multiple containers to run. I thought let’s put those in a custom bridge docker network for security reasons AND only connect the nginx container (part of that software) to a VLAN as well, with the use of the macvlan network type. This didn’t work.

As it turns out (see where it goes wrong below) the docker host sends the pong reply NOT over the interface where the request came from. Why is this ???

Steps to reproduce

I have simplified my setup to 1 container. This setup results in the same problem.

  1. created 2 networks. (taiga is the custom docker bridge network)
docker network create taiga

docker network create -d macvlan --subnet 192.168.205.0/24 --gateway 192.168.205.1 -o parent=eth1.205 vlan205_exposed
  1. content of the docker compose file
version: "3.6"
services:
  tcpdump:
    image: corfr/tcpdump
    networks:
      vlan205_exposed:
        ipv4_address: 192.168.205.40
      taiga:

networks:
  vlan205_exposed:
    external: true
  taiga:
    external: true
  1. bring the container up with
docker-compose -f docker-compose-test.yml up -d
  1. when I now use the tcpdump command on the docker LXC you can see the request arrives at eth1 and the reply goes out over eth0. This wrong.
root@DCKR:/opt/test_stack# tcpdump -n -i any icmp
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
12:52:29.158351 eth1  P   IP 192.168.101.155 > 192.168.205.40: ICMP echo request, id 7, seq 45316, length 64
12:52:29.158351 eth1.205 P   IP 192.168.101.155 > 192.168.205.40: ICMP echo request, id 7, seq 45316, length 64
12:52:29.158370 vethec5ee45 P   IP 192.168.205.40 > 192.168.101.155: ICMP echo reply, id 7, seq 45316, length 64
12:52:29.158370 br-872e0fea5a9e In  IP 192.168.205.40 > 192.168.101.155: ICMP echo reply, id 7, seq 45316, length 64
12:52:29.158383 eth0  Out IP 192.168.205.40 > 192.168.101.155: ICMP echo reply, id 7, seq 45316, length 64
  1. when I comment out the custom bridge network taiga in the docker compose file, everything works as expected:
    ...
    image: corfr/tcpdump
    networks:
      vlan205_exposed:
        ipv4_address: 192.168.205.40
      #taiga:
...

root@DCKR:/opt/test_stack# tcpdump -n -i any icmp
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
12:51:34.041502 eth1  P   IP 192.168.101.155 > 192.168.205.40: ICMP echo request, id 7, seq 45261, length 64
12:51:34.041502 eth1.205 P   IP 192.168.101.155 > 192.168.205.40: ICMP echo request, id 7, seq 45261, length 64
12:51:34.041522 eth1.205 Out IP 192.168.205.40 > 192.168.101.155: ICMP echo reply, id 7, seq 45261, length 64
12:51:34.041522 eth1  Out IP 192.168.205.40 > 192.168.101.155: ICMP echo reply, id 7, seq 45261, length 64