Help with macvlan network + vpn container where i cannot access service from outside the container

General Discussions
1

Here is my setup. I have a macvlan network configured. There are 4 other containers that i am able to access without any issues using the same network. I am trying to setup a gluetun with wireguard and I am not able to access the services running inside the container from outside of the container.

Here is my docker compose

gluetun:
    container_name: gluetun
    image: qmcgaw/gluetun
    cap_add:
      - NET_ADMIN
    devices:
      - "/dev/net/tun:/dev/net/tun"
    restart: unless-stopped
    volumes:
      - /volume1/docker/gluetun:/gluetun
    environment:
      TZ: 'America/Los_Angeles'
      VPNSP: 'mullvad'
      VPN_TYPE: 'wireguard'
      WIREGUARD_PRIVATE_KEY: 'xxx'
      WIREGUARD_ADDRESS: 'xxx'
      CITY: 'Gothenburg'
      OWNED: 'yes'
      FIREWALL_OUTBOUND_SUBNETS: '192.168.1.0/24'
      FIREWALL_DEBUG: 'on'
      LOG_LEVEL: 'debug'
    networks:
      macvlan:
        ipv4_address: 192.168.1.244
    ports:
      - "6767:6767/tcp"

I have confirmed that the container that is setup to use the above service using network_mode: 'service:gluetun' is indeed connected to the vpn. I can also issue a curl call to 192.168.1.244:6767 from within the gluetun container or the container using the gluetun network and it returns data.

However I am unable to access 192.168.1.244:6767 from outside the container. I am not sure if its an iptables/firewall issue or if something is wrong with my macvlan setup or somethign else. As i have other 4 containers working I am leaning towards something specific going on with the gluetun docker image. Any one have pointers on what could be the issue?

Here is the config of the macvlan network.

"Name": "macvlan",
        "Id": "a32efea998a2b2fecb46783d1b2e1805f37d5802d5f44559af887934cf8e3908",
        "Created": "2020-04-24T17:49:58.888650035-07:00",
        "Scope": "local",
        "Driver": "macvlan",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "192.168.1.0/24",
                    "IPRange": "192.168.1.240/28",
                    "Gateway": "192.168.1.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },

iptables info from within the gluetun container

docker exec -it gluetun sh -c 'iptables -nvL'
Chain INPUT (policy DROP 10 packets, 625 bytes)
 pkts bytes target     prot opt in     out     source               destination
  183 16085 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
12175   16M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    2   148 ACCEPT     all  --  eth0   *       0.0.0.0/0            192.168.1.0/24
    0     0 ACCEPT     tcp  --  wg0    *       0.0.0.0/0            0.0.0.0/0            tcp dpt:58247
    0     0 ACCEPT     udp  --  wg0    *       0.0.0.0/0            0.0.0.0/0            udp dpt:58247
    0     0 ACCEPT     tcp  --  wg0    *       0.0.0.0/0            0.0.0.0/0            tcp dpt:54963
    0     0 ACCEPT     udp  --  wg0    *       0.0.0.0/0            0.0.0.0/0            udp dpt:54963

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
  183 16085 ACCEPT     all  --  *      lo      0.0.0.0/0            0.0.0.0/0
 9719 1002K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
   11   652 ACCEPT     all  --  *      eth0    192.168.1.244        192.168.1.0/24
    0     0 ACCEPT     all  --  *      eth0    192.168.1.244        192.168.1.0/24
    1   176 ACCEPT     udp  --  *      eth0    0.0.0.0/0            185.213.154.68       udp dpt:51820
   30  1824 ACCEPT     all  --  *      wg0     0.0.0.0/0            0.0.0.0/0

IP route info from within the container

docker exec -it gluetun sh -c 'ip route show'
default via 192.168.1.1 dev eth0
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.244

iptables info on the docker host

sudo iptables -nvL
Chain INPUT (policy ACCEPT 8383K packets, 3661M bytes)
 pkts bytes target     prot opt in     out     source               destination         
  69M   23G DOS_PROTECT  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain FORWARD (policy ACCEPT 82M packets, 35G bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 9677K packets, 25G bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp  --  !docker-f7e98769 docker-f7e98769  0.0.0.0/0            172.18.0.3           tcp dpt:5000
    0     0 ACCEPT     tcp  --  !docker-f7e98769 docker-f7e98769  0.0.0.0/0            172.18.0.3           tcp dpt:1935
    0     0 ACCEPT     tcp  --  !docker-f7e98769 docker-f7e98769  0.0.0.0/0            172.18.0.2           tcp dpt:9001
    0     0 ACCEPT     tcp  --  !docker-f7e98769 docker-f7e98769  0.0.0.0/0            172.18.0.2           tcp dpt:1883
    0     0 ACCEPT     tcp  --  !docker-b3c5b777 docker-b3c5b777  0.0.0.0/0            172.21.0.2           tcp dpt:8091
    0     0 ACCEPT     tcp  --  !docker-b3c5b777 docker-b3c5b777  0.0.0.0/0            172.21.0.2           tcp dpt:3000

Chain DOCKER-ISOLATION-STAGE-1 (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 DOCKER-ISOLATION-STAGE-2  all  --  docker-b3c5b777 !docker-b3c5b777  0.0.0.0/0            0.0.0.0/0           
    0     0 DOCKER-ISOLATION-STAGE-2  all  --  docker-f7e98769 !docker-f7e98769  0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-ISOLATION-STAGE-2 (3 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 DROP       all  --  *      docker-b3c5b777  0.0.0.0/0            0.0.0.0/0           
    0     0 DROP       all  --  *      docker-f7e98769  0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-USER (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOS_PROTECT (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     icmp --  eth1   *       0.0.0.0/0            0.0.0.0/0            icmptype 8 limit: avg 1/sec burst 5
    0     0 DROP       icmp --  eth1   *       0.0.0.0/0            0.0.0.0/0            icmptype 8
    0     0 RETURN     tcp  --  eth1   *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x17/0x04 limit: avg 1/sec burst 5
    0     0 DROP       tcp  --  eth1   *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x17/0x04
  104  6176 RETURN     tcp  --  eth1   *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x17/0x02 limit: avg 10000/sec burst 100
    0     0 DROP       tcp  --  eth1   *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x17/0x02
    0     0 RETURN     icmp --  eth0   *       0.0.0.0/0            0.0.0.0/0            icmptype 8 limit: avg 1/sec burst 5
    0     0 DROP       icmp --  eth0   *       0.0.0.0/0            0.0.0.0/0            icmptype 8
49132 1965K RETURN     tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x17/0x04 limit: avg 1/sec burst 5
80467 3219K DROP       tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x17/0x04
95348 5727K RETURN     tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x17/0x02 limit: avg 10000/sec burst 100
    0     0 DROP       tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x17/0x02

IP route info on the docker host

ip route show
default via 192.168.1.1 dev eth0  src 192.168.1.21 
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.0.1 linkdown 
172.18.0.0/16 dev docker-f7e98769  proto kernel  scope link  src 172.18.0.1 
172.21.0.0/16 dev docker-b3c5b777  proto kernel  scope link  src 172.21.0.1 
192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.21 
192.168.1.0/24 dev eth1  proto kernel  scope link  src 192.168.1.22 
192.168.1.240/28 dev macvlan-br0  scope link

route for macvlan-br0 is setup using the following commands. I did this so containers on maclvan could reach each other. Followed Synology Community

#Host macvlan bridge recreate
ip link add macvlan-br0 link eth1 type macvlan mode bridge
# Add a new IP for the link. This should be available in your network.
ip addr add 192.168.1.239/32 dev macvlan-br0
ip link set macvlan-br0 up
# Add a route to all the containers running in macvlan subnet 
ip route add 192.168.1.240/28 dev macvlan-br0

This is running on Synology DSM 6.2.4
Version:

docker --version
Docker version 20.10.3, build b35e731

Edit: I had raised an issue in the image repo and thinking is that it might be specific to my docker setup vs the image. So posting here to see if any one can help determine the root cause.

2

You already beat the macvlan kernel restriction that a child interface is not allowed to interact with the parent interface directly, by adding a “shim” child interface to your host. As the host is able to use the child interface to interact with the other macvlan child interfaces. Other hosts in your network should not be affected by this kernel restriction.

Please try to use a different cidr netmask on your shim interface macvlan-br0: /32 means it has a netmaks with 1 ip in it. Also I don’t remember if the shim interface’s ip must be inside or outside the docker macvlan ip-range. I guess outside is the proper approach if the netmask is at least /27.

Though this should only affect the Syno to gluetan communication. From gluetan, you would need to use the shim interfaces ip to access the Syno.

The published port on gluetun is not necessary as there is no port publishing with macvlan interfaces.

Did you disable the Synology firewall for the sake of testing?

3

I used the following as the guide for “shim” https://community.synology.com/enu/forum/1/post/120118 (I have updated the post). Like you mentioned this should only impact communication from synology host.

I have other containers running in the same network that i am able to access. For example i am able to make the following call curl http://192.168.1.241:6767/ from my nas without issues.

Right now i am not able to access the end point from any device on the network.

Regarding firewall, its not running :).

4

I can also confirm that it works fine when i am running in bridge mode

docker exec -it gluetun sh -c 'iptables -nvL ; ip route show'
Chain INPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  860 66411 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
 286K  521M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
   38  2280 ACCEPT     all  --  eth0   *       0.0.0.0/0            172.19.0.0/16       
    0     0 ACCEPT     tcp  --  wg0    *       0.0.0.0/0            0.0.0.0/0            tcp dpt:58247
    0     0 ACCEPT     udp  --  wg0    *       0.0.0.0/0            0.0.0.0/0            udp dpt:58247
    0     0 ACCEPT     tcp  --  wg0    *       0.0.0.0/0            0.0.0.0/0            tcp dpt:54963
    0     0 ACCEPT     udp  --  wg0    *       0.0.0.0/0            0.0.0.0/0            udp dpt:54963

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  860 66411 ACCEPT     all  --  *      lo      0.0.0.0/0            0.0.0.0/0           
 187K  120M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  *      eth0    172.19.0.2           172.19.0.0/16       
    0     0 ACCEPT     all  --  *      eth0    172.19.0.2           192.168.1.0/24      
    1   176 ACCEPT     udp  --  *      eth0    0.0.0.0/0            185.213.154.67       udp dpt:51820
  124  7448 ACCEPT     all  --  *      wg0     0.0.0.0/0            0.0.0.0/0           


default via 172.19.0.1 dev eth0 
172.19.0.0/16 dev eth0 proto kernel scope link src 172.19.0.2