Setting Default Gateway to a container

Docker Engine Compose
1

I’m trying to create a vpn container that acts as a gateway for other container to use.

My docker-compose.yml is

version: '2'

services:
  mullvad:
    container_name: <container_name>
    image: <vpn_server_image>
    command: sleep infinity
    volumes:
    - "./openvpn:/etc/openvpn"
    networks:
      vpn:
        ipv4_address: 172.20.0.1
    devices:
    - "/dev/net/tun:/dev/net/tun"
    privileged: true
    cap_add:
    - NET_ADMIN

networks:
  vpn:
    driver: bridge
    ipam:
      driver: default
      config:
      - subnet: 172.20.0.0/16
        gateway 172.20.0.1

When I try to run the app I get an address already in use error:

# docker-compose up -d
Recreating <container_name>

ERROR: for <container_name>  Address already in use
Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "compose/cli/main.py", line 63, in main
AttributeError: 'ProjectError' object has no attribute 'msg'
docker-compose returned -1

network inspection suggests that the address shouldn’t be in use:

# docker network inspect <container_name>_vpn
[
    {
        "Name": "<container_name>_vpn",
        "Id": "<guid>",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.20.0.0/16",
                    "Gateway": "172.20.0.1"
                }
            ]
        },
        "Internal": false,
        "Containers": {},
        "Options": {},
        "Labels": {}
    }
]

What’s going on here?

In case I’m just doing it horribly wrong, what I want is a “vpn” network that other containers can attach to. When they do so, they are given a default route to the vpn container. This container will be configured to bridge the docker network to the vpn tunnel.

1 Like
2

I have a hunch as to what is going on here:

When docker is creating the network bridge that will be used for the virtual network, is uses the “gateway” option passed into the network config as the ip-address for a HOST virtual interface that is connected the new network by default.

Thus, when the network driver tries to setup the network for your vpn container, the address you wanted has already been used.

To see this in person, try creating your network by hand:

$docker network create \
--driver=bridge \
--subnet=172.20.0.0/16 \
--gateway=172.20.0.1 \
testnet

...

$docker network inspect testnet
[
    {
        "Name": "testnet",
        "Id": "a742e6afc6c8d3ac1919d0ba4820686b06355af2ce1aa621b54144a5d6320fc9",
        "Scope": "local",
        "Driver": "bridge",
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.20.0.0/16",
                    "Gateway": "172.20.0.1"
                }
            ]
        },
        "Containers": {},
        "Options": {}
    }
]

Then immediately call ifconfig on the host:

$ifconfig
br-a742e6afc6c8 Link encap:Ethernet  HWaddr 02:42:8e:0c:ad:30
      inet addr:172.20.0.1  Bcast:0.0.0.0  Mask:255.255.0.0
      UP BROADCAST MULTICAST  MTU:1500  Metric:1
      RX packets:0 errors:0 dropped:0 overruns:0 frame:0
      TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:0
      RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

Unfortunately, I have yet to figure out how to work around this behavior in any sane way…
There are a couple of hacks you can use, such as manually creating the network bridge then removing the host interface, but they are all pretty kludgey.

There is also pipework, which is a script that streamlines the kludgey part, but I wouldn’t call it a good solution.

If anyone know of a better way to solve this, I would LOVE to know!

3

Hmm, you’re right. I thought that using the --internal flag on docker network create might stop this happening, but apparently not.

4

So do the Docker devs ever actually look at this forum or is it just users helping users?

1 Like
5

I ran into exactly the same issue and really would like to know how to bring up the gateway interface inside a container.

6

Had a similar use-case: Encapsulate a transparent proxy inside a container that will forward traffic over a corporate firewall to reach the internet (cntlm / redsocks). Tried bridges, mcvlan, …: It will always end in “address already in use”.

Only way to fix this currently seems to modify the routing table of the host system by using policy based routing:

ip rule add from 172.19.0.0/16 table 200 # the network you like to forward over the gateway container
ip route add default via 172.19.0.2 table 200 # your gateway container
iptables -D FORWARD 1 # remove the DOCKER-ISOLATION rule to enable bridge / bridge routing
1 Like
7

I think you will need to set your gateway as “Gateway”: “172.20.0.1/16” So use --gateway=172.20.0.1/16 while creating the network

8

Doesn’t work for me. I get:

$ docker network create --subnet 172.20.0.0/28 --gateway 172.20.0.0/28 test
Error response from daemon: invalid gateway address 172.20.0.0/28 in Ipam configuration
9

Just thought I would update this for people that want this to persist across reboots, save the following to /etc/systemd/network/10-NAMEOFDOCKERINTERFACE.network:

[Match]
#Set this to the interface that you want the route added to eg: br-3de204ae1471
Name=NAMEOFDOCKERINTERFACE

[Network]
#Set the address to the IP of the interface above
Address=172.19.0.1
#Set this to the default gateway
DNS=172.19.0.100

[Route]
Table=200
#Set this to the default gateway
Gateway=172.19.0.100

[RoutingPolicyRule]
Table=200
#Set this to the subnet you want to route via the gateway above
From=172.19.0.0/16

[RoutingPolicyRule]
To=172.19.0.100
Table=200
10

@agrothberg you specified a wrong gateway, that’s the network address you specify! a gateway is the ip address of a host which is not the same!

11

apparently the problem persist!

When you specify a gateway for a docker network, it is the adress of the virtual router from docker daemon
and so you cannot take the same address for a container.

The same for --aux-address

So to my knowledge you cannot specify a gateway for a specific container which would be another container

12

but we could use cap-add= …

13

Is there still no clean solution to this issue today ?

14

Hello,

I had the same problem as I assign every docker network a custom outgoing IP address by disabling the com.docker.network.bridge.enable_ip_masquerade on the network and defining a custom SNAT rule via nftables.

When connecting two of these networks to one container to connect two projects with each other, the containers always using “the wrong” network for outgoing communication.

I solved the problem with creating a transit docker network which has the --internal flag set.

docker network create -d bridge -o com.docker.network.bridge.enable_ip_masquerade=false project_a
docker network create -d bridge -o com.docker.network.bridge.enable_ip_masquerade=false project_b
docker network create -d bridge --internal transit_project_a_b

docker network connect project_a project_a_app
docker network connect project_b project_b_app
docker network connect transit_project_a_b project_a_app
docker network connect transit_project_a_b project_b_app

I link this solution alot more than hacking again with nftables or iptables around this.

Thank you very much for the hint with the --internal flag, @richardpayne