Swarm with a static IP - Multiple interfaces

madplt · July 19, 2019, 8:20am

I’ve been banging my head over getting networking to work the way I want it to.

Situation:

2 physical interfaces
ens160 10.10.0.13/24 => VLAN 10 on the router
ens192 10.20.0.2/16 => VLAN 20 on the router

Goal:
Having several stacks with each container having their own static IP address and using the firewall for routing (no direct communication)

I have initialized the swarm with docker swarm init --advertise-addr=10.10.0.13 as this is the primary IP address to use.

Compose file:

version: '3.5'

services:
  portainer:
    image: portainer/portainer
    deploy:
      replicas: 1
      update_config:
        parallelism: 1
      restart_policy:
        condition: on-failure
    command: --no-analytics
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /opt/data/portainer:/data
    networks:
      mgmt-vlan:
        ipv4_address: 10.10.0.5
    ports:
      - 9000:9000

networks:
  mgmt-vlan:
    external: true

I have tried several commands to get the mgmt-vlan network going (havent even started on the other network yet since this one doesn’t work…):

docker network create -d macvlan --subnet 10.10.0.0/24 --gateway 10.10.0.1 -o parent=ens160 --scope swarm mgmt-vlan
docker network inspect mgmt-vlan
[
    {
        "Name": "mgmt-vlan",
        "Id": "mp33mlsa2d4ve83vhc4nd8r1z",
        "Created": "2019-07-19T07:21:44.018736517Z",
        "Scope": "swarm",
        "Driver": "macvlan",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "",
            "Options": null,
            "Config": []
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": null,
        "Options": null,
        "Labels": null
    }
]

As you can see the IPAM part is empty

docker network create -d overlay --subnet=10.10.0.0/24 --gateway=10.10.0.1 --aux-address="exclude_host=10.10.0.13" -o parent=ens160 --scope swarm mgmt-vlan
docker network inspect mgmt-vlan
[
    {
        "Name": "mgmt-vlan",
        "Id": "28w2wd4uclvm05th672jh7zo1",
        "Created": "2019-07-19T07:22:17.536742512Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.10.0.0/24",
                    "Gateway": "10.10.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": null,
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4099",
            "parent": "ens160"
        },
        "Labels": null
    }
]

docker stack deploy -c docker-core.yml core
docker inspect a9067438d42e

...
"Networks": {
                "ingress": {
                    "IPAMConfig": {
                        "IPv4Address": "10.255.0.71"
                    },
                    "Links": null,
                    "Aliases": [
                        "a9067438d42e"
                    ],
                    "NetworkID": "fwchqo21sxv8oey03zpl6wihb",
                    "EndpointID": "5bc9f00788b501d7ffe1cdb41cc7f4d13d392bc2ad9270e6e684dfb167232cfa",
                    "Gateway": "",
                    "IPAddress": "10.255.0.71",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:0a:ff:00:47",
                    "DriverOpts": null
                },
                "mgmt-vlan": {
                    "IPAMConfig": {
                        "IPv4Address": "10.10.0.3"
                    },
                    "Links": null,
                    "Aliases": [
                        "a9067438d42e"
                    ],
                    "NetworkID": "28w2wd4uclvm05th672jh7zo1",
                    "EndpointID": "7c328e1896e9c7b2d7d22b35b2bdc05621786df53ca62d420bf7f1d98f626360",
                    "Gateway": "",
                    "IPAddress": "10.10.0.3",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:0a:32:00:03",
                    "DriverOpts": null
                }
            }
...

This one does not take its assigned IP and neither of them is pingable/reachable.

Anyone have an idea what I’m missing?

Thanks!

meyay · July 19, 2019, 9:25am

You need to create a macvlan configuration on each node first and then the real macvlan that uses the configuration.

on each node: docker network create --config-only --subnet 10.10.0.0/24 -o parent=ens160 --ip-range 10.10.0.0/24 mgmt-vlan-config

Though, make sure the ips in --ip-range are not assigned by a dhcp server - otherwise you will end up with ip collisions AND make sure that --ip-range does not cover ips that are manualy assigned in your local lan. You will want to have a subnet of your subet that is exclusivly managed by macvlan.

on a master node: docker network create -d macvlan --scope swarm --config-from mgmt-vlan-config mgmt-vlan

madplt · July 19, 2019, 9:31am

Thanks for your reply

I have executed both commands and started the stack again

docker network create --config-only --subnet 10.10.0.0/24 -o parent=ens160 --ip-range 10.10.0.0/24 mgmt-vlan-config
docker network create -d macvlan --scope swarm --config-from mgmt-vlan-config mgmt-vlan
docker stack deploy -c docker-core.yml core
docker inspect 770098e2ab5a
...
"mgmt-vlan": {
                    "IPAMConfig": {},
                    "Links": null,
                    "Aliases": [
                        "770098e2ab5a"
                    ],
                    "NetworkID": "fjblg5b8y8j840j8x0nvm5ipl",
                    "EndpointID": "4edf215536fcc6bd324f627aad073579b0dc16768ce90c608465599b43e7de44",
                    "Gateway": "10.10.0.1",
                    "IPAddress": "10.10.0.2",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:0a:32:00:02",
                    "DriverOpts": null
                }
...

However as you can see it used 10.10.0.2 instead of 10.10.0.5 what I’ve assigned in the compose.yml

    networks:
      mgmt-vlan:
        ipv4_address: 10.10.0.5

meyay · July 19, 2019, 9:33am

i complelty ignored that you use docker swarm. Last time I checked assigning ipv4_address is not possible with swarm. Providing a static IP, but beeing able to have n replicas of the container would not realy work, would it?

Honestly: I know noone who actualy uses macvlan in productive environments.

madplt · July 19, 2019, 9:36am

hmmm. Do you possibly have a solution/method which could work? I’d like to use compose to run different .yml files with multiple containers.

eg

core.yml
 portainer

logging.yml
 elastic
 logstash
 kibana

and each of them a static IP

meyay · July 19, 2019, 9:41am

Hmm, are you using swarm or docker-compose?

The macvlan creation I posted was aimed toward swarm. Though, it will not allow to assign fixed ips. They WILL change with every task beeing scheduled and every container beeing created by a task.

Assigning fixed ip addresses with a standalone macvlan per host in combination with docker-compose on the other side works.

You might want to check this epic on github, to see what actualy is available in Docker Swarm.

madplt · July 19, 2019, 10:04am

Perhaps what I’m trying is a little exotic

The main goal is to have all containers communicate with each other via the firewall.

I’m using swarm (docker swarm init --advertise-addr=10.10.0.13)

But I’m very open to other solutions.

meyay · July 19, 2019, 10:08am

Maybee there is someone arround who doesn’t know that what you are asking its not implemented in swarm and finds a solution for your

Good luck!

I am eager to see how this one plays out.

madplt · July 19, 2019, 10:10am

Thanks so far! I’ve updated the OP with the image and some extra info about the goal. Hope someone else knows something

madplt · July 19, 2019, 1:36pm

Tried another experimental thing; ipvlan

docker network create -d ipvlan --subnet=10.10.0.0/24 --gateway=10.10.0.1 -o ipvlan_mode=l2 -o parent=ens224  mgmt-vlan

docker-compose up

Put the ens224 and the vswitch (esxi) on promisc and now I’m able to ping to/from the container. No idea if this is a requirement, but I’m not touching that anymore since its working

meyay · July 19, 2019, 3:36pm

@madplt: amazing, I wasn’t aware that ipvlan is even an option. How did you enable it?

Promiscouse mode on vSwitches is a requirement for macvlan as well.

Assigning a fixed ip for swarm services is still not solved.

update: I found a nice discussion about macvlan and ipvlan: http://hicu.be/macvlan-vs-ipvlan

update2: adding "experimental": true to the daemon.json seems sufficient. ipvlan requires kernel 4.2 or later.