Swarm with a static IP - Multiple interfaces

madplt · July 19, 2019, 8:20am

I’ve been banging my head over getting networking to work the way I want it to.

Situation:

2 physical interfaces
ens160 10.10.0.13/24 => VLAN 10 on the router
ens192 10.20.0.2/16 => VLAN 20 on the router

Goal:
Having several stacks with each container having their own static IP address and using the firewall for routing (no direct communication)

I have initialized the swarm with docker swarm init --advertise-addr=10.10.0.13 as this is the primary IP address to use.

Compose file:

version: '3.5'

services:
  portainer:
    image: portainer/portainer
    deploy:
      replicas: 1
      update_config:
        parallelism: 1
      restart_policy:
        condition: on-failure
    command: --no-analytics
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /opt/data/portainer:/data
    networks:
      mgmt-vlan:
        ipv4_address: 10.10.0.5
    ports:
      - 9000:9000

networks:
  mgmt-vlan:
    external: true

I have tried several commands to get the mgmt-vlan network going (havent even started on the other network yet since this one doesn’t work…):

docker network create -d macvlan --subnet 10.10.0.0/24 --gateway 10.10.0.1 -o parent=ens160 --scope swarm mgmt-vlan
docker network inspect mgmt-vlan
[
    {
        "Name": "mgmt-vlan",
        "Id": "mp33mlsa2d4ve83vhc4nd8r1z",
        "Created": "2019-07-19T07:21:44.018736517Z",
        "Scope": "swarm",
        "Driver": "macvlan",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "",
            "Options": null,
            "Config": []
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": null,
        "Options": null,
        "Labels": null
    }
]

As you can see the IPAM part is empty

docker network create -d overlay --subnet=10.10.0.0/24 --gateway=10.10.0.1 --aux-address="exclude_host=10.10.0.13" -o parent=ens160 --scope swarm mgmt-vlan
docker network inspect mgmt-vlan
[
    {
        "Name": "mgmt-vlan",
        "Id": "28w2wd4uclvm05th672jh7zo1",
        "Created": "2019-07-19T07:22:17.536742512Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.10.0.0/24",
                    "Gateway": "10.10.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": null,
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4099",
            "parent": "ens160"
        },
        "Labels": null
    }
]

docker stack deploy -c docker-core.yml core
docker inspect a9067438d42e

...
"Networks": {
                "ingress": {
                    "IPAMConfig": {
                        "IPv4Address": "10.255.0.71"
                    },
                    "Links": null,
                    "Aliases": [
                        "a9067438d42e"
                    ],
                    "NetworkID": "fwchqo21sxv8oey03zpl6wihb",
                    "EndpointID": "5bc9f00788b501d7ffe1cdb41cc7f4d13d392bc2ad9270e6e684dfb167232cfa",
                    "Gateway": "",
                    "IPAddress": "10.255.0.71",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:0a:ff:00:47",
                    "DriverOpts": null
                },
                "mgmt-vlan": {
                    "IPAMConfig": {
                        "IPv4Address": "10.10.0.3"
                    },
                    "Links": null,
                    "Aliases": [
                        "a9067438d42e"
                    ],
                    "NetworkID": "28w2wd4uclvm05th672jh7zo1",
                    "EndpointID": "7c328e1896e9c7b2d7d22b35b2bdc05621786df53ca62d420bf7f1d98f626360",
                    "Gateway": "",
                    "IPAddress": "10.10.0.3",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:0a:32:00:03",
                    "DriverOpts": null
                }
            }
...

This one does not take its assigned IP and neither of them is pingable/reachable.

Anyone have an idea what I’m missing?

Thanks!

meyay · July 19, 2019, 9:25am

You need to create a macvlan configuration on each node first and then the real macvlan that uses the configuration.

on each node: docker network create --config-only --subnet 10.10.0.0/24 -o parent=ens160 --ip-range 10.10.0.0/24 mgmt-vlan-config

Though, make sure the ips in --ip-range are not assigned by a dhcp server - otherwise you will end up with ip collisions AND make sure that --ip-range does not cover ips that are manualy assigned in your local lan. You will want to have a subnet of your subet that is exclusivly managed by macvlan.

on a master node: docker network create -d macvlan --scope swarm --config-from mgmt-vlan-config mgmt-vlan

madplt · July 19, 2019, 9:31am

Thanks for your reply

I have executed both commands and started the stack again

docker network create --config-only --subnet 10.10.0.0/24 -o parent=ens160 --ip-range 10.10.0.0/24 mgmt-vlan-config
docker network create -d macvlan --scope swarm --config-from mgmt-vlan-config mgmt-vlan
docker stack deploy -c docker-core.yml core
docker inspect 770098e2ab5a
...
"mgmt-vlan": {
                    "IPAMConfig": {},
                    "Links": null,
                    "Aliases": [
                        "770098e2ab5a"
                    ],
                    "NetworkID": "fjblg5b8y8j840j8x0nvm5ipl",
                    "EndpointID": "4edf215536fcc6bd324f627aad073579b0dc16768ce90c608465599b43e7de44",
                    "Gateway": "10.10.0.1",
                    "IPAddress": "10.10.0.2",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:0a:32:00:02",
                    "DriverOpts": null
                }
...

However as you can see it used 10.10.0.2 instead of 10.10.0.5 what I’ve assigned in the compose.yml

    networks:
      mgmt-vlan:
        ipv4_address: 10.10.0.5

meyay · July 19, 2019, 9:33am

i complelty ignored that you use docker swarm. Last time I checked assigning ipv4_address is not possible with swarm. Providing a static IP, but beeing able to have n replicas of the container would not realy work, would it?

Honestly: I know noone who actualy uses macvlan in productive environments.

madplt · July 19, 2019, 9:36am

hmmm. Do you possibly have a solution/method which could work? I’d like to use compose to run different .yml files with multiple containers.

eg

core.yml
 portainer

logging.yml
 elastic
 logstash
 kibana

and each of them a static IP

meyay · July 19, 2019, 9:41am

Hmm, are you using swarm or docker-compose?

The macvlan creation I posted was aimed toward swarm. Though, it will not allow to assign fixed ips. They WILL change with every task beeing scheduled and every container beeing created by a task.

Assigning fixed ip addresses with a standalone macvlan per host in combination with docker-compose on the other side works.

You might want to check this epic on github, to see what actualy is available in Docker Swarm.

madplt · July 19, 2019, 10:04am

Perhaps what I’m trying is a little exotic

The main goal is to have all containers communicate with each other via the firewall.

I’m using swarm (docker swarm init --advertise-addr=10.10.0.13)

But I’m very open to other solutions.

meyay · July 19, 2019, 10:08am

Maybee there is someone arround who doesn’t know that what you are asking its not implemented in swarm and finds a solution for your

Good luck!

I am eager to see how this one plays out.

madplt · July 19, 2019, 10:10am

Thanks so far! I’ve updated the OP with the image and some extra info about the goal. Hope someone else knows something

madplt · July 19, 2019, 1:36pm

Tried another experimental thing; ipvlan

docker network create -d ipvlan --subnet=10.10.0.0/24 --gateway=10.10.0.1 -o ipvlan_mode=l2 -o parent=ens224  mgmt-vlan

docker-compose up

Put the ens224 and the vswitch (esxi) on promisc and now I’m able to ping to/from the container. No idea if this is a requirement, but I’m not touching that anymore since its working

meyay · July 19, 2019, 3:36pm

@madplt: amazing, I wasn’t aware that ipvlan is even an option. How did you enable it?

Promiscouse mode on vSwitches is a requirement for macvlan as well.

Assigning a fixed ip for swarm services is still not solved.

update: I found a nice discussion about macvlan and ipvlan: http://hicu.be/macvlan-vs-ipvlan

update2: adding "experimental": true to the daemon.json seems sufficient. ipvlan requires kernel 4.2 or later.

Topic		Replies	Views
Docker Swarm macvlan Setup for Static IPs Fails with "Gateway Address Already in Use" Swarm docker , swarm	1	31	November 12, 2024
Address conflict using Macvlan network driver in Swarm Swarm docker , swarm	4	4374	August 24, 2018
SWARM and MAC VLAN Driver Swarm	0	1252	May 8, 2018
How to compose replicas to get different IPs from host network? General	2	688	January 24, 2024
Docker Swarm 1.13 Static IP's for Containers? Swarm docker	16	29399	August 4, 2020

Swarm with a static IP - Multiple interfaces

Related topics