MAC and IP address duplication in 3 node Swarm MACVLAN

Docker Engine Swarm
1

Hi, I have 3 node docker swarm. I’m trying to run a number of applications that use multicast for discovery and decided to use MACVLAN so that they’d have an IP address on the existing subnet.

Using portainer, I configured a MACVLAN config for each node of
Subnet - 192.168.1.0/24
Gateway - 192.268.1.254
IP Range - 192.168.1.192/28
The resulting config is:

[
    {
        "Name": "mfo-network",
        "Id": "f3a841c058980cc47ce67c56184e1c6a246f84638af4636bc93b567237e31050",
        "Created": "2025-05-03T13:48:57.451545374-04:00",
        "Scope": "local",
        "Driver": "null",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "192.168.1.0/24",
                    "IPRange": "192.168.1.192/28",
                    "Gateway": "192.168.1.254"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": true,
        "Containers": {},
        "Options": {
            "parent": "eno1"
        },
        "Labels": {}
    }
]

I then created a network for the containers to attach to:

[
    {
        "Name": "mfonet",
        "Id": "lgowmxtub0zsgbw0t3wo9y2xj",
        "Created": "2025-05-03T17:50:14.585793545Z",
        "Scope": "swarm",
        "Driver": "macvlan",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "",
            "Options": null,
            "Config": []
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": "mfo-network"
        },
        "ConfigOnly": false,
        "Containers": null,
        "Options": null,
        "Labels": null
    }
]

When I start the stacks, each container is assigned an IP address and MAC address as expected, but each node seems to work independently, so the first application on each node is always assigned

IP address: 192.168.1.192
MAC: 02:42:c0:a8:01:c0

Which is obviously not useful.

How should MACVLAN be configured so that each container always gets a unique IP address? Do I have to create different subnet range for each node, and then connect them with the same network definition?

Thanks

Alan

2

Please share the output of docker info. The output of your network inspects does not look you are running a recent docker-ce version.

From the looks of both network inspects, it looks like you followed an instruction like this:

It has been years since I tried macvlan with swarm (following the above instructions), but I don’t recall having any issues with double usage of ips or mac-addresses. If you did configure the macvlan like described in the blog post, and it still shows the behavior you experience, you should consider raising an issue in the Moby GitHub project.

Identical mac-addresses in the same network will freak out most (or even all?) switches.

3

Just to be sure: are you aware that unlike plain containers, swarm services do not allow setting fixed ips?

A /28 ip range will have 16 ip address, which will be randomly assigned to the swarm service tasks that are attached to the swarm scoped macvlan network.

4

Output from docker info - I’m using OpenSUSE MicroOS and installed their current version.

Client:
 Version:    27.5.1-ce
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.22.0
    Path:     /usr/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  2.35.1
    Path:     /usr/lib/docker/cli-plugins/docker-compose

Server:
 Containers: 14
  Running: 14
  Paused: 0
  Stopped: 0
 Images: 16
 Server Version: 27.5.1-ce
 Storage Driver: overlay2
  Backing Filesystem: btrfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: active
  NodeID: uitl4zwnd63t0e8bwcixhl2j8
  Is Manager: true
  ClusterID: ytynbihm5161drxlrrr6f9ja5
  Managers: 3
  Nodes: 3
  Default Address Pool: 10.0.0.0/8  
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 192.168.1.14
  Manager Addresses:
   192.168.1.14:2377
   192.168.1.15:2377
   192.168.1.16:2377
 Runtimes: oci runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 57f17b0a6295a39009d861b89e3b3b87b005ca27
 runc version: v1.2.6-0-ge89a29929c77
 init version: 
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.14.4-1-default
 Operating System: openSUSE MicroOS
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 23.25GiB
 Name: jarvis
 ID: 3b353f8a-d1a2-40bd-be1f-6aea4fbbdd85
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

I followed this post: Using MACVLAN in Portainer.io. I’ve seen the link you added before, and prior to use portainer to configure the MACVLAN, the command line I used to configure the MACVLAN previously was pretty similar, the only difference was my IP address range was a true subnet of the main LAN subnet, not the starting point to the end of the subnet address range.

Two questions:
The docker nodes should cooperate on container address assignment to prevent conflicting MAC and IP addresses?
And, the --ip-range just defines the starting point of the range, which doesn’t have to be the logical start of a subnet as defined by the --ip-range address and subnet mask? What purpose does the subnet mask serve in the ip-range definition?

Thank you for responding, I appreciate your help.

5

I am aware that I can’t use fixed IP address. I have played with workarounds, and found out that each ip-range on a node must have a unique gateway, which is an unfortunate limitation.

And I was deliberately carving out a “subnet” of 16 addresses - though I don’t need that many IP addresses, just more than 8.

I have not see any randomness. In all scenarios, each docker node hands out the lowest address first, then the next etc. very predictable, as long as you know which node will host which application, which appears to be more random.

But it begs the question - is there a way of having a predictable IP address for a given container - I’m not wed to MACVLAN?

Thanks again

6

So it’s a package maintained and supported by OpenSUSE or its community. Only the packages from docker’s official repos for the supported platforms are guaranteed to implement vanilla docker behavior. For instance the output of your inspects differs from the output I get, which could be due to the different versions (I used 28.0.4 and 28.1.1) or differing implementations.

From what I remember they did. I would have remembered if my switches freaked out due to non unique mac-address.

The ip-range uses the CIDR notation to mark the starting ip and how “broad” the range is. It is not a subnet. The ip-range should be outside any dhcp range, as the docker network will provide its own ip address management. If any ip in the ip-range is taken from a device outside the docker network, it needs to be marked as aux-address, so ipam won’t assign this ips to any container.

It is not restricted to the ip-range. A gateway ip can be used by exactly one docker network.

Might be true for the first deployment. But every time a swarm service task is started, it will have a different ip, thus whenever you upate the swarm service configuration and deploy it, on every system boot, whenever the service task terminates and gets replaced. I wouldn’t call this predictable at all.

Unfortunately not.

Probably not the solution you are looking for, but I will still share it:
You can run a reverse proxy like traefik deployed as global service. It can handle layer7 and layer4 traffic and forward ingress traffic to the right service task. You could leverage keepalived, ucarp, or anything that implements the vrrp protocol to introduce a failover ip that is assigned to one of the nodes, and moves to another node if the node is down. This allows to reach the traefik container using the failover ip.

Layer7 http(s) traffic could be easily reverse proxied based on domain names and/or path prefixes. Name resolution of course would be needed to be handled by an external dns server.
Layer4 traffic can reverse proxy traffic from a specific incoming port to a specific container and port.

I never needed more than this.

7

I just tested it by creating my macvlan configs (same of each node):

docker network create \
  --config-only \
  --subnet 192.168.x.0/24 \
  --gateway=192.168.x.1 \
  --ip-range 192.168.x.192/28 \
  -o parent=eth0 \
  swarm-macvlan-config

Then create the swarm scoped macvlan network:

docker network create \
  -d macvlan \
  --scope swarm \
  --config-from swarm-macvlan-config \
  swarm-macvlan

Then deploy a service with multiple replicas:

docker service create --network swarm-macvlan --replicas 5 --publish 18080:80 nginx

Then check the ouput of docker network inspect on each node:

node1:

...
        "Containers": {
            "334de35ffd3cb7c97ba1e106e801f026bdbb246ed23b1e324a60a6bbc241208f": {
                "Name": "vigorous_shtern.4.vcqb9sbm9rdsyyqhubjmfvbvw",
                "EndpointID": "792b59e00e315e712ab60be9fc2db202fd2e9bf6fdd1fbbffa0825e1d85b4c6a",
                "MacAddress": "4a:b4:97:8e:cd:bc",
                "IPv4Address": "192.168.x.193/24",
                "IPv6Address": ""
            },
            "b23c848a700d37560ccc387be9d742d41b0d7275bab65461529bded4bcc773cb": {
                "Name": "vigorous_shtern.2.67dh4r8za9mhxqcect7bor0iz",
                "EndpointID": "be169b87d4edb80f1dc28b0df4d32945e5db62a1fd933dd8d816fb187c081ebe",
                "MacAddress": "7a:08:4e:6e:09:c0",
                "IPv4Address": "192.168.x.192/24",
                "IPv6Address": ""
            }
        },
...

node2:

...
        "Containers": {
            "030b5c41c9c63fb38b80c6904a6eb26d0a83012e1a1894bd5edfbb923d4b9d6f": {
                "Name": "vigorous_shtern.1.aucvcdurjcf2toyggeo31n0dm",
                "EndpointID": "a8781b88fc602f4291f2b5f714f1d54f63db0fd015880bc80f25cb97c90756d7",
                "MacAddress": "be:50:ce:22:b9:33",
                "IPv4Address": "192.168.x.192/24",
                "IPv6Address": ""
            }
        },
...

node3:

...
        "Containers": {
            "45ccd502d41484e58b2d90ea34ab4dcdaee4cfb3d7562e820b282c06370d148e": {
                "Name": "vigorous_shtern.5.0inlf8sytlmog82gtjszqrse6",
                "EndpointID": "411e03620a8f8df4809bdd6231fb2fc65216846339a62f528791d8cd0780fc39",
                "MacAddress": "62:2f:b0:8d:82:4e",
                "IPv4Address": "192.168.x.193/24",
                "IPv6Address": ""
            },
            "50ea9e9725902f884310544193d7a86daf7cfb05e38c5e469b27fcc6282f7a1d": {
                "Name": "vigorous_shtern.3.ru59t3qxeq2n65o4aobk7iev1",
                "EndpointID": "12225394a81df7d2a91404cba642ca408f86a67fb8569c96be89b6ce8d44162f",
                "MacAddress": "96:d0:0f:0e:c9:ee",
                "IPv4Address": "192.168.x.192/24",
                "IPv6Address": ""
            }
        },
...

So indeed the ip addresses are not coordinated, but the mac addresses do not collide. So the double assignment of ip addresses is indeed a bug.

We had situations in the past, where Portainer was responsible for non-unique mac addresses, when a container configuration was cloned to create a new container based on the old config. But I don’t see how this would apply for swarm service/stack deployments.

1 Like
8

I submitted a bug report in the Moby project: IPAM for Swarm scoped macvlan networks is not coordinated amogst nodes · Issue #49921 · moby/moby · GitHub

2 Likes
9

Thank you, and thank you for testing the setup as well. Non-duplicate IP addresses is good enough for my setup. The lack of predictable IP addresses is inconvenient, but as they’re largely for management interfaces, I can survive that. All the user apps are discoverable by their clients and so far have worked.

I’d looked at traefik as well as metallb in the context of k3s, when I was first looking in to this and concluded it was unnecessarily complex. I may revisit, but I think they don’t support multicast discovery properly, which a couple of my apps need.

I’m currently working around this be using a different subnet range on each node, which avoids the duplicate MAC and duplicate IP issue. I’ll mark your testing as the solution.

I appreciate the help - thank you!

10

Welcome!

Indeed, multicast is only available with host networking, macvlan or ipvlan L2. But now it makes sense that you don’t need to rely on fixed ip addresses.

I am curious how anything of what we discussed applies to k3s. It shouldn’t be using docker as a container runtime, as Kubernetes doesn’t need it.

11

It doesn’t apply to k3s. When I started on this journey of setting up a 3 node cluster for my home network I was choosing between k3s and docker swarm. Family acceptance factor was the top need, i.e. it had to just work - which in my context translated to robustness, simplicity and multicast support. Hence MACVLAN and thus docker swarm.

12

Makes sense!

The simplicity of Docker Swarm is its biggest advantage (everything is easier to setup and maintain) and disadvantage (it only supports basic features, even less than plain docker containers) at the same time :slight_smile:

1 Like