Docker swarm load balance container ip clashes with service container ip

I have a swarm cluster with 3 managers and 3 workers. When I run docker network inspect on the worker nodes, I found the load balancer (created by Swarm, has the name like “lb-xxx”) container’s IP on worker1 clashes with a containers overlay IP on worker 2.

I have to manually run docker container kill on worker2 to kill the container to let Swarm assign a new IP for it.

Does anyone know the possible cause of the problem and how to prevent it from happening? Thanks

For example, in the below output, container lb-srncn_srncn on worker1 and srncn_mysbts.2.g7uyulsbbewgipbqmxk1lrg3r on worker2 have clashed IP.

docker network inspect on worker1, most of the container info is removed for readability.

[
    {
        "Name": "srncn_srncn",
        "Id": "k16v291x1evczfgol74kwn000",
        "Created": "2019-11-22T01:48:05.744154684Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.1.0/24",
                    "Gateway": "10.0.1.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "060085c63ddb6c9e0a17a5c4da39914375fcb942823d86ce00be314de7ba0609": {
                "Name": "srncn_jenkins-slave-1.1.uphvpq33t3gsa76pduhhgvfyg",
                "EndpointID": "55db5d997f89bb0dd653004f769eb1181404fb3a44febd44151300c497bb64f3",
                "MacAddress": "02:42:0a:00:01:e0",
                "IPv4Address": "10.0.1.224/24",
                "IPv6Address": ""
	    },
            "lb-srncn_srncn": {
                "Name": "srncn_srncn-endpoint",
                "EndpointID": "060a47ef50d98bc76c977edb736407a276191d2836033cf8fb571ae1305a7fe7",
                "MacAddress": "02:42:0a:00:01:fd",
                "IPv4Address": "10.0.1.253/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4098"
        },
        "Labels": {
            "com.docker.stack.namespace": "srncn"
        },
        "Peers": [
            {
                "Name": "f791445b0688",
                "IP": "10.107.243.51"
            },
            {
                "Name": "7b0a295e1abe",
                "IP": "10.107.243.49"
            },
            {
                "Name": "2ccdb685ebb5",
                "IP": "10.107.243.54"
            }
        ]
    }
]

docker network inspect on worker2, most of the container info is removed for readability.

[
    {
        "Name": "srncn_srncn",
        "Id": "k16v291x1evczfgol74kwn000",
        "Created": "2019-07-08T23:06:34.314105027Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.1.0/24",
                    "Gateway": "10.0.1.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "026f8ced9d9cebf433c8b81a462e1b5096c460596f0dc7a8742e14bbde3241fe": {
                "Name": "srncn_mysbts.2.g7uyulsbbewgipbqmxk1lrg3r",
                "EndpointID": "314592d107e6ec1ff9dc06ac4f5c65ceea2bd3cd299942be27c5046bcaea4813",
                "MacAddress": "02:42:0a:00:01:fd",
                "IPv4Address": "10.0.1.253/24",
                "IPv6Address": ""
            },
            "lb-srncn_srncn": {
                "Name": "srncn_srncn-endpoint",
                "EndpointID": "81ad57c4ffa00135919454886042ebafeea9c819e4e4353ebc6d8c68e9d2705e",
                "MacAddress": "02:42:0a:00:01:05",
                "IPv4Address": "10.0.1.5/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4098"
        },
        "Labels": {
            "com.docker.stack.namespace": "srncn"
        },
        "Peers": [
            {
                "Name": "7b0a295e1abe",
                "IP": "10.107.243.49"
            },
            {
                "Name": "2ccdb685ebb5",
                "IP": "10.107.243.54"
            },
            {
                "Name": "f791445b0688",
                "IP": "10.107.243.51"
            }
        ]
    }
]

docker info output

Containers: 69
Running: 7
Paused: 0
Stopped: 62
Images: 140
Server Version: 18.09.5
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
NodeID: vjespnrwxzyldbjimbvb8mtfc
Is Manager: false
Node Address: 10.107.243.51
Manager Addresses:
10.107.243.36:2377
10.107.243.48:2377
10.107.243.57:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-51-generic
Operating System: Ubuntu 18.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.852GiB
Name: docker-worker-1
ID: RFSB:ZU63:DMAU:OQKU:XHDP:55KB:WMWM:5RYG:62CV:I4EQ:YDNU:PXG5
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Registry Mirrors:
https://xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Live Restore Enabled: false
Product License: Community Engine
WARNING: No swap limit support

1 Like