I have a swarm cluster with 3 managers and 3 workers. When I run docker network inspect on the worker nodes, I found the load balancer (created by Swarm, has the name like “lb-xxx”) container’s IP on worker1 clashes with a containers overlay IP on worker 2.
I have to manually run docker container kill on worker2 to kill the container to let Swarm assign a new IP for it.
Does anyone know the possible cause of the problem and how to prevent it from happening? Thanks
For example, in the below output, container lb-srncn_srncn on worker1 and srncn_mysbts.2.g7uyulsbbewgipbqmxk1lrg3r on worker2 have clashed IP.
docker network inspect on worker1, most of the container info is removed for readability.
[
{
"Name": "srncn_srncn",
"Id": "k16v291x1evczfgol74kwn000",
"Created": "2019-11-22T01:48:05.744154684Z",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.1.0/24",
"Gateway": "10.0.1.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"060085c63ddb6c9e0a17a5c4da39914375fcb942823d86ce00be314de7ba0609": {
"Name": "srncn_jenkins-slave-1.1.uphvpq33t3gsa76pduhhgvfyg",
"EndpointID": "55db5d997f89bb0dd653004f769eb1181404fb3a44febd44151300c497bb64f3",
"MacAddress": "02:42:0a:00:01:e0",
"IPv4Address": "10.0.1.224/24",
"IPv6Address": ""
},
"lb-srncn_srncn": {
"Name": "srncn_srncn-endpoint",
"EndpointID": "060a47ef50d98bc76c977edb736407a276191d2836033cf8fb571ae1305a7fe7",
"MacAddress": "02:42:0a:00:01:fd",
"IPv4Address": "10.0.1.253/24",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4098"
},
"Labels": {
"com.docker.stack.namespace": "srncn"
},
"Peers": [
{
"Name": "f791445b0688",
"IP": "10.107.243.51"
},
{
"Name": "7b0a295e1abe",
"IP": "10.107.243.49"
},
{
"Name": "2ccdb685ebb5",
"IP": "10.107.243.54"
}
]
}
]
docker network inspect on worker2, most of the container info is removed for readability.
[
{
"Name": "srncn_srncn",
"Id": "k16v291x1evczfgol74kwn000",
"Created": "2019-07-08T23:06:34.314105027Z",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.1.0/24",
"Gateway": "10.0.1.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"026f8ced9d9cebf433c8b81a462e1b5096c460596f0dc7a8742e14bbde3241fe": {
"Name": "srncn_mysbts.2.g7uyulsbbewgipbqmxk1lrg3r",
"EndpointID": "314592d107e6ec1ff9dc06ac4f5c65ceea2bd3cd299942be27c5046bcaea4813",
"MacAddress": "02:42:0a:00:01:fd",
"IPv4Address": "10.0.1.253/24",
"IPv6Address": ""
},
"lb-srncn_srncn": {
"Name": "srncn_srncn-endpoint",
"EndpointID": "81ad57c4ffa00135919454886042ebafeea9c819e4e4353ebc6d8c68e9d2705e",
"MacAddress": "02:42:0a:00:01:05",
"IPv4Address": "10.0.1.5/24",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4098"
},
"Labels": {
"com.docker.stack.namespace": "srncn"
},
"Peers": [
{
"Name": "7b0a295e1abe",
"IP": "10.107.243.49"
},
{
"Name": "2ccdb685ebb5",
"IP": "10.107.243.54"
},
{
"Name": "f791445b0688",
"IP": "10.107.243.51"
}
]
}
]
docker info output
Containers: 69
Running: 7
Paused: 0
Stopped: 62
Images: 140
Server Version: 18.09.5
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
NodeID: vjespnrwxzyldbjimbvb8mtfc
Is Manager: false
Node Address: 10.107.243.51
Manager Addresses:
10.107.243.36:2377
10.107.243.48:2377
10.107.243.57:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-51-generic
Operating System: Ubuntu 18.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.852GiB
Name: docker-worker-1
ID: RFSB:ZU63:DMAU:OQKU:XHDP:55KB:WMWM:5RYG:62CV:I4EQ:YDNU:PXG5
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Registry Mirrors:
https://xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Live Restore Enabled: false
Product License: Community Engine
WARNING: No swap limit support