Docker Community Forums

Share and learn in the Docker community.

Error with IP attribution in a Docker Swarm

swarm

(Sunchain) #1

Hello everyone,
I have trouble tranferring a solution from AWS EC2 instances to OVH.
This solution uses Docker Swarm. Initially, some problems occured because the VXLAN option was not set on the server, but I got that fixed.
Nevertheless, my containers can’t seem to be able to communicate with one another. After some investigation, I found that one or several container have multiple IP address but I can’t determine why.

My setup is :

$ uname -a
Linux <hostname> 4.9.135-xxxx-std-ipv6-64 #1 SMP Mon Oct 22 13:00:30 UTC 2018 x86_64 GNU/Linux
$ docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:23:18 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:22:21 2018
  OS/Arch:          linux/amd64
  Experimental:     false

$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 18.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: <SNIP>
 Is Manager: true
 ClusterID: <SNIP>
 Managers: 3
 Nodes: 3
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 10
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Autolock Managers: false
 Root Rotation In Progress: false
 Node Address: <SNIP>
 Manager Addresses:
  <SNIP>:2377
  <SNIP>:2377
  <SNIP>:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.135-xxxx-std-ipv6-64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.38GiB
Name: <SNIP>
ID: <SNIP>
Docker Root Dir: /home/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

The stack I am trying to launch is an Hyperledger Fabric’s CA which worked like a charm on AWS EC2 instances. Here’s the output of my intermediate CA :

Post https://ca_root:7054/enroll: dial tcp 10.0.9.7:7054: getsockopt: no route to host

So I inspected inside the container ca_root via several ways :

$ docker inspect ca_ca_root.1.47n1a736rvl1p829b8741j8w3
...
"NetworkSettings": {
            "Bridge": "",
            "SandboxID": "59c545167c5978679185b5398c44b06b9a18c54cd5df767213f3e8724636d0df",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "7054/tcp": null
            },
            "SandboxKey": "/var/run/docker/netns/59c545167c59",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "ca_default": {
                    "IPAMConfig": {
                        "IPv4Address": "10.0.9.8"
                    },
                    "Links": null,
                    "Aliases": [
                        "66aa53da2aeb"
                    ],
                    "NetworkID": "wq39rq1oza2cr4ey95thtwbi4",
                    "EndpointID": "308fa4122eeb1320f812168597cea5954624ca886a54d24e14d4214d42045725",
                    "Gateway": "",
                    "IPAddress": "10.0.9.8",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:0a:00:09:08",
                    "DriverOpts": null
                }
            }
        }
...

So IP address is 10.0.9.8, not 10.0.9.7 … and it’s been confirmed inside the network :

$ docker network inspect ca_default
[
    {
        "Name": "ca_default",
        "Id": "wq39rq1oza2cr4ey95thtwbi4",
        "Created": "2018-11-09T13:48:30.97700976Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.9.0/24",
                    "Gateway": "10.0.9.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            <SNIP>
            "66aa53da2aebacdb176140230f1f4abb40c39f60307600d17fca894487866bfa": {
                "Name": "ca_ca_root.1.47n1a736rvl1p829b8741j8w3",
                "EndpointID": "308fa4122eeb1320f812168597cea5954624ca886a54d24e14d4214d42045725",
                "MacAddress": "02:42:0a:00:09:08",
                "IPv4Address": "10.0.9.8/24",
                "IPv6Address": ""
            }
           <SNIP>
            "lb-ca_default": {
                "Name": "ca_default-endpoint",
                "EndpointID": "c931f0c8053966ed242d32824b9b5936bf24c9957ec1557bdeec586b0ef0aad2",
                "MacAddress": "02:42:0a:00:09:04",
                "IPv4Address": "10.0.9.4/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4105"
        },
        "Labels": {
            "com.docker.stack.namespace": "ca"
        },
        "Peers": [
           <SNIP>
        ]
    }
]

I tried to get the IP address inside the container :

$ docker exec -it ca_ca_root.1.47n1a736rvl1p829b8741j8w3 /bin/bash
...
# hostname -I
10.0.9.8 172.19.0.4 

Once again, I obtained 10.0.9.8 … but then, I tried :

$ docker service inspect ca_ca_root
"Endpoint": {
            "Spec": {
                "Mode": "vip"
            },
            "VirtualIPs": [
                {
                    "NetworkID": "wq39rq1oza2cr4ey95thtwbi4",
                    "Addr": "10.0.9.7/24"
                }
            ]
        }

Here is the 10.0.9.7 IP address …

I Honestly do not know how to reproduce this bug and I can not share the docker-compose file.

I already tried to find a solution but they were inapplicable or outdated. Can you help me on this one?

Thanks a lot