Docker overlay network without docker-machine

Hi everyone,

I’m trying to create a swarm cluster with overlay network but, i don’t wanna use docker-machine because i’m using my own hardware and i do not want to use cloud service.

my scenario is:
docker-host-01: host with centos - ip 172.19.232.69
docker-host-02: host with centos - ip 172.19.232.81
docker-host-03: host with centos - ip 172.19.232.91

my docker env(all hosts)

docker version
Client:
 Version:      1.11.0
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   4dc5990
 Built:        Wed Apr 13 18:40:36 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.0
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   4dc5990
 Built:        Wed Apr 13 18:40:36 2016
 OS/Arch:      linux/amd64
[root@host-docker-01 marcos]# 


docker info
 WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 16
Server Version: 1.11.0
Storage Driver: devicemapper
 Pool Name: docker-253:0-50823327-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 1.54 GB
 Data Space Total: 107.4 GB
 Data Space Available: 6.705 GB
 Metadata Space Used: 4.063 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.143 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2015-12-01)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 3.10.0-327.13.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 3
Total Memory: 488.8 MiB
Name: host-docker-01
ID: BBZ3:UZV4:BBCJ:BEZL:KDB2:37A5:7DVP:RYDA:Z6RX:JENV:QZNM:VO6O
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false

uname -a
Linux host-docker-01 3.10.0-327.13.1.el7.x86_64 #1 SMP Thu Mar 31 16:04:38 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

docker -H :4000 info
Containers: 5
 Running: 5
 Paused: 0
 Stopped: 0
Images: 5
Server Version: swarm/1.2.0
Role: primary
Strategy: spread
Filters: health, port, dependency, affinity, constraint
Nodes: 2
docker-host-01: 172.19.232.69:2375
  └ Status: Healthy
  └ Containers: 3
  └ Reserved CPUs: 0 / 1
  └ Reserved Memory: 0 B / 513.5 MiB
  └ Labels: executiondriver=, kernelversion=3.10.0-327.10.1.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=devicemapper
  └ Error: (none)
  └ UpdatedAt: 2016-04-15T12:08:52Z
  └ ServerVersion: 1.11.0
docker-host-02: 172.19.232.81:2375
  └ Status: Healthy
  └ Containers: 2
  └ Reserved CPUs: 0 / 1
  └ Reserved Memory: 0 B / 513.5 MiB
  └ Labels: executiondriver=, kernelversion=3.10.0-327.10.1.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=devicemapper
  └ Error: (none)
  └ UpdatedAt: 2016-04-15T12:09:03Z
  └ ServerVersion: 1.11.0
Plugins: 
 Volume: 
 Network: 
Kernel Version: 3.10.0-327.10.1.el7.x86_64
Operating System: linux
Architecture: amd64
CPUs: 2
Total Memory: 1.003 GiB
Name: 69eb9543f908
Docker Root Dir: 
Debug mode (client): false
Debug mode (server): false
WARNING: No kernel memory limit support

On docker-host-03

  1. I ran docker

     docker daemon -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock 
    
  2. I started consul container

     docker run -d -p 8500:8500 --name consul progrium/consul -server -bootstrap
    

On docker-host-01

  1. I started docker

     docker daemon -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --cluster-store=consul://172.19.232.91:8500  --cluster-advertise=enp0s3:2375 &
    
  2. docker swarm manager

     docker run -d --restart=unless-stopped -p 4000:4000 swarm manage -H :4000 --replication --advertise 172.19.232.69:4000  consul://172.19.232.91:8500
    
  3. add node to cluster

     docker run -d swarm join --advertise=172.19.232.69:2375 consul://172.19.232.91:8500
    

docker ps
[root@localhost marcos]# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS              PORTS                                                                            NAMES
2dc0a3d99b5f        swarm               "/swarm join --advert"   52 seconds ago       Up 52 seconds       2375/tcp                                                                         condescending_visvesvaraya
edb7ea38875c        swarm               "/swarm manage -H :40"   About a minute ago   Up About a minute   2375/tcp, 0.0.0.0:4000->4000/tcp                                                 drunk_brown
8ccb8fd5ac1e        progrium/consul     "/bin/start -server -"   2 minutes ago        Up 2 minutes        53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp   consul

On docker-host-02:

  1. I started docker

     docker daemon -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --cluster-store=consul://172.19.232.91:8500  --cluster-advertise=enp0s3:2375 &
    
  2. add node to cluster

     docker run -d swarm join --advertise=172.19.232.81:2375 consul://172.19.232.91:8500
    

I’ve created a overlay network:

docker -H :4000 network create -d overlay info
docker -H :4000 network create -d overlay info2
docker -H :4000 network create -d overlay teste

on docker-host-01

docker network ls
NETWORK ID          NAME                DRIVER
c22a48c6e1c6        bridge              bridge              
b81f5dd1de16        docker_gwbridge     bridge              
bb2235c6fcf2        host                host                
c8819e13d845        info                overlay             
fd3f780ca95f        info2               overlay             
e16fac408255        none                null                
eac2986e5683        teste               overlay 

on docker-host-02

docker network ls
NETWORK ID          NAME                DRIVER
31f5c2bc3d6a        bridge              bridge              
18cc30825c00        docker_gwbridge     bridge              
a8f035e0def0        host                host                
c8819e13d845        info                overlay             
fd3f780ca95f        info2               overlay             
e8215e819164        none                null                
eac2986e5683        teste               overlay

I’ve created web01 and web02 containers on info2 network

docker -H :4000 run -ti -d --name web01 --net info2 nginx
docker -H :4000 run -ti -d --name web02 --net info2 nginx

docker -H :4000 ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
edeeb902ee41        nginx               "nginx -g 'daemon off"   22 seconds ago      Up 21 seconds       80/tcp, 443/tcp     docker-host-02/web02
0313b6327111        nginx               "nginx -g 'daemon off"   15 hours ago        Up 15 hours         80/tcp, 443/tcp     docker-host-01/web01

docker inspect web01
"NetworkSettings": {
            "Bridge": "",
            "SandboxID": "1e235ca08b87fc2ac236077c3608b2210667bf1ada32b1c608da8376ae2c59bf",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "443/tcp": null,
                "80/tcp": null
            },
            "SandboxKey": "/var/run/docker/netns/1e235ca08b87",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "info2": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "fd3f780ca95ff75d016a199209f7b7f18796d1db95aa87a316bffab4957d55d7",
                    "EndpointID": "f154fd78243e1e17c81270660b743f29376fce702571281dcc08e1f196d9b1fc",
                    "Gateway": "",
                    "IPAddress": "10.0.1.3",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:0a:00:01:03"
                }
            }
        }

docker inspect web02
"NetworkSettings": {
            "Bridge": "",
            "SandboxID": "d5aa486e4e8f0f439dff677fccd266acd7d5f87c421a374f214c69ae17acad26",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "443/tcp": null,
                "80/tcp": null
            },
            "SandboxKey": "/var/run/docker/netns/d5aa486e4e8f",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "info2": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "fd3f780ca95ff75d016a199209f7b7f18796d1db95aa87a316bffab4957d55d7",
                    "EndpointID": "2fbd1fa70cbd9ed551786b06dceca729339ba6a792e45a6b9d146e74dd03f843",
                    "Gateway": "",
                    "IPAddress": "10.0.1.4",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:0a:00:01:04"
                }
            }
        }

but when i try docker exec web02 ping web01

docker exec  web02 ping 10.0.1.3
docker exec  web02 ping 10.0.1.4

these comands didn’t work

I forgot some step ?

What is the output of your ping commands?

docker exec  web02 ping 10.0.1.3
docker exec  web02 ping 10.0.1.4

What happens if you try tcp traffic, like trying to curl one of those containers from another container?

FWIW, you can use Docker Machine with the generic driver to perform the provisioning process on any host where you have SSH access, you don’t have to use a cloud service.

As for your issue as posted, I recall there may have been some issues with overlay / libnetwork and ICMP traffic. Like @programmerq said, I highly suggest trying a different type of traffic e.g. curl. Your setup looks sound to me.

hey Jeff and Nathan,

there is no response from ping commands… no character… nothing…
i’ve created a container, docker run -ti --rm nginx /bin/bash and ping didn’t work inside this containers too. network bug?

ping command was working before swarm setup.

i’ve started from beginning(just in case)… re-install centos, docker and rebuild the cluster (now i’m trying with tls)… after all setup I will try different type of traffic and I will back to comment.

thanks

That sounds more like you are having trouble running the ping process-- ping will always give some output.

What happens if you do something like this?

docker run --name target --net=info2 nginx:alpine
docker run --rm -it --net=info2 alpine ping -c 1 target

hey Jeff,

thank you

I have rebuilt all steps and now is woking

[root@docker-host-02 ~]# docker run --rm -it --net=info2 alpine ping target
PING target (10.0.0.2): 56 data bytes
64 bytes from 10.0.0.2: seq=0 ttl=64 time=0.639 ms
64 bytes from 10.0.0.2: seq=1 ttl=64 time=0.608 ms
64 bytes from 10.0.0.2: seq=2 ttl=64 time=0.709 ms
64 bytes from 10.0.0.2: seq=3 ttl=64 time=0.686 ms
64 bytes from 10.0.0.2: seq=4 ttl=64 time=0.703 ms
64 bytes from 10.0.0.2: seq=5 ttl=64 time=0.668 ms

target is running on docker-host-03 :slight_smile:

1 Like