Overlay Network Issues

General
1

Hi There,

Having some overlay network drama’s! need some assistance please.
Apologies if this is the wrong place to ask this question, happy to be redirected to the right location.

Cannot get docker overlay network to work from docker host to docker host.
Containers with then the same Overlay network cannot ping each other on different docker hosts.
Additionally network inspect reports differently depending upon how it is queried.

Have a docker 3 node swarm Cluster

Server Version: swarm/1.1.3
Role: primary
Strategy: spread
Filters: health, port, dependency, affinity, constraint
Nodes: 3
docker1: 192.168.1.231:2375
└ Status: Healthy
└ Containers: 10
└ Reserved CPUs: 0 / 1
└ Reserved Memory: 0 B / 2.033 GiB
└ Labels: executiondriver=, kernelversion=4.5.0-1.el7.elrepo.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=devicemapper
└ Error: (none)
└ UpdatedAt: 2016-05-03T05:42:52Z
docker2: 192.168.1.35:2375
└ Status: Healthy
└ Containers: 6
└ Reserved CPUs: 0 / 1
└ Reserved Memory: 0 B / 2.033 GiB
└ Labels: executiondriver=, kernelversion=4.5.0-1.el7.elrepo.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=devicemapper
└ Error: (none)
└ UpdatedAt: 2016-05-03T05:43:16Z
docker3: 192.168.1.36:2375
└ Status: Healthy
└ Containers: 4
└ Reserved CPUs: 0 / 1
└ Reserved Memory: 0 B / 2.033 GiB
└ Labels: executiondriver=, kernelversion=4.5.0-1.el7.elrepo.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=devicemapper
└ Error: (none)
└ UpdatedAt: 2016-05-03T05:42:47Z
Plugins:
Volume:
Network:
Kernel Version: 4.5.0-1.el7.elrepo.x86_64
Operating System: linux
Architecture: amd64
CPUs: 3
Total Memory: 6.099 GiB
Name: 107ce6ae3416
Docker Root Dir:
Debug mode (client): false
Debug mode (server): false
WARNING: No kernel memory limit support

Example

Create an Overlay Network

docker network create --driver overlay --subnet=192.168.150.0/24 ORANGE

on Docker node 1
docker run -itd --name orangebox2 --net=ORANGE --hostname=orangebox2 busybox

on Docker node 2
docker run -itd --name orangebox1 --net=ORANGE --hostname=orangebox1 busybox

Neither container can ping each other

docker inspect from both nodes are equal

[
{
“Name”: “ORANGE”,
“Id”: “c6d8d8577b6c71c463478ac624fd8e2a0ee8d8d805b06feebc61850822886840”,
“Scope”: “global”,
“Driver”: “overlay”,
“EnableIPv6”: false,
“IPAM”: {
“Driver”: “default”,
“Options”: {},
“Config”: [
{
“Subnet”: “192.168.150.0/24”
}
]
},
“Internal”: false,
“Containers”: {
“22258e3657d1791633c1d916e068a172d91b8e0c53660d07dcb5011497999117”: {
“Name”: “orangebox2”,
“EndpointID”: “af8ecf3f0c11dbbdf1a365104cbed667c1bfcf4b0c1ecb8f468b35a6d5b21981”,
“MacAddress”: “02:42:c0:a8:96:03”,
“IPv4Address”: “192.168.150.3/24”,
“IPv6Address”: “”
},
“ep-c6cfbaba2103f8eaa47c60e49e55be239effbc67b8e0fa933836a7413c3d0167”: {
“Name”: “orangebox1”,
“EndpointID”: “c6cfbaba2103f8eaa47c60e49e55be239effbc67b8e0fa933836a7413c3d0167”,
“MacAddress”: “02:42:c0:a8:96:02”,
“IPv4Address”: “192.168.150.2/24”,
“IPv6Address”: “”
}
},
“Options”: {},
“Labels”: {}
}
]

Docker inspect from the Cluster however gives repeat values

docker -H :4000 network inspect ORANGE

[
{
“Name”: “ORANGE”,
“Id”: “c6d8d8577b6c71c463478ac624fd8e2a0ee8d8d805b06feebc61850822886840”,
“Scope”: “global”,
“Driver”: “overlay”,
“EnableIPv6”: false,
“IPAM”: {
“Driver”: “default”,
“Options”: {},
“Config”: [
{
“Subnet”: “192.168.150.0/24”
}
]
},
“Internal”: false,
“Containers”: {
“22258e3657d1791633c1d916e068a172d91b8e0c53660d07dcb5011497999117”: {
“Name”: “orangebox2”,
“EndpointID”: “af8ecf3f0c11dbbdf1a365104cbed667c1bfcf4b0c1ecb8f468b35a6d5b21981”,
“MacAddress”: “02:42:c0:a8:96:03”,
“IPv4Address”: “192.168.150.3/24”,
“IPv6Address”: “”
},
“8335c3b18b145d26427374f3925b3907e3604c47c8b9674ddf98b4c5f439dd5f”: {
“Name”: “orangebox1”,
“EndpointID”: “c6cfbaba2103f8eaa47c60e49e55be239effbc67b8e0fa933836a7413c3d0167”,
“MacAddress”: “02:42:c0:a8:96:02”,
“IPv4Address”: “192.168.150.2/24”,
“IPv6Address”: “”
},
“ep-af8ecf3f0c11dbbdf1a365104cbed667c1bfcf4b0c1ecb8f468b35a6d5b21981”: {
“Name”: “orangebox2”,
“EndpointID”: “af8ecf3f0c11dbbdf1a365104cbed667c1bfcf4b0c1ecb8f468b35a6d5b21981”,
“MacAddress”: “02:42:c0:a8:96:03”,
“IPv4Address”: “192.168.150.3/24”,
“IPv6Address”: “”
},
“ep-c6cfbaba2103f8eaa47c60e49e55be239effbc67b8e0fa933836a7413c3d0167”: {
“Name”: “orangebox1”,
“EndpointID”: “c6cfbaba2103f8eaa47c60e49e55be239effbc67b8e0fa933836a7413c3d0167”,
“MacAddress”: “02:42:c0:a8:96:02”,
“IPv4Address”: “192.168.150.2/24”,
“IPv6Address”: “”
}
},
“Options”: {},
“Labels”: null
}
]

This is on CentOS7 Kernel 4.5.0-1.el7.elrepo.x86_64

The Docker Daemon is started with the following options to support Overlay Networks

Example from docker1

[Service]ExecStart=ExecStart=/usr/bin/docker daemon -H fd:// -H tcp://192.168.1.231:2375 --cluster-store=consul://192.168.1.231:8500 --cluster-advertise=192.168.1.231:2375

Thanks

2

Does non-ICMP protocol work, or is it all connectivity having issues? How about result of dig or drill on the DNS aliases?

3

Hi Nathan,

Thanks for taking a look.

I have 3 centos6 containers, one on each of the docker hosts, to aid in debugging.

I can’t SSH between any of the containers either. I have installed SSH server of course and I can ssh to localhost or from the docker host server directly into a container via the 172.18.0.0 network.

Dig works fine:

[root@orangebox1 /]# dig orangebox2

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.37.rc1.el6_7.7 <<>> orangebox2
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43175
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;orangebox2.                    IN      A

;; ANSWER SECTION:
orangebox2.             600     IN      A       192.168.150.3

;; Query time: 10 msec
;; SERVER: 127.0.0.11#53(127.0.0.11)
;; WHEN: Wed May  4 23:22:32 2016
;; MSG SIZE  rcvd: 54

Hopefully my understanding is correct:
Each container has 2 interfaces, one on the docker bridge network (172.18.0.0) and the other on the
Overlay network (ORANGE in my case)
eth0 - 192.168.150.2
eth1 - 172.18.0.2

/etc/hosts is populated with the local name entry
192.168.150.2 orangebox1
172.18.0.2 orangebox1

and finally the route
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 172.18.0.1 0.0.0.0 UG 0 0 0 eth1
172.18.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth1
192.168.150.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

Not sure what I need to check next?
Thanks for your help.
Keith.

4

Lab-1
I have 3 Centos7 servers running Docker engine. Each Centos7 Server is a VMware Virtual Machine, running with a Bridged Network on VMware Workstation.
Here the overlay network fails

Lab-2
Copied the Same Virtual Machines onto the same physical VMware workstation server, changed the network type to a be local vmnet and the overlay network works.

I need to figure out how this should be correctly configured