Can't reach other container through overlay network

General
1

Hello,

I have the following architecture:

5 nodes in a cluster;
1 overlay network named “mx” subnet 172.16.0.0/24;
each host is a Consul client or server;
using --cluster-store consul://127.0.0.1:8500 and --cluster-advertise=10.4.0.192:2375;

host #2 runs a container named broker2, which is under the overlay network mx.
all other hosts can reach it and communicate to it though the network, except host #1.

When I start a container on host #1 and ping “broker2”, I can’t reach it:

docker exec -ti emailprocess-worker_192 ping broker2
PING broker2 (172.16.0.33) 56(84) bytes of data.

but I can ping any other container of any other host from this container.

If I do the reverse path, ping emailprocess-worker_192 from broker2, it succeeds:

docker exec -ti broker2 ping emailprocess-worker_192
PING emailprocess-worker_192 (172.16.0.40): 56 data bytes
64 bytes from 172.16.0.40: seq=0 ttl=64 time=0.358 ms
64 bytes from 172.16.0.40: seq=1 ttl=64 time=0.192 ms

and then, after this, the ping from emailprocess-worker_192 to broker2 starts working:

docker exec -ti emailprocess-worker_192 ping broker2
PING broker2 (172.16.0.33) 56(84) bytes of data.
64 bytes from broker2.mx (172.16.0.33): icmp_seq=1 ttl=64 time=0.214 ms
64 bytes from broker2.mx (172.16.0.33): icmp_seq=2 ttl=64 time=0.249 ms

until I stop/start the container, when it can’t ping broker2 anymore.

Docker version 1.12.1, build 23cf638
OS: Ubuntu Server 14.04 x64

2

I’m having the same issue. Have been looking all over for answers and tried multiple different setups.

3

yep, no answers all over the web. problem still happens, so I have to stop the entire cluster and recreate the networks.

4

In my home setup I have created networks for different “service groups” accessed via different stacks,

docker network create --internal --attachable -d overlay --scope swarm public
docker network create --internal --attachable -d overlay --scope swarm intranet
docker network create --attachable -d overlay --scope swarm backend

Then say I have an public facing container I attach it to the public network from a compose file llke

version: '3.4'
services:
  site:
    image: trajano.net/site
    networks:
    - public
networks:
  public:
    external:
      name: public

In my Apache (not nginx because nginx charges extra for allowing missing upstreams and recovering from it on startup) proxy container I also put it in the public network and the default one so I can expose the port.

version: '3.4'
services:
  web:
    image: trajano.net/web
    ports:
    - "80:80"
    - "443:443"
    networks: 
    - public
    - default
networks:
  public:
    external:
      name: public

Then in my Apache config I have

ProxyPass "/site "http://site"
ProxyPassReverse "/site "http://site"
RewriteRule  ^/site$ /site/ [R]