DNS lookup does not work for services running in different hosts

Docker for AWS
,
1

Expected behavior

Every service can ping any service no matter where in the Swarm they are running

Actual behavior

If the services are not running on the same host, the lookup does not work

Additional Information

I believe this is a problem with user defined overlay networks in AWS

NETWORK ID NAME DRIVER SCOPE
9af7ac6ef099 ridge bridge local
3935dd174aab docker_gwbridge bridge local
wpmbm7cagpl2 default overlay swarm
k9jfupa8z5wx elk overlay swarm
bl97hjqlzdqq monitoring overlay swarm
4b0880738192 host host local
j0sikz1i0mar ingress overlay swarm
234d1c386fc3 none null local
jh87m4u8hyia proxy overlay swarm

Docker CE for AWS 17.06.0-ce (17.06.0-ce-aws2)

This is happening even if the destiny of the lookup is running on a Manager or on a Worker

Steps to reproduce the behavior

Let the Swarm decide where to put the services, if they are in different node, problem appears
Use deploy to constrain where the services are running, if in the same host, dns resolution work

  1. …
  2. …
2

Things are getting just more and more weird:

redis1:
image: redis:4-alpine
networks:
- default
ports:
- 6379

example:
image: cc250080/daimler-be:0.0.1.132
ports:
- 8080
networks:
- default
- proxy

Both on the same node, a manager, both in the overlay network “default”.

#docker exec -ti “EXAMPLE” /bin/bash
$ping redis1
redis1: unknown host

root@872e3d2641fd:/# nslookup redis1
Server: 127.0.0.11
Address: 127.0.0.11#53

** server can’t find redis1: NXDOMAIN

Maybe is the service discovey mechanism what has got broken? The DNS server the Swarm configures is reachable and working

3

More info…

I had add “extra_hosts” on the compose file, so the “example” container is now able to ping redis1.

Nevertheless, it is not able to connect to it, it gets connection refused:

root@6b925258ce12:/# telnet redis1 6379
Trying 10.0.1.20…
telnet: Unable to connect to remote host: Connection refused

But, redis, by looking at its logs and at netstat, seems correctly awaiting for connections:

~ # docker service logs -f exampe_redis1

redis1.1.efg0xyp17y8t@ip-172-28-44-153.eu-central-1.compute.internal | 1:M 07 Sep 08:13:06.143 * Ready to accept connections

~ # docker exec -ti “08a9bceee39f” netstat -nlp | grep 6379
tcp 0 0 0.0.0.0:6379 0.0.0.0:* LISTEN -
tcp 0 0 :::6379 :::* LISTEN -

So… Networking is broken?