Embedded DNS - not running

We have a trouble, that the embedded DNS doesn’t listen inside the container.

Test env
2-node swarm cluster (docker1-srva.exprm, docker2-srva.exprm) environment

1) startup of the service

root@docker1-srva.exprm:/# docker service create --publish 8888:80 --name nginx nginx

2) listening services inside the container

  • inside the container listens only the nginx
  • there should be also the dockerd as DNS resolver
root@docker2-srva.exprm:/home/znovak# docker exec -it nginx.1.6zf93j5goi7z3nvtnpta3f4yf netstat -l -t -n -p
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      1/nginx: master pro 

3) network namespace

  • in the container related namespace is NOT any DNS resolver iptables rule
root@docker2-srva.exprm:/run# docker exec -it nginx.1.6zf93j5goi7z3nvtnpta3f4yf ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 10.100.0.76/32 brd 10.100.0.76 scope global lo
       valid_lft forever preferred_lft forever
214: eth0@if215: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether 02:42:0a:64:00:4d brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.100.0.77/16 brd 10.100.255.255 scope global eth0
       valid_lft forever preferred_lft forever
216: eth1@if217: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:12:00:06 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet 172.18.0.6/16 brd 172.18.255.255 scope global eth1
       valid_lft forever preferred_lft forever



root@docker2-srva.exprm:/run# ip netns exec 14b022bbd439 ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 10.100.0.76/32 brd 10.100.0.76 scope global lo
       valid_lft forever preferred_lft forever
214: eth0@if215: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether 02:42:0a:64:00:4d brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.100.0.77/16 brd 10.100.255.255 scope global eth0
       valid_lft forever preferred_lft forever
216: eth1@if217: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:12:00:06 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet 172.18.0.6/16 brd 172.18.255.255 scope global eth1
       valid_lft forever preferred_lft forever



root@docker2-srva.exprm:/run# ip netns exec 14b022bbd439 iptables -t nat -nvL
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 REDIRECT   tcp  --  *      *       0.0.0.0/0            10.100.0.77          tcp dpt:8888 redir ports 80

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 14 packets, 878 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 14 packets, 878 bytes)
 pkts bytes target     prot opt in     out     source               destination

EXPRM environment

  • all instances are on the Debian 9 see bellow
root@docker2-srva.exprm:/run# uname -a
Linux docker2-srva.exprm 4.9.0-8-amd64 #1 SMP Debian 4.9.110-3+deb9u4 (2018-08-21) x86_64 GNU/Linux

Output of docker version:

  • on all nodes is the same version
root@docker2-srva.exprm:/home/znovak# docker version
Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Thu Apr 26 07:17:14 2018
 OS/Arch:      linux/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.5
  Git commit:   9ee9f40
  Built:        Thu Apr 26 07:15:24 2018
  OS/Arch:      linux/amd64
  Experimental: false

Output of docker info:

root@docker2-srva.exprm:/home/znovak# docker version
Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Thu Apr 26 07:17:14 2018
 OS/Arch:      linux/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.5
  Git commit:   9ee9f40
  Built:        Thu Apr 26 07:15:24 2018
  OS/Arch:      linux/amd64
  Experimental: false
root@docker2-srva.exprm:/home/znovak# docker info
Containers: 4
 Running: 4
 Paused: 0
 Stopped: 0
Images: 5
Server Version: 18.03.1-ce
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: ommznbnl88kg7e4u0bao8ug2y
 Is Manager: false
 Node Address: 10.24.2.121
 Manager Addresses:
  10.24.2.120:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.0-8-amd64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.862GiB
Name: docker2-srva.exprm
ID: USJ6:XIYB:BNYS:OI3U:QH4W:LG4N:H5ST:NEPN:CPDZ:MFHT:NKYA:RW6H
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 53
 Goroutines: 132
 System Time: 2018-09-24T12:33:25.636190295+02:00
 EventsListeners: 4
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

The problem was in DNS configuration on host machine:

  • in the /etc/resolv.conf we have defined ndots
  • when you remove ndots configuration, then docker starts embedded DNS resolver
options ndots: 2
1 Like

Thanks @gopay, I encountered the same issue and the tip to check ndots configuration is really helpful.