Docker Community Forums

Share and learn in the Docker community.

DNS resolution not working in containers

This is a duplicate of the Stack Overflow post here, where I have not received an answer.

I have a docker container and I am unable to run DNS lookups from inside containers, although it works fine from the docker host.

The configuration management code that builds the Docker host is known to work on a standard RHEL 7 image from the marketplace, therefore the problem is known to be something inside the SOE RHEL 7 image.

RHEL 7.2 / Docker version 1.12.6, build 88a4867/1.12.6. Container is RHEL 7.3. SELinux in enabled/permissive mode.

Some config:

# /etc/sysconfig/docker
OPTIONS='--dns=10.0.0.10 --dns=10.0.0.11 --dns-search=example.com'
DOCKER_CERT_PATH=/etc/docker
ADD_REGISTRY='--add-registry registory.example.com'
no_proxy=169.254.169.254,localhost,127.0.0.1,registory.example.com
http_proxy=http://proxy.example.com:8080
https_proxy=http://proxy.example.com:8080
ftp_proxy=http://proxy.example.com:8080

Resolver config in the container and host is the same:

# /etc/resolv.conf
search example.com
nameserver 10.0.0.10
nameserver 10.0.0.11

If I restart the docker daemon with --debug I see the following in journalctl -u docker.service:

Aug 08 11:44:23 myhost.example.com dockerd-current[17341]: time="2017-08-08T11:44:23.430769581+10:00" level=debug msg="Name To resolve: http://proxy.example.com."
Aug 08 11:44:23 myhost.example.com dockerd-current[17341]: time="2017-08-08T11:44:23.431488213+10:00" level=debug msg="Query http://proxy.example.com.[1] from 172.18.0.6:38189, forwarding to udp:10.162.182.101"
Aug 08 11:44:27 myhost.example.com dockerd-current[17341]: time="2017-08-08T11:44:27.431772666+10:00" level=debug msg="Read from DNS server failed, read udp 172.18.0.6:38189->10.162.182.101:53: i/o timeout"

Following that observation further, it turns out I can get some networking to work if I specify an IP address instead of the DNS name of the proxy; although that really is just a way of avoiding using DNS and not a real fix.

I noted suspicious-looking rules in iptables:

# iptables -n -L DOCKER-ISOLATION -v --line-numbers
Chain DOCKER-ISOLATION (1 references)
num   pkts bytes target     prot opt in     out     source               destination         
1        0     0 DROP       all  --  br-1d6a05c10468 docker0  0.0.0.0/0            0.0.0.0/0           
2        0     0 DROP       all  --  docker0 br-1d6a05c10468  0.0.0.0/0            0.0.0.0/0           
3    34903   11M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

However after deleting those two DROP rules I continued to see the issue.

Full iptables

# iptables -nL -v
Chain INPUT (policy ACCEPT 2518 packets, 1158K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
23348 9674K DOCKER-ISOLATION  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0           
23244 9667K DOCKER     all  --  *      br-1d6a05c10468  0.0.0.0/0            0.0.0.0/0           
23232 9667K ACCEPT     all  --  *      br-1d6a05c10468  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
  104  6230 ACCEPT     all  --  br-1d6a05c10468 !br-1d6a05c10468  0.0.0.0/0            0.0.0.0/0           
   12   700 ACCEPT     all  --  br-1d6a05c10468 br-1d6a05c10468  0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 2531 packets, 414K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp  --  !br-1d6a05c10468 br-1d6a05c10468  0.0.0.0/0            172.18.0.2           tcp dpt:443
    0     0 ACCEPT     tcp  --  !br-1d6a05c10468 br-1d6a05c10468  0.0.0.0/0            172.18.0.2           tcp dpt:80
    0     0 ACCEPT     tcp  --  !br-1d6a05c10468 br-1d6a05c10468  0.0.0.0/0            172.18.0.3           tcp dpt:389

Chain DOCKER-ISOLATION (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  br-1d6a05c10468 docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 DROP       all  --  docker0 br-1d6a05c10468  0.0.0.0/0            0.0.0.0/0           
23348 9674K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Bridge config

# ip addr show docker0
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether 02:42:a8:73:db:bb brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
# ip addr show br-1d6a05c10468
3: br-1d6a05c10468: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether 02:42:d5:b6:2d:f5 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 scope global br-1d6a05c10468
       valid_lft forever preferred_lft forever

and

# docker network inspect bridge 
[
    {
        "Name": "bridge",
        "Id": "e159ddd37386cac91e0d011ade99a51f9fe887b8d32d212884beace67483af44",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

In the logs:

Aug 04 17:33:32 myhost.example.com systemd[1]: Starting Docker Application Container Engine...
Aug 04 17:33:33 myhost.example.com dockerd-current[2131]: time="2017-08-04T17:33:33.056770003+10:00" level=info msg="libcontainerd: new containerd process, pid: 2140"
Aug 04 17:33:34 myhost.example.com dockerd-current[2131]: time="2017-08-04T17:33:34.740346421+10:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Aug 04 17:33:34 myhost.example.com dockerd-current[2131]: time="2017-08-04T17:33:34.741164354+10:00" level=info msg="Loading containers: start."
Aug 04 17:33:34 myhost.example.com dockerd-current[2131]: .........................time="2017-08-04T17:33:34.903371015+10:00" level=info msg="Firewalld running: true"
Aug 04 17:33:35 myhost.example.com dockerd-current[2131]: time="2017-08-04T17:33:35.325581993+10:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address" 
Aug 04 17:33:36 myhost.example.com dockerd-current[2131]: time="2017-08-04T17:33:36+10:00" level=info msg="Firewalld running: true"
Aug 04 17:33:37 myhost.example.com dockerd-current[2131]: time="2017-08-04T17:33:37+10:00" level=info msg="Firewalld running: true"
Aug 04 17:33:37 myhost.example.com dockerd-current[2131]: time="2017-08-04T17:33:37+10:00" level=info msg="Firewalld running: true"
Aug 04 17:33:38 myhost.example.com dockerd-current[2131]: time="2017-08-04T17:33:38+10:00" level=info msg="Firewalld running: true"
Aug 04 17:33:39 myhost.example.com dockerd-current[2131]: time="2017-08-04T17:33:39+10:00" level=info msg="Firewalld running: true"
Aug 04 17:33:40 myhost.example.com dockerd-current[2131]: time="2017-08-04T17:33:40+10:00" level=info msg="Firewalld running: true"
Aug 04 17:33:40 myhost.example.com dockerd-current[2131]: time="2017-08-04T17:33:40+10:00" level=info msg="Firewalld running: true"
Aug 04 17:33:42 myhost.example.com dockerd-current[2131]: time="2017-08-04T17:33:42+10:00" level=info msg="Firewalld running: true"
Aug 04 17:33:42 myhost.example.com dockerd-current[2131]: time="2017-08-04T17:33:42+10:00" level=info msg="Firewalld running: true"
Aug 04 17:33:43 myhost.example.com dockerd-current[2131]: time="2017-08-04T17:33:43.541905145+10:00" level=info msg="Loading containers: done."
Aug 04 17:33:43 myhost.example.com dockerd-current[2131]: time="2017-08-04T17:33:43.541975618+10:00" level=info msg="Daemon has completed initialization"
Aug 04 17:33:43 myhost.example.com dockerd-current[2131]: time="2017-08-04T17:33:43.541998095+10:00" level=info msg="Docker daemon" commit="88a4867/1.12.6" graphdriver=devicemapper version=1.12.6
Aug 04 17:33:43 myhost.example.com dockerd-current[2131]: time="2017-08-04T17:33:43.548508756+10:00" level=info msg="API listen on /var/run/docker.sock"
Aug 04 17:33:43 myhost.example.com systemd[1]: Started Docker Application Container Engine.

From the container, I can ping the default gateway but all name resolution fails.

I noticed one weird thing in the log, but I now know that this is a red herring - see response at the Stack Overflow post):

# journalctl -u docker.service |grep insmod > /tmp/log # \n's replaced with real newlines
Jul 26 23:59:02 myhost.example.com dockerd-current[3185]: time="2017-07-26T23:59:02.056295890+10:00" level=warning msg="Running modprobe bridge br_netfilter failed with message: insmod /lib/modules/3.10.0-514.26.2.el7.x86_64/kernel/net/bridge/bridge.ko 
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-arptables: No such file or directory
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory
modprobe: ERROR: Error running install command for bridge
modprobe: ERROR: could not insert 'bridge': Unknown error 253
insmod /lib/modules/3.10.0-514.26.2.el7.x86_64/kernel/net/llc/llc.ko 
insmod /lib/modules/3.10.0-514.26.2.el7.x86_64/kernel/net/802/stp.ko 
install /sbin/modprobe --ignore-install bridge && /sbin/sysctl -q -w net.bridge.bridge-nf-call-arptables=0 net.bridge.bridge-nf-call-iptables=0 net.bridge.bridge-nf-call-ip6tables=0 
insmod /lib/modules/3.10.0-514.26.2.el7.x86_64/kernel/net/bridge/br_netfilter.ko 
, error: exit status 1"

Update: and this is coming from:

# tail -2 /etc/modprobe.d/dist.conf
# Disable netfilter on bridges when the bridge module is loaded
install bridge /sbin/modprobe --ignore-install bridge && /sbin/sysctl -q -w net.bridge.bridge-nf-call-arptables=0 net.bridge.bridge-nf-call-iptables=0 net.bridge.bridge-nf-call-ip6tables=0

Also:

# cat /proc/sys/net/bridge/bridge-nf-call-{arp,ip,ip6}tables
1
1
1

However, even after I do this:

# for i in /proc/sys/net/bridge/bridge-nf-call-{arp,ip,ip6}tables ; do echo 0 > $i ; done
Still no luck.

I have spent several days on this now; any help much appreciated.

I now know the issue affects UDP only and I can work-around the issue using:

# head -1 /etc/sysconfig/docker
OPTIONS="--dns=10.0.0.10 --dns=10.0.0.11 --dns-search=example.com --dns-opt=use-vc"

Specifying use-vc in resolv.conf tells the resolver to use TCP instead of UDP.

Thank you so much alex