Docker pull not using correct DNS server when private registry on VPN

I just started using the Docker for Mac beta, but immediately ran into an issue pulling from my company’s private registry which is hosted inside our VPN. It seems the docker daemon is trying to use Google’s DNS servers instead of the DNS servers set by the VPN.

Expected behavior

Docker pull should work when registry is inside VPN.

Actual behavior

I noticed that it tries to use Google’s DNS server (8.8.4.4) to perform name resolution:

$ docker pull docker.mycompany.io/alpine:3.3
Error response from daemon: Get https://docker.mycompany.io/v1/_ping: dial tcp: lookup docker.mycompany.io on 8.8.4.4:53: no such host

I also tried restarted Docker after logging into the VPN (I noticed that restart is needed every time I change networks).

Information

From OS X, you can see resolv.conf has been updated to the VPN’s name servers:

$ cat /etc/resolv.conf
#
# Mac OS X Notice
#
# This file is not used by the host name and address resolution
# or the DNS query routing mechanisms used by most processes on
# this Mac OS X system.
#
# This file is automatically generated.
#
search eng.mycompany.com
nameserver 172.31.18.70
nameserver 172.31.27.35
nameserver 172.31.47.153

And I can perform name resolution

$ nslookup docker.mycompany.io
Server:         172.31.27.35
Address:        172.31.27.35#53

Name:   docker.mycompany.io
Address: 172.31.20.162

$ pinata diagnose -u
OS X: version 10.11.4 (build: 15E65)
Docker.app: version v1.11.1-beta10
Running diagnostic tests:
[OK]      docker-cli
[OK]      Moby booted
[OK]      driver.amd64-linux
[OK]      vmnetd
[OK]      osxfs
[OK]      db
[OK]      slirp
[OK]      menubar
[OK]      environment
[OK]      Docker
[OK]      VT-x
Docker logs are being collected into /tmp/20160505-122624.tar.gz
Most specific failure is: No error was detected
Your unique id is: 5ED5A8AB-0944-4019-BEF3-2D11D3EE6FF9
Please quote this in all correspondence.

Steps to reproduce the behavior

  1. Setup docker registry on private network
  2. VPN into the private network
  3. Issue a ‘docker pull’ command on an image in the registry
4 Likes

For anyone else who hits this issue, you can workaround this by accessing the Docker VM directly and modifying the configuration inside the VM:

  1. Access the terminal of the Docker Alpine VM using GNU screen:
    $ screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty

  2. Login as user root. No password necessary

  3. Edit /etc/hosts and/or /etc/resolv.conf such that you can resolve your internal registry server from inside the Docker VM

  4. To quit the screen app, type CTRL-A, then CTRL-\

Note that these changes survive until Docker daemon is restarted. After which the steps will need to be performed again.

8 Likes

The next problem you will likely run into after fixing DNS issues, is that your private registry is untrusted and does not have a signed certificate. To get past this:

Edit /etc/init.d/docker and add --insecure-registry <registry_hostname> to the DOCKER_OPTS line.

Credit goes here:

To those that are experiencing this issue:

  1. What’s the contents of /etc/resolv.conf on your Mac

  2. How does this compare to scutil --dns ?

  3. Which nameservers did you expect to be used?

The reason I ask is that we currently use resolv.conf on the Mac and reports from most users have been that it’s contents matches the DNS servers they expect to be used when connected to VPN. If that’s not the case, please let us know!

  1. my /etc/resolv.conf:

nameserver 8.8.8.8 nameserver 8.8.4.4

  1. scutil --dns:

DNS configuration (for scoped queries)
resolver #1 nameserver[0] : 8.8.8.8 nameserver[1] : 8.8.4.4 if_index : 4 (en0) flags : Scoped, Request A records Reachable

resolver #2 search domain[0] : tfly-internal.com search domain[1] : sc.tfly-internal.com search domain[2] : aws.tfly-internal.com nameserver[0] : 10.30.9.94 nameserver[1] : 10.30.9.60 if_index : 10 (utun0) flags : Scoped, Request A records Reachable, Directly Reachable Address order : 1

  1. I expect the Docker daemon to use our internal DNS server, since our Docker registry host name is internal.
1 Like

Hi Dave,

Why not have a Forum note which describes the configs from Host <->
Machine/VM <-> Containers.
Please include details of what ‘<->’ could be and why/when it would be
used? Unless there is one
already?

Regards,

There is a plan to publish some more details on what’s happening under the hood in Docker for Mac, but there aren’t any concrete timelines for that yet…

Awesome! Thanks @gcarre!

We now have issue (#3124) tracking this on our bug tracker and we hope to have a fix available soon! More reports will be helpful though so please keep them coming!

1 Like

I’m getting the same problems simply trying to use Docker4Mac on my company network and trying to access our company registry. My mac’s nameservers are correctly set to 192.168.0.47 and 192.168.0.48 but if I ‘screen’ into the docker alpine linux vm the /etc/resolv.conf file contains only the google nameservers. As a result, pulls from the internal registry fail.

From what other people have said, is the Docker4Mac designed to pick up the nameserver config from the host Mac and replicate it in the VM?

One thing I did note that might have an impact is that I’m using a MacBook Air with an ethernet adaptor which means en0 is the wifi card (which is off) and my normal network connection is on en3.

My network setup is ‘nat’ and native/port-forwarding is true. Using hostnet for my network setup caused Docker4Mac to fail to start (maybe related to the en0 / en3 stuff)

Here is my OS X /etc/resolv.conf and scutil --dns (company name obfuscated):

$ cat /etc/resolv.conf
#
# Mac OS X Notice
#
# This file is not used by the host name and address resolution
# or the DNS query routing mechanisms used by most processes on
# this Mac OS X system.
#
# This file is automatically generated.
#
search eng.mycompany.com
nameserver 172.31.18.70
nameserver 172.31.27.35
nameserver 172.31.47.153

$ scutil --dns
DNS configuration

resolver #1
  search domain[0] : eng.mycompany.com
  nameserver[0] : 172.31.18.70
  nameserver[1] : 172.31.27.35
  nameserver[2] : 172.31.47.153
  flags    : Request A records
Reachable

resolver #2
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records
Not Reachable
  order    : 300000

resolver #3
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
Not Reachable
  order    : 300200

resolver #4
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
Not Reachable
  order    : 300400

resolver #5
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
Not Reachable
  order    : 300600

resolver #6
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
Not Reachable
  order    : 300800

resolver #7
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
Not Reachable
  order    : 301000

DNS configuration (for scoped queries)

resolver #1
  search domain[0] : eng.mycompany.com
  nameserver[0] : 172.31.18.70
  nameserver[1] : 172.31.27.35
  nameserver[2] : 172.31.47.153
  if_index : 4 (en0)
  flags    : Scoped, Request A records
Reachable

Here is Alpine Docker’s /etc/resolv.conf after restarting Docker after logging into our VPN:

moby:~# cat /etc/resolv.conf
nameserver 8.8.8.8
nameserver 8.8.4.4

This seems broken to me. Could you not somehow leverage OSX’s name resolution? Proxy requests through to that? Otherwise…I forsee further acrobatics. If I want custom settings, I have to console into the VM and modify /etc/hosts, each time the system is restarted? It’s my local box…I’m not looking for DNS resolution, I’m looking for local name resolution.

Something seems iffy with how docker is resolving hostnames. In the example below, while the OS can resolve registry.blah.com, it looks like docker-engine can’t. If I change the hosts entry to just “registry” it works just fine.

moby:~# grep registry /etc/hosts
127.0.0.1       localhost registry.blah.com
moby:~# docker tag 8b162eee2794 registry.blah.com:5000/registry
moby:~# docker push registry.blah.com:5000/registry
The push refers to a repository [registry.blah.com:5000/registry]
Get https://registry.blah.com:5000/v1/_ping: dial tcp: lookup registry.blah.com on 192.168.65.1:53: no such host
moby:~# ping -c 1 registry.blah.com
PING registry.blah.com (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: seq=0 ttl=64 time=0.101 ms

--- registry.blah.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.101/0.101/0.101 ms
2 Likes

Hi Dave, where can we go to see updates on defect #3124?

As far as I know, the bug tracker is not public so you won’t be able to follow on updates. However, if the issue is fixed you should see the issue number in the release notes.

I am facing the same problem with Version 1.11.1-beta14 (build: 8670). When I hop on VPN i get this

➜ ~ docker pull redis Using default tag: latest Pulling repository docker.io/library/redis Network timed out while trying to connect to https://index.docker.io/v1/repositories/library/redis/images. You may want to check your internet connection or if you are behind a proxy.

Fingers crossed #3124 gets into the next release candidate, this causes me real pain when working from home

Facing the same issue with Docker version 1.12.0-rc2, build 906eacd, experimental, encountered via https://forums.docker.com/t/docker-pull-timeout/15015:

$ docker pull hello-world

Using default tag: latest
Pulling repository docker.io/library/hello-world
Network timed out while trying to connect to https://index.docker.io/v1/repositories/library/hello-world/images. You may want to check your internet connection or if you are behind a proxy.

$ cat /etc/resolv.conf

search intra.company.io
nameserver 10.9.69.70
nameserver 8.8.8.8

$ scutil --dns

DNS configuration

resolver #1
  search domain[0] : intra.company.io
  nameserver[0] : 10.9.69.70
  nameserver[1] : 8.8.8.8
  flags    : Request A records
Reachable

resolver #2
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records
Not Reachable
  order    : 300000

resolver #3
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
Not Reachable
  order    : 300200

resolver #4
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
Not Reachable
  order    : 300400

resolver #5
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
Not Reachable
  order    : 300600

resolver #6
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
Not Reachable
  order    : 300800

resolver #7
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
Not Reachable
  order    : 301000

DNS configuration (for scoped queries)

resolver #1
  search domain[0] : intra.company.io
  nameserver[0] : 10.9.69.70
  nameserver[1] : 8.8.8.8
  if_index : 5 (en1)
  flags    : Scoped, Request A records
Reachable

$ screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty

$ cat /etc/resolv.conf

search local
nameserver 192.168.65.1

Adding 8.8.8.8 to the VM image’s resolv.conf fixes the issue. I don’t know where 192.168.65.1 is coming from.

I would expect to be able to pull images both from index.docker.io and from docker.intra.company.io, without messing with any details in the VM image. I’d also expect to be able to pull from docker.myhomenetwork.local if I connect to that via VPN.

When using Docker Toolbox, its possible to configure the Virtualbox VM to use the host’s resolver. This gets around all of these problems. Any thing the Mac host can resolve, including /etc/hosts entries and domain specific DNS settings in /etc/resolver are automatically picked up, even when changed on the fly.

You do this by stopping the default VM, then the following before starting it again.
VBoxManage modifyvm default --natdnshostresolver1 on

The ability to do something similar would be great. Even better, have it happen automatically since the aim here is to have the xhyve VM as largely transparent. I assume this is something xhyve would need to be able to do.

1 Like

DNS resolution seems to have been fixed. With the current version 1.12.0-rc4-beta20 (build: 10404), I no longer have to apply any workarounds inside the moby VM to push/pull from our internal registry in our VPN.

Nope, doesn’t work for me, I still get an “i/o timeout” trying to login to our internal private registry using the latest “Docker for MAC” Beta - Version 1.12.0-rc4-beta20 (build: 10404) … it still is trying the wrong (external fallback) IP to communicate via https

Not working for me either in 1.12.0-rc4-beta20 (build: 10404)