Docker for Mac & Host VPN DNS don't cooperate

Expected behavior

When connecting to an internally hosted registry over company VPN, I expect the Docker daemon (and running containers) to respect the host’s DNS configuration to look up the hostname.

Actual behavior

The Docker daemon is unable to resolve the hostname for the internal registry, which results in me being unable to pull the containers that I need.

IP addresses and hosts have been obfuscated:

> docker pull my.registry.internal.company.com/some/container
Error response from daemon: unable to ping registry endpoint https://my.registry.internal.company.com/v0/
v2 ping attempt failed with error: Get https://my.registry.internal.company.com/v2/: dial tcp: lookup my.registry.internal.company.com on 169.254.0.1:53: no such host
 v1 ping attempt failed with error: Get https://my.registry.internal.company.com/v1/_ping: dial tcp: lookupmy.registry.internal.company.com on 169.254.0.1:53: no such host

Information

I do have “VPN Compatibility Mode” enabled in Settings.

Debug Output:

nick@Nicholass-MacBook-Pro ~> pinata diagnose -u
OS X: version 10.11.4 (build: 15E65)
Docker.app: version v1.10.3-beta5
Running diagnostic tests:
[OK]      docker-cli
[OK]      Moby booted
[OK]      driver.amd64-linux
[OK]      vmnetd
[OK]      lofs
[OK]      osxfs
[OK]      db
[OK]      slirp
[OK]      menubar
[OK]      environment
[OK]      Docker
[OK]      VT-x

A snippet of my relavent DNS configuration:

nick@Nicholass-MacBook-Pro ~> scutil --dns
DNS configuration

resolver #1
  search domain[0] : internal.company.com
  search domain[1] : consul
  nameserver[0] : 8.8.8.8
  nameserver[1] : 208.67.222.222
  nameserver[2] : 8.8.4.4
  if_index : 4 (en0)
  flags    : Request A records
Reachable

resolver #2
  domain   : internal.company.com
  nameserver[0] : $COMPANY_DNS_SERVER1
  nameserver[1] : $COMPANY_DNS_SERVER2
  flags    : Request A records
Reachable
  order    : 100600

The $COMPANY_DNS_SERVER addresses are not in /etc/resolv.conf

A curl of the endpoint works:

> curl https://my.registry.internal.company.com/v2/
{}

Unfortunately, I don’t have a great reproduction case as it likely involves a complicated DNS setup that can’t be replicated without fairly specific infrastructure. If I can lend a hand at all at debugging this, please let me know!

6 Likes

I am in the same boat. With docker-machine and a virtualbox vm, I fix this issue with:

      VBoxManage modifyvm $machine \
        --natdnsproxy1 on \
        --natdnshostresolver1 on

Is there some equivalent configuration I can set with docker native for mac?

1 Like

I just have the same problem, as this post I created No such host for registry with a custom hostname . But I still don’t know how to solve this

I bumped into this issue as well, but discovered a workaround that works until next Docker daemon restart.

https://forums.docker.com/t/docker-pull-not-using-correct-dns-server-when-private-registry-on-vpn/11117/2?u=jessesuen

3 Likes

I have also bumped into this issue. My stats:

OS X: version 10.11.4 (build: 15E65)
Docker.app: version v1.11.1-beta12
Running diagnostic tests:
[OK]      Moby booted
[OK]      driver.amd64-linux
[OK]      vmnetd
[OK]      osxfs
[OK]      db
[OK]      slirp
[OK]      menubar
[OK]      environment
[OK]      Docker
[OK]      VT-x

I followed the instructions from the last user, am able to login into the VM, but adding DNS server to /etc/resolve.conf does not allow the name resolution to work; the only method I found to work is to manually add entries into /etc/hosts :frowning:

Same problem. we don’t have private registry, but our dns server responds to every query.

Examples:

% docker pull debian                                                                                              
Using default tag: latest
Pulling repository docker.io/library/debian
Error while pulling image: Get https://index.docker.io/v1/repositories/library/debian/images: dial tcp: lookup index.docker.io on 192.168.65.1:53: server misbehaving

Trying to add a dns server to new container

Step 5 : RUN echo "nameserver 8.8.8.8" > /etc/resolv.conf
 ---> Using cache
 ---> a34ca48b238a
Step 6 : RUN cat /etc/resolv.conf
 ---> Running in 9715fece76d9
search local
nameserver 192.168.65.1

I also hit this problem when trying to run docker when at home and try connecting go the work registries through the VPN.

nice workaround, the issue with this is that it’s not technically correct as this is more of a DNS fallback rather than a true split dns. I’m guessing docker folk will need to add support for a type of DNS forwarding to the outer host, or replicate the split DNS functionality in the vm.

I’m having the same issue as well. Does anyone know how to solve this?

For what it’s worth, we happen to have two VPN options when we connect at work: the typical configuration, and a split configuration. For the split config, it would prevent all traffic not internal to the company’s network through the local network configuration. However, that’s a configuration determined on the server end, not on the host.

For security reasons, our split-network configuration is going away, and once we tried the standard one we realized none of us could access our Docker containers as we used to. I’m not sure how deep anyone dug, but our resolution was to switch VPN clients.

We blamed the Cisco AnyConnect client for being unable to properly manage this. We tried using the OSS openconnect client, easily downloaded via homebrew and invoked when needed:

eval `openconnect --authenticate https://<your-vpn-address>`;
   [ -n $COOKIE ] && echo $COOKIE |
   sudo openconnect --cookie-on-stdin $HOST --servercert $FINGERPRINT --pid-file=/usr/local/var/run/openconnect.pid

End the VPN connection later with:

sudo kill -INT `cat /usr/local/var/run/openconnect.pid`

I’ll bet this change may solve the problems for some folks having this issue. Possibly not all of them, and probably leaves a root issue outstanding, but better that than hitting the proverbial brick wall.

–Scott

So I’m not sure that I have exactly the same problem, but its similar.

We use consul for service discovery, and one nice thing is to add the consul servers as a dns resolver on my local machine so I can use service discovery to find our applications in our network. I add this by creating a file on my host mac system at /etc/resolver/consul which specifies the ip and port of our consul master nodes. I can then ping a service like elasticsearch.service.consul.

Unfortunately the docker for mac application does not use the host networking stack. In fact using the screen technique from up above, I can see that the nameservers listed in /etc/resolv.conf are not even close to matching the nameservers from my local machine. I would just add the consul server to /etc/resolv.conf.tail and dhclient include it, but the resolver syntax does not provide a way to specify a different port.

Does anybody know how the networking works for docker for mac? Is it some sort of man in the middle like application? The ip addresses that the docker for mac vm have do not really make that much sense to me, as its a 192.168 address which our dhcp servers are not giving out.