Docker pull not using correct DNS server when private registry on VPN

As far as I know, the bug tracker is not public so you won’t be able to follow on updates. However, if the issue is fixed you should see the issue number in the release notes.

I am facing the same problem with Version 1.11.1-beta14 (build: 8670). When I hop on VPN i get this

➜ ~ docker pull redis Using default tag: latest Pulling repository docker.io/library/redis Network timed out while trying to connect to https://index.docker.io/v1/repositories/library/redis/images. You may want to check your internet connection or if you are behind a proxy.

Fingers crossed #3124 gets into the next release candidate, this causes me real pain when working from home

Facing the same issue with Docker version 1.12.0-rc2, build 906eacd, experimental, encountered via https://forums.docker.com/t/docker-pull-timeout/15015:

$ docker pull hello-world

Using default tag: latest
Pulling repository docker.io/library/hello-world
Network timed out while trying to connect to https://index.docker.io/v1/repositories/library/hello-world/images. You may want to check your internet connection or if you are behind a proxy.

$ cat /etc/resolv.conf

search intra.company.io
nameserver 10.9.69.70
nameserver 8.8.8.8

$ scutil --dns

DNS configuration

resolver #1
  search domain[0] : intra.company.io
  nameserver[0] : 10.9.69.70
  nameserver[1] : 8.8.8.8
  flags    : Request A records
Reachable

resolver #2
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records
Not Reachable
  order    : 300000

resolver #3
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
Not Reachable
  order    : 300200

resolver #4
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
Not Reachable
  order    : 300400

resolver #5
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
Not Reachable
  order    : 300600

resolver #6
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
Not Reachable
  order    : 300800

resolver #7
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
Not Reachable
  order    : 301000

DNS configuration (for scoped queries)

resolver #1
  search domain[0] : intra.company.io
  nameserver[0] : 10.9.69.70
  nameserver[1] : 8.8.8.8
  if_index : 5 (en1)
  flags    : Scoped, Request A records
Reachable

$ screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty

$ cat /etc/resolv.conf

search local
nameserver 192.168.65.1

Adding 8.8.8.8 to the VM image’s resolv.conf fixes the issue. I don’t know where 192.168.65.1 is coming from.

I would expect to be able to pull images both from index.docker.io and from docker.intra.company.io, without messing with any details in the VM image. I’d also expect to be able to pull from docker.myhomenetwork.local if I connect to that via VPN.

When using Docker Toolbox, its possible to configure the Virtualbox VM to use the host’s resolver. This gets around all of these problems. Any thing the Mac host can resolve, including /etc/hosts entries and domain specific DNS settings in /etc/resolver are automatically picked up, even when changed on the fly.

You do this by stopping the default VM, then the following before starting it again.
VBoxManage modifyvm default --natdnshostresolver1 on

The ability to do something similar would be great. Even better, have it happen automatically since the aim here is to have the xhyve VM as largely transparent. I assume this is something xhyve would need to be able to do.

1 Like

DNS resolution seems to have been fixed. With the current version 1.12.0-rc4-beta20 (build: 10404), I no longer have to apply any workarounds inside the moby VM to push/pull from our internal registry in our VPN.

Nope, doesn’t work for me, I still get an “i/o timeout” trying to login to our internal private registry using the latest “Docker for MAC” Beta - Version 1.12.0-rc4-beta20 (build: 10404) … it still is trying the wrong (external fallback) IP to communicate via https

Not working for me either in 1.12.0-rc4-beta20 (build: 10404)

This is not working for me either. I also tried the suggestions above up updating the resolve.conf within Apline and I still get the error “no route to host” when trying to pull images from my companies internal repository.

and just for reference, still not working in Version 1.12.0-beta21 (build: 11019), and even if I edit host and resolve.conf in the container it doesn’t really work it seems… the login works, but the pull/push e.g. times out

I am still unable to access a private repository on my companies intranet using Version 1.12.1-beta24 (build:11525). I tried using docker run --rm busybox traceroute and this failed to ever find a path from 172.17.0.1.

1 Like

looks like the resolve.conf fix doesn’t work with the latest beta, however adding the relevant host IPs mappings to /etc/hosts seems to work

I have tried both editing my /etc/resolve.conf to add the same entries as the host system and add the ip and name that I am trying to connect with to /etc/hosts without any success. I am still unable to ping the internal docker repository at my company that is on the intranet (172.20.7.x) and not available through the internet.

I have even tried to follow the advice that worked for someone else on this thread https://github.com/docker/docker/issues/25645, but it still didn’t work.

If I docker network inspect 72bd17050252 (bridge network) I get the following:

“IPAM”: {
“Driver”: “default”,
“Options”: null,
“Config”: [
{
“Subnet”: “172.17.0.0/16”,
“Gateway”: “172.17.0.1”
}
]
}

The internal docker registry that I am trying to pull docker images from is on 172.20.7.80. I can seem to get a ping, traceroute, or wget all time out because they can’t connect to the remote host.

Correction:

/etc/hosts workaround is required to pull images from an internal registry, but containers are still not able to resolve internal hosts.

The below workaround was necessary for internal resolution by containers. In our case, we are building our code using maven and maven has to pull from an internal maven repository.

Thanks to the link to https://github.com/docker/docker/issues/25645 by @dwitherspoon, I was able to find another workaround:

cd ~/Library/Containers/com.docker.docker/Data/database/
git reset --hard
cd com.docker.driver.amd64-linux
echo '{ "dns" : [ "1.2.3.4" ] }' > etc/docker/daemon.json
git add etc/docker/daemon.json
git commit -m 'update config' 

1.2.3.4 is the IP address of your internal DNS server. The service restarts automatically.

1 Like

I have found it easier to redirect commands into the tty file like so:
DNS=8.8.8.8 echo "root" >> ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty echo "" >> ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty sleep 1 echo "echo \"nameserver $DNS\" > /etc/resolv.conf" >> ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty

You can then use your screen commands to verify the changes, but the above steps are easier to script.

This solved the issue for me. Though what I did instead is change the first line to:

DNS=`cat /etc/resolv.conf`

Thanks much!

I’m having the same problem and have spent many hours trying to find a solution. Hopefully the information I provide below (including a work-around using a user-defined network) helps Docker with an investigation and/or helps you to get past this problem. Note that some info has been redacted.

I’m using Docker for Mac, stable version:

$ docker --version
Docker version 1.12.3, build 6b644ec

Part of our Docker build includes pulling images from an in-house artifactory repository. For some reason, starting today, this began failing. During a Docker build, numerous artifacts would pull from artifactory just fine and then suddenly the build would hit an inability to resolve the same host (that had been resolving just perfectly seconds before in the same Docker build).

Note that I am on VPN using Cisco AnyConnect.

I’ve found this to be very reproducible. Here’s a very simple Dockerfile that builds an alpine image and installs bash. You can then hop into the image to start trying to resolve hosts.

Start with Dockerfile:

FROM alpine:3.4

RUN apk update && \
    apk add --no-cache \
      bash \
      curl

Now build and run bash:

$ docker build -t myimage .
$ docker run -it myimage bash

When I try to lookup our in-house artifactory inside the container:

for i in {0..10}; do nslookup artifactory.corp.redacted.com; done

A number of the lookups succeed and others fail. Huh?!

Here’s resolve.conf and scutil --dns on my host Mac:

$ cat /etc/resolv.conf
search can.redacted.com corp.redacted.com
nameserver a.b.c.d
nameserver a.e.f.d
nameserver 192.168.1.1

$ scutil --dns
DNS configuration

resolver #1
  search domain[0] : can.redacted.com
  search domain[1] : corp.redacted.com
  nameserver[0] : a.b.c.d
  nameserver[1] : a.e.f.d
  nameserver[2] : 192.168.1.1
  flags    : Request A records, Request AAAA records
Reachable, Directly Reachable Address
  order    : 1

resolver #2
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
Not Reachable
  order    : 300000

resolver #3
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
Not Reachable
  order    : 300200

resolver #4
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
Not Reachable
  order    : 300400

resolver #5
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
Not Reachable
  order    : 300600

resolver #6
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
Not Reachable
  order    : 300800

resolver #7
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
Not Reachable
  order    : 301000

DNS configuration (for scoped queries)

resolver #1
  search domain[0] : corp.redacted.com
  nameserver[0] : 192.168.1.1
  if_index : 4 (en0)
  flags    : Scoped, Request A records
Reachable, Directly Reachable Address

resolver #2
  search domain[0] : can.redacted.com
  search domain[1] : corp.redacted.com
  nameserver[0] : a.b.c.d
  nameserver[1] : a.e.f.d
  nameserver[2] : 192.168.1.1
  if_index : 12 (utun0)
  flags    : Scoped, Request A records, Request AAAA records
Reachable, Directly Reachable Address
  order    : 1

So what does the Docker container have?:

bash-4.3# cat /etc/resolv.conf
# Generated by dhcpcd from eth0.dhcp
# /etc/resolv.conf.head can replace this line
domain local
search can.redacted.com corp.redacted.com
nameserver 192.168.65.1
nameserver 192.168.65.10
nameserver 192.168.65.9
nameserver 192.168.65.8
nameserver 192.168.65.7
nameserver 192.168.65.6
nameserver 192.168.65.5
nameserver 192.168.65.4
nameserver 192.168.65.3
# /etc/resolv.conf.tail can replace this line

That’s very surprising. I thought Docker was supposed to make this match the host’s /etc/resolv.conf. I have no idea what these IP addresses point to, but I thought I’d pick one or two and do some nslookups. For any one I choose, the results seem to fluctuate between success and failure.

An extremely odd detail to add is that I can only see this problem with alpine containers. While in all cases the /etc/resolv.conf is the same, the actual problem is not reproducible in ubuntu or busybox containers. I’ve begun to wonder if the problem is caused by parallel querying of DNS servers as described in http://gliderlabs.viewdocs.io/docker-alpine/caveats/ but I can’t get past the fact that /etc/resolv.conf just makes no sense to me. Shot in the dark: I wonder if the Docker daemon itself is running alpine and when running alpine containers it somehow gets into a weird mode that hits on the parallelism described in the caveats?

I noticed that I was able to create a custom bridge network and use that instead. When you do so, the /etc/resolv.conf that is generated points only at Docker’s embedded DNS server, and things work beautifully.

$ docker network create -d bridge my-bridge-network
097f790a0417e48e25b91185f00a334532e8a8bc46ad54dfb99740fa4ac9050b
$ docker run -it -network=my-bridge-network image-name bash
bash-4.3# cat /etc/resolv.conf
search can.redacted.com corp.redacted.com
nameserver 127.0.0.11
options ndots:0

Unfortunately, I’ve found no way to specify that this network be used when doing a Docker build.

Something odd is going on with the default Docker bridge network.

Additional discussion here: https://github.com/docker/for-mac/issues/540

@jefsayshi @ianmacl your script worked for me. Thanks.
I’ve got the same error just after updating 1.13.0 even without VPN:
Docker version 1.13.0, build 49bf474