docker build fails while docker pull works as expected

I trying to build a jenkins image starting FROM jenkins/jenkins:2.440.1-lts and it fails with error: failed to resolve source metadata for docker.io

If I run docker build .

docker build  .
[+] Building 30.1s (2/2) FINISHED                                                                                                                           docker:default
 => [internal] load build definition from Dockerfile                                                                                                                  0.0s
 => => transferring dockerfile: 1.49kB                                                                                                                                0.0s
 => ERROR [internal] load metadata for docker.io/jenkins/jenkins:2.440.1-lts                                                                                         30.0s
------
 > [internal] load metadata for docker.io/jenkins/jenkins:2.440.1-lts:
------
Dockerfile:1
--------------------
   1 | >>> FROM jenkins/jenkins:2.440.1-lts
   2 |
   3 |     ADD --chown=1000:1000 plugins.txt /var/jenkins_home/
--------------------
ERROR: failed to solve: DeadlineExceeded: DeadlineExceeded: DeadlineExceeded: jenkins/jenkins:2.440.1-lts: failed to resolve source metadata for docker.io/jenkins/jenkins:2.440.1-lts: failed to authorize: DeadlineExceeded: failed to fetch anonymous token: Get "https://auth.docker.io/token?scope=repository%3Ajenkins%2Fjenkins%3Apull&service=registry.docker.io": dial tcp 54.196.99.49:443: i/o timeout

If I run docker pull all is good:

docker pull jenkins/jenkins:2.440.1-lts
2.440.1-lts: Pulling from jenkins/jenkins
7bb465c29149: Pull complete
c18f106040a7: Pull complete
f826d0f09907: Pull complete
bc3128b515a7: Pull complete
45dc40627eb2: Pull complete
ce448981c34b: Pull complete
f2e0835aa94f: Pull complete
bce8503d6e45: Pull complete
3353a3a64a1f: Pull complete
5b9963212983: Pull complete
0a769ec2c083: Pull complete
6bbf8192cfe6: Pull complete
Digest: sha256:01c0b0cf789fa24253090fccea264df223b5e09b14a0ea59f0847c70bdc0f31c
Status: Downloaded newer image for jenkins/jenkins:2.440.1-lts
docker.io/jenkins/jenkins:2.440.1-lts

If I run docker build AFTER docker pull the build process is good since the image is already present locally.

docker build  .
[+] Building 2.4s (12/13)                                                                                                                                   docker:default
 => [internal] load build definition from Dockerfile                                                                                                                  0.0s
 => => transferring dockerfile: 1.49kB                                                                                                                                0.0s
 => [internal] load metadata for docker.io/jenkins/jenkins:2.440.1-lts                                                                                                0.0s
 => [internal] load .dockerignore                                                                                                                                     0.0s
 => => transferring context: 2B                                                                                                                                       0.0s
 => [1/9] FROM docker.io/jenkins/jenkins:2.440.1-lts                                                                                                                  0.2s
 => [internal] load build context                                                                                                                                     0.0s
 => => transferring context: 6.94kB                                                                                                                                   0.0s
 => [2/9] ADD --chown=1000:1000 plugins.txt /var/jenkins_home/                                                                                                        0.0s
docker info
Client: Docker Engine - Community
 Version:    27.3.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.17.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.7
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

The environment is: no direct internet connection, only through a proxy. It looks like docker build does not respect the Environment set at /etc/systemd/system/docker.service.d/http-proxy.conf. This config is respected by docker pull.

The DNS works as expected (so docker run also knows how to download an image which is not present locally):

docker run --rm busybox nslookup docker.io
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
2fce1e0cdfc5: Pull complete
Digest: sha256:c230832bd3b0be59a6c47ed64294f9ce71e91b327957920b6929a0caa8353140
Status: Downloaded newer image for busybox:latest
Server:         172.16.0.2
Address:        172.16.0.2:53

Non-authoritative answer:
Name:   docker.io
Address: 3.224.227.198
Name:   docker.io
Address: 44.219.3.189
Name:   docker.io
Address: 44.193.181.103

Non-authoritative answer:
Name:   docker.io
Address: 2600:1f18:2148:bc02:4f78:625e:1ac1:714c
Name:   docker.io
Address: 2600:1f18:2148:bc01:2822:9e2a:c82a:af0
Name:   docker.io
Address: 2600:1f18:2148:bc00:a518:1574:fea8:ccdb

I even tried to put proxy IP addresses instead FQDNs same issue. The same error can be found in jounalctl -u docker.service. Same error happens if I add/use /etc/docker/daemon.json.

I tried adding this docker client config but still no luck:

cat ~/.docker/config.json
{
 "proxies": {
   "default": {
     "httpProxy": "http://proxy.domain.internal:3128",
     "httpsProxy": "http://proxy.domain.internal:3129",
     "noProxy": "*.domain.internal,127.0.0.1"
   }
 }
}

I also tried this hack with docker buildx create --use --driver-opt:

docker buildx create --use --driver-opt env.http_proxy=http//proxy.domain.internal:3128 --driver-opt env.https_proxy=http://proxy.domain.internal:3128 --driver-opt '"env.no_proxy='*.domain.internal,127.0.0.1'"'
inspiring_matsumoto

docker buildx build .
[+] Building 35.7s (3/3) FINISHED                                                                                                     docker-container:inspiring_matsumoto
 => [internal] booting buildkit                                                                                                                                       5.2s
 => => pulling image moby/buildkit:buildx-stable-1                                                                                                                    4.6s
 => => creating container buildx_buildkit_inspiring_matsumoto0                                                                                                        0.6s
 => [internal] load build definition from Dockerfile                                                                                                                  0.0s
 => => transferring dockerfile: 1.49kB                                                                                                                                0.0s
 => ERROR [internal] load metadata for docker.io/jenkins/jenkins:2.440.1-lts                                                                                         30.3s
------
 > [internal] load metadata for docker.io/jenkins/jenkins:2.440.1-lts:
------
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
Dockerfile:1

I also tried this and still same error:

docker build --build-arg HTTP_PROXY=http://proxy.domain.internal:3128 \
--build-arg HTTPS_PROXY=http://proxy.domain.internal:3128 --build-arg NO_PROXY="*.domain.internal,127.0.0.1" \
--build-arg http_proxy=http://proxy.domain.internal:3128 --build-arg https_proxy=http://proxy.domain.internal:3128 \
--build-arg no_proxy="*.domain.internal,127.0.0.1" 
[+] Building 30.1s (2/2) FINISHED                                                                                                                           docker:default
 => [internal] load build definition from Dockerfile                                                                                                                  0.0s
 => => transferring dockerfile: 1.49kB                                                                                                                                0.0s
 => ERROR [internal] load metadata for docker.io/jenkins/jenkins:2.440.1-lts                                                                                         30.0s
------
 > [internal] load metadata for docker.io/jenkins/jenkins:2.440.1-lts:
------
Dockerfile:1
--------------------
   1 | >>> FROM jenkins/jenkins:2.440.1-lts
   2 |
   3 |     ADD --chown=1000:1000 plugins.txt /var/jenkins_home/
--------------------
ERROR: failed to solve: DeadlineExceeded: DeadlineExceeded: DeadlineExceeded: jenkins/jenkins:2.440.1-lts: failed to resolve source metadata for docker.io/jenkins/jenkins:2.440.1-lts: failed to authorize: DeadlineExceeded: failed to fetch anonymous token: Get "https://auth.docker.io/token?scope=repository%3Ajenkins%2Fjenkins%3Apull&service=registry.docker.io": dial tcp 54.196.99.49:443: i/o timeout

I tried all the above with the IP address of the proxy (not the FQDN) and receive the same error. Did even docker system prune -a, systemctl restart docker, rebooted the instance and docker build --no-cache just to have a clean env and Again docker pull works! docker build does not respect the proxy settings.

Afaik, you need to configure buildkit to use your http proxy as mirror, so that it pulls the images through the proxy.

Create a TOML file at /etc/buildkitd.toml with the following content

debug = true
[registry."docker.io"]
mirrors = ["http://proxy.example.com:3128"]

Then create a build that uses this configuration:

$ docker buildx create --use --bootstrap \
--name mybuilder \
--driver docker-container \
--config /etc/buildkitd.toml

hello,

with this solution it is trying to find endpoint /v2/jenkins... on the proxy URL. I donā€™t have a mirror of docker.io locally on the proxy instance. I want docker buildx to use the proxy settings same as docker pull.

cat /etc/buildkitd.toml
debug = true
[registry."docker.io"]
mirrors = ["http://proxy.domain.internal:3128"]
docker buildx create --use --bootstrap \
--name mybuilder \
--driver docker-container \
--config /etc/buildkitd.toml
[+] Building 1.8s (1/1) FINISHED
 => [internal] booting buildkit                                                                                                                                       1.8s
 => => pulling image moby/buildkit:buildx-stable-1                                                                                                                    0.9s
 => => creating container buildx_buildkit_mybuilder0                                                                                                                  0.9s
mybuilder
docker buildx ls
NAME/NODE        DRIVER/ENDPOINT                   STATUS    BUILDKIT   PLATFORMS
mybuilder*       docker-container
 \_ mybuilder0    \_ unix:///var/run/docker.sock   running   v0.16.0    linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
default          docker
 \_ default       \_ default                       running   v0.16.0    linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
docker buildx --builder mybuilder build .
[+] Building 30.1s (2/2) FINISHED                                                                                                               docker-container:mybuilder
 => [internal] load build definition from Dockerfile                                                                                                                  0.0s
 => => transferring dockerfile: 1.49kB                                                                                                                                0.0s
 => ERROR [internal] load metadata for docker.io/jenkins/jenkins:2.440.1-lts                                                                                         30.0s
------
 > [internal] load metadata for docker.io/jenkins/jenkins:2.440.1-lts:
------
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
Dockerfile:1
--------------------
   1 | >>> FROM jenkins/jenkins:2.440.1-lts
   2 |
   3 |     ADD --chown=1000:1000 plugins.txt /var/jenkins_home/
--------------------
ERROR: failed to solve: jenkins/jenkins:2.440.1-lts: failed to resolve source metadata for docker.io/jenkins/jenkins:2.440.1-lts: failed to do request: Head "https://proxy.domain.internal:3128/v2/jenkins/jenkins/manifests/2.440.1-lts?ns=docker.io": http: server gave HTTP response to HTTPS client

I canā€™t reproduce it. I just set the proxy settings as described in the documentation here: https://docs.docker.com/engine/daemon/proxy/#daemon-configuration
Then I restarted the docker service:

systemctl restart docker

and I knew my proxy config worked, because when I started to build, it immediately showed this error:

ERROR: failed to solve: centos:8: failed to resolve source metadata for docker.io/library/centos:8: failed to do request: Head "https://registry-1.docker.io/v2/library/centos/manifests/8": proxyconnect tcp: dial tcp: lookup proxy.example.com on 127.0.0.53:53: no such host (did you mean centos?)

which was expected as this was just copied from the documentationā€™s example. Then I tried to create a container-based builder (without the proxy settings) as described here: https://docs.docker.com/build/building/multi-platform/#create-a-custom-builder

docker buildx create \
  --name container-builder \
  --driver docker-container \
  --use --bootstrap

added the proxy settings back, restarted docker again and it still failed, so it used the proxy.

I donā€™t see any traces of a custom builder in your logs, so you it should have worked for you if you restarted the daemon after changing the config.

Can you try to use a wrong proxy url to see if your buildkit uses the proxy and the connection fails later for any reason or buildkit doesnā€™t use the proxy at all?

hello,

I removed the existing proxy config from /etc/docker/daemon.json and systemd drop-in /etc/systemd/system/docker.service.d/http-proxy.conf

rm -rf /etc/docker/daemon.json
rm -rf /etc/systemd/system/docker.service.d/http-proxy.conf
systemctl daemon-reload
systemctl restart docker.service
docker buildx create   --name container-builder   --driver docker-container   --use --bootstrap
[+] Building 15.0s (1/1) FINISHED
 => ERROR [internal] booting buildkit                                                                                                                                15.0s
 => => pulling image moby/buildkit:buildx-stable-1                                                                                                                   15.0s
------
 > [internal] booting buildkit:
------
ERROR: Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaitingheaders)

No proxy = no internet = canā€™t create the driver
put a bogus proxy in /etc/docker/daemon.json => not working (normal)

cat /etc/docker/daemon.json
{
  "proxies": {
    "http-proxy": "http://193.193.193.193:3128",
    "https-proxy": "http://193.193.193.193:3128"
  }
}
systemctl restart docker.service
docker buildx create   --name container-builder   --driver docker-container   --use --bootstrap
[+] Building 15.0s (1/1) FINISHED
 => ERROR [internal] booting buildkit                                                                                                                                15.0s
 => => pulling image moby/buildkit:buildx-stable-1                                                                                                                   15.0s
------
 > [internal] booting buildkit:
------
ERROR: Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaitingheaders)

put the correct proxy in /etc/docker/daemon.json => I can now build the driver (why it can pull images moby/ā€¦ but not docker.io/ā€¦?)

systemctl restart docker.service
docker buildx create   --name container-builder   --driver docker-container   --use --bootstrap
[+] Building 5.7s (1/1) FINISHED
 => [internal] booting buildkit                                                                                                                                       5.7s
 => => pulling image moby/buildkit:buildx-stable-1                                                                                                                    4.6s
 => => creating container buildx_buildkit_container-builder0                                                                                                          1.1s
container-builder

At this point I can try to build my image with good proxy + driver(see first line that it uses the driver):
I put a bogus proxy + restarted docker and the output is exactlly the same.

docker buildx --builder container-builder build --progress=plain .
#0 building with "container-builder" instance using docker-container driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 1.54kB done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/jenkins/jenkins:2.440.1-lts
#2 ERROR: failed to do request: Head "https://registry-1.docker.io/v2/jenkins/jenkins/manifests/2.440.1-lts": dial tcp 3.219.239.5:443: i/o timeout
------
 > [internal] load metadata for docker.io/jenkins/jenkins:2.440.1-lts:
------
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
Dockerfile:1
--------------------
   1 | >>> FROM jenkins/jenkins:2.440.1-lts
   2 |
   3 |     ADD --chown=1000:1000 plugins.txt /var/jenkins_home/
--------------------
ERROR: failed to solve: DeadlineExceeded: DeadlineExceeded: DeadlineExceeded: jenkins/jenkins:2.440.1-lts: failed to resolve source metadata for docker.io/jenkins/jenkins:2.440.1-lts: failed to do request: Head "https://registry-1.docker.io/v2/jenkins/jenkins/manifests/2.440.1-lts": dial tcp 3.219.239.5:443: i/o timeout

By the way if I use a drop-in in systemd unit file for docker and also use the /etc/docker/daemon.json, whatever I put in the second will overwrite the drop-in (from docker.service logs):

level=warning msg="overriding existing proxy variable with value from configuration" name=HTTP_PROXY new-value="http://193.193.193.193:3128" old-value="http://proxy.domain.internal:3128"
level=warning msg="overriding existing proxy variable with value from configuration" name=HTTPS_PROXY new-value="http://193.193.193.193:3128" old-value="http://proxy.domain.internal:3128"

So docker service takes into consideration the bogus proxy from /etc/docker/daemon.json, docker build doesnā€™t.

Also it seems that there are a lot of issues with buildtkit+proxy: Issues Ā· moby/buildkit Ā· GitHub

while browsing through them I found Client-side token fetching can be disabled by defining BUILDKIT_NO_CLIENT_TOKEN=1 and guess what: it works!

1 Like

I canā€™t explain it. I have the same buildkit version as you and I canā€™t reproduce the issue. When buildkit was introduced first, I didnā€™t like it to be honest. Strange bugs were solved by disabling buildkit, but I havenā€™t heard about those kind of issues for a recently. Apparently, networking could still be improved in buildkit.

Sorry if you already told us, but I tried to scroll up looking for that so I ask: have you tried it on another machine? I wonder if this issue would happen on a new machine where you are sure you have not set anything before. Maybe there is a setting that breaks your proxy settings and clears the variable before buildkit reads it. No idea how though.

Redeployed the VM (Ubuntu 24.04 LTS) on AWS, docker-ce official installed, same behavior.

Was the previous host also a VM on AWS or at least the same Ubuntu 24.04 distro? Iā€™m trying to find the difference and itā€™s true that I tried Ubuntu 22.04 VM on my macOS with an ARM cpu.