Hello everyone, since yesterday (2023-08-23 around 6PM CEST), I get the following issue on all my CI pipelines
+ docker build -t temp:3471d940-f00f-4426-a424-a6fefc6f8e6d -f Dockerfile.buildenv .
#0 building with "default" instance using docker driver
#1 [internal] load .dockerignore
#1 transferring context: 129B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile.buildenv
#2 transferring dockerfile: 489B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/debian:buster
#3 ERROR: failed to authorize: failed to fetch oauth token: Post "https://auth.ipv6.docker.com/token": dial tcp [2600:1f18:2148:bc02:4bf0:3a98:55ae:e3d5]:443: connect: network is unreachable
#4 [auth] library/debian:pull token for registry-1.docker.io
#4 DONE 0.0s
------
> [internal] load metadata for docker.io/library/debian:buster:
------
Dockerfile.buildenv:1
--------------------
1 | >>> FROM debian:buster
2 |
3 | # Install dependencies:
--------------------
ERROR: failed to solve: debian:buster: failed to authorize: failed to fetch oauth token: Post "https://auth.ipv6.docker.com/token": dial tcp [2600:1f18:2148:bc02:4bf0:3a98:55ae:e3d5]:443: connect: network is unreachable
I have checked all the config files, software updates and I can ensure without any doubt that absolutely nothing has changed on this machine between the time this processes worked (around 3 hours earlier) and now
I tried to disable IPv6 in the Docker daemon configuration file and restart it but this did not help
Any clue on what happened and how it could be solved ?
#0 building with "default" instance using docker driver
#1 [internal] load build definition from Dockerfile.prod
#1 transferring dockerfile: 768B done
#1 DONE 0.0s
#2 [internal] load .dockerignore
#2 transferring context: 357B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/node:16
#3 ERROR: failed to authorize: failed to fetch anonymous token: Get "https://auth.ipv6.docker.com/token?scope=repository%3Alibrary%2Fnode%3Apull&service=registry.docker.io": dial tcp [2600:1f18:2148:bc02:4bf0:3a98:55ae:e3d5]:443: connect: cannot assign requested address
------
> [internal] load metadata for docker.io/library/node:16:
------
WARNING: buildx: git was not found in the system. Current commit information was not captured by the build
Dockerfile.prod:1
--------------------
1 | >>> FROM node:16 as base
2 |
3 | ENV APP_HOME=/usr/src/app/
--------------------
ERROR: failed to solve: node:16: failed to authorize: failed to fetch anonymous token: Get "https://auth.ipv6.docker.com/token?scope=repository%3Alibrary%2Fnode%3Apull&service=registry.docker.io": dial tcp [2600:1f18:2148:bc02:4bf0:3a98:55ae:e3d5]:443: connect: cannot assign requested address
export DOCKER_BUILDKIT=0 fixes the problem, so it seems that the legacy builder does not have the same problem.
I can manually pull the base image of my dockerfile with docker pull ..., afterwards the build also works (because the image already exists locally
curl -X POST https://auth.ipv6.docker.com directly also does not work (whereas the same address with .ipv4. works)
I did not find any way to prevent docker build to use the ipv6 auth url (I guess this is the root problem)
I’m running my build-script in another docker container (and call docker build in this cicd-container),
I can’t run my build-script inside the pipeline container but it works outside of the container directly on the server.
(even though they have the same docker version and the docker socket is simply mapped into the pipeline-container)
More system info:
# docker -v
Docker version 24.0.5, build ced0996
# docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS
default * docker
default default running v0.11.6+0a15675913b7 linux/amd64, linux/amd64/v2, linux/386
@davids88export DOCKER_BUILDKIT=0 && docker build .....
if docker build … is the command which fails
In my case, I had to change de default environment of my machine and Jenkins container
via /etc/environment and docker -e DOCKER_BUILDKIT=0 jenkins
Hello friends. I am on the infrastructure team at Docker. We are fixing this. I am sorry you all have run into this bug!
From what we have seen, this only happens for dual-stack networks where customers are running docker-in-docker.
We found that after the release of IPv6 for Docker Hub registry endpoints, some assumptions made in our system were no longer valid since all of the major endpoints were now dual-stack. This is why you are being forced to reach auth.ipv6.docker.com.
Another option is to enable IPv6 for Docker containers since the problem is with using IPv6 from within a Docker container. The host machine can reach the proper endpoints fine, from what we have seen.
We are working on a fix right now.
@itsreddi do you mean self-hosted runners or runners hosted by GitHub? Could you share relevant parts of your GitHub Actions workflow?
Ah. Actually, a better question might be… How are you consuming the Docker server? Are you mounting the host file /var/run/docker.sock? If so, turning off BuildKit with export DOCKER_BUILDKIT=0 should solve the problem. You can also turn on IPV6 in the daemon configuration settings as mentioned before as well.
We have opened an incident on our status page and are investigating a solution.
Thank you for your patience while we worked on this fix. We’ve just finished rolling out the solution to use a dual-stack authentication endpoint, as the ipv6 only auth endpoint was setup for the beta release.