Can I use docker buildx with ADD instruction that requires a custom certificate authority?

Using docker buildx I am trying to download an artifact from an internal artifactory server over https. The certificate is signed by our own CA. I can successfully
reference the server as a docker repository, but I want to access it over https using the ADD instruction. When I do that, I get ERROR: failed to solve: failed to load cache key: Get "https://foo-internal/some-artifact": x509: certificate signed by unknown authority.

Here is a very minimal example that reproduces the issue for me:

Given:

  • an artifactory docker repository at dockerhub.xxxx.internal with a certifcate signed by My-Custom-CA.pem
  • the following buildkit.toml:
[registry."dockerhub.xxxx.internal"]
  ca=["./My-Custom-CA.pem"]
  • the following minimal Dockerfile:
FROM dockerhub.xxxx.internal/amazonlinux:2.0.20230307.0

This command works fine:

docker buildx create --use --config buildkitd.toml && \
docker buildx build --load .

However, if I add an ADD instruction to download from the same server, so the Dockerfile becomes:

FROM dockerhub.xxxx.internal/amazonlinux:2.0.20230307.0

ADD https://dockerhub.xxxx.internal/artifactory/dockerhub-prod/repository.catalog repository.catalog

The same instruction gives the following output:

[+] Building 4.4s (7/7) FINISHED                                                                                                                                                                                                  
 => [internal] booting buildkit                                                                                                                                                                                              2.1s
 => => pulling image moby/buildkit:buildx-stable-1                                                                                                                                                                           1.4s
 => => creating container buildx_buildkit_determined_burnell0                                                                                                                                                                0.7s
 => [internal] load build definition from Dockerfile                                                                                                                                                                         0.0s
 => => transferring dockerfile: 235B                                                                                                                                                                                         0.0s
 => [internal] load .dockerignore                                                                                                                                                                                            0.0s
 => => transferring context: 2B                                                                                                                                                                                              0.0s
 => [internal] load metadata for dockerhub.xxxx.internal/amazonlinux:2.0.20230307.0                                                                                                                                         1.9s
 => ERROR https://dockerhub.xxxx.internal/artifactory/dockerhub-prod/repository.catalog                                                                                                                                     0.3s
 => [1/2] FROM dockerhub.xxxx.internal/amazonlinux:2.0.20230307.0@sha256:8a3fbbbaf93665e495fd66e86c7c5d46de44ab0bc74460b97489820747e0a164                                                                                   0.3s
 => => resolve dockerhub.xxxx.internal/amazonlinux:2.0.20230307.0@sha256:8a3fbbbaf93665e495fd66e86c7c5d46de44ab0bc74460b97489820747e0a164                                                                                   0.0s
 => => sha256:042c9cfa8a36c0ffe86667a7dd7d488f78cbe295aa845213c01fdf8784165a92 0B / 64.13MB                                                                                                                                  0.3s
 => CANCELED [2/2] ADD https://dockerhub.xxxx.internal/artifactory/dockerhub-prod/repository.catalog repository.catalog                                                                                                     0.0s
------
 > https://dockerhub.xxxx.internal/artifactory/dockerhub-prod/repository.catalog:
------
ERROR: failed to solve: failed to load cache key: Get "https://dockerhub.xxxx.internal/artifactory/dockerhub-prod/repository.catalog": x509: certificate signed by unknown authority

Is there any way to get the buildx build agent to respect my custom certificate authority when downloading using the ADD instruction?

The are two scopes involved:

  • host scope used by buildkit to fetch/push images
  • container scope used when building the image (each instruction in the Dockerfile uses a new container, which result in a layer of the final image)

You took care of the host scope. Now you need to take care of the container scope as well. The ca certificate needs to be copied into the image and the ca trust store needs to be updated.

Something like this should do the trick:

COPY ./My-Custom-CA.pem /etc/pki/ca-trust/source/anchors/My-Custom-CA.crt 
RUN  yum install ca-certificates \
  && update-ca-trust force-enable \
  && update-ca-trust extract

ADD ...

Make sure your certificate has the .crt file ending when you copy it to the target path, otherwise the certificate will be ignored when update-ca-trust extract is used.

Thanks for your response. Unfortunately, either it doesn’t work or I misunderstood what your advice. Here is my new Dockerfile:

FROM dockerhub.xxxx.internal/amazonlinux:2.0.20230307.0

COPY ./My-Custom-CA.pem /etc/pki/ca-trust/source/anchors/My-Custom-CA.crt

RUN  yum install ca-certificates \
  && update-ca-trust force-enable \
  && update-ca-trust extract

ADD https://dockerhub.xxxx.internal/artifactory/dockerhub-prod/repository.catalog repository.catalog

The output now is:

[+] Building 4.7s (8/10)                                                                                                                                                                                                          
 => [internal] booting buildkit                                                                                                                                                                                              2.3s
 => => pulling image moby/buildkit:buildx-stable-1                                                                                                                                                                           1.6s
 => => creating container buildx_buildkit_frosty_curran0                                                                                                                                                                     0.7s
 => [internal] load .dockerignore                                                                                                                                                                                            0.0s
 => => transferring context: 2B                                                                                                                                                                                              0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                                         0.0s
 => => transferring dockerfile: 411B                                                                                                                                                                                         0.0s
 => [internal] load metadata for dockerhub.xxxx.internal/amazonlinux:2.0.20230307.0                                                                                                                                         1.9s
 => [internal] load build context                                                                                                                                                                                            0.0s
 => => transferring context: 1.48kB                                                                                                                                                                                          0.0s
 => [1/4] FROM dockerhub.xxxx.internal/amazonlinux:2.0.20230307.0@sha256:8a3fbbbaf93665e495fd66e86c7c5d46de44ab0bc74460b97489820747e0a164                                                                                   0.3s
 => => resolve dockerhub.xxxx.internal/amazonlinux:2.0.20230307.0@sha256:8a3fbbbaf93665e495fd66e86c7c5d46de44ab0bc74460b97489820747e0a164                                                                                   0.0s
 => => sha256:042c9cfa8a36c0ffe86667a7dd7d488f78cbe295aa845213c01fdf8784165a92 0B / 64.13MB                                                                                                                                  0.4s
 => ERROR https://dockerhub.xxxx.internal/artifactory/dockerhub-prod/repository.catalog                                                                                                                                     0.3s
 => CANCELED [2/4] COPY ./My-Custom-CA.pem /etc/pki/ca-trust/source/anchors/My-Custom-CA.crt                                                                                                                                 0.0s
------
 > https://dockerhub.xxxx.internal/artifactory/dockerhub-prod/repository.catalog:
------
ERROR: failed to solve: failed to load cache key: Get "https://dockerhub.xxxx.internal/artifactory/dockerhub-prod/repository.catalog": x509: certificate signed by unknown authority

The final error is the same, but I’m confused by the CANCELED [2/4] COPY ./My-Custom-CA.pem /etc/pki/ca-trust/source/anchors/My-Custom-CA.crt which is before the ADD instruction gives the error.

Seems the parallel execution tries to fetch the artifact before the COPY command.
I can’t really say why it runs those instructions in parallel.

Just for the sake of trying, do you mind testing if it works with a multi-stage build:

FROM dockerhub.xxxx.internal/amazonlinux:2.0.20230307.0 as ca-base

COPY ./My-Custom-CA.pem /etc/pki/ca-trust/source/anchors/My-Custom-CA.crt

RUN  yum install ca-certificates \
  && update-ca-trust force-enable \
  && update-ca-trust extract

FROM ca-base
ADD https://dockerhub.xxxx.internal/artifactory/dockerhub-prod/repository.catalog repository.catalog

Same deal:

← snip →

 => ERROR https://dockerhub.xxxx.internal/artifactory/dockerhub-prod/repository.catalog                                                                                                                                     0.4s
 => [ca-base 1/3] FROM dockerhub.xxxx.internal/amazonlinux:2.0.20230307.0@sha256:8a3fbbbaf93665e495fd66e86c7c5d46de44ab0bc74460b97489820747e0a164                                                                           0.4s
 => => resolve dockerhub.xxxx.internal/amazonlinux:2.0.20230307.0@sha256:8a3fbbbaf93665e495fd66e86c7c5d46de44ab0bc74460b97489820747e0a164                                                                                   0.0s
 => => sha256:042c9cfa8a36c0ffe86667a7dd7d488f78cbe295aa845213c01fdf8784165a92 0B / 64.13MB                                                                                                                                  0.4s
 => CANCELED [ca-base 2/3] COPY ./My-Custom-CA.pem /etc/pki/ca-trust/source/anchors/My-Custom-CA.crt                                                                                                                         0.0s
------
 > https://dockerhub.xxxx.internal/artifactory/dockerhub-prod/repository.catalog:
------
ERROR: failed to solve: failed to load cache key: Get "https://dockerhub.xxxx.internal/artifactory/dockerhub-prod/repository.catalog": x509: certificate signed by unknown authority

Incidentally, if I add wget to the yum install and then instead of ADD, I do:

RUN wget https://dockerhub.xxxx.internal/artifactory/dockerhub-prod/repository.catalog -O repository.catalog

it does work. (Note the lack of any --ca-certificate or --ca-directory options.)

So, I have a workaround, but I’d like to get this figured out. My context is that I am trying to migrate our builds
from docker build to docker buildx build and I am not the author of many of these Dockerfile’s.

Is the ==> CANCELED [2/4] COPY portion of the output telling us that perhaps the load of that layer is cancelled as opposed to the build? That would at least be less confusing about the order of things…

I created this response, but forgot to post it:
So the ca certificate is used from the trust store, but the ADD instruction still has no chance to use it or maybe wouldn’t even use it all. Apparently the ADD institution in works differently than I thought.

I am curious if the same thing happens, if the first stage is built as standalone image, and this built image is used in a standalone Dockerfile.

today’s response:

Builds are processed in parallel. It looks to me like the failed ADD instruction caused the other unprocessed instructions to get calnceled

An ADD instruction with a URL source, AFAIK, is a straightforward GET request made by the build host (in this case, buildx). All the steps taken so far add the custom certificate authority to the build stage(s), not to the host. This is why the RUN wget works - the GET happens from inside the build stage where the CA is trusted.

I don’t know if there is a way to add the CA certificate to the build host environment. For my own Dockerfiles, I prefer to use RUN wget or RUN curl for fetching any remote artifacts instead of ADD; precisely because the RUN commands give me control over proxies, ca certs etc.