Using docker buildx I am trying to download an artifact from an internal artifactory server over https. The certificate is signed by our own CA. I can successfully
reference the server as a docker repository, but I want to access it over https using the ADD instruction. When I do that, I get ERROR: failed to solve: failed to load cache key: Get "https://foo-internal/some-artifact": x509: certificate signed by unknown authority.
Here is a very minimal example that reproduces the issue for me:
Given:
an artifactory docker repository at dockerhub.xxxx.internal with a certifcate signed by My-Custom-CA.pem
However, if I add an ADD instruction to download from the same server, so the Dockerfile becomes:
FROM dockerhub.xxxx.internal/amazonlinux:2.0.20230307.0
ADD https://dockerhub.xxxx.internal/artifactory/dockerhub-prod/repository.catalog repository.catalog
container scope used when building the image (each instruction in the Dockerfile uses a new container, which result in a layer of the final image)
You took care of the host scope. Now you need to take care of the container scope as well. The ca certificate needs to be copied into the image and the ca trust store needs to be updated.
Make sure your certificate has the .crt file ending when you copy it to the target path, otherwise the certificate will be ignored when update-ca-trust extract is used.
The final error is the same, but I’m confused by the CANCELED [2/4] COPY ./My-Custom-CA.pem /etc/pki/ca-trust/source/anchors/My-Custom-CA.crt which is before the ADD instruction gives the error.
=> ERROR https://dockerhub.xxxx.internal/artifactory/dockerhub-prod/repository.catalog 0.4s
=> [ca-base 1/3] FROM dockerhub.xxxx.internal/amazonlinux:2.0.20230307.0@sha256:8a3fbbbaf93665e495fd66e86c7c5d46de44ab0bc74460b97489820747e0a164 0.4s
=> => resolve dockerhub.xxxx.internal/amazonlinux:2.0.20230307.0@sha256:8a3fbbbaf93665e495fd66e86c7c5d46de44ab0bc74460b97489820747e0a164 0.0s
=> => sha256:042c9cfa8a36c0ffe86667a7dd7d488f78cbe295aa845213c01fdf8784165a92 0B / 64.13MB 0.4s
=> CANCELED [ca-base 2/3] COPY ./My-Custom-CA.pem /etc/pki/ca-trust/source/anchors/My-Custom-CA.crt 0.0s
------
> https://dockerhub.xxxx.internal/artifactory/dockerhub-prod/repository.catalog:
------
ERROR: failed to solve: failed to load cache key: Get "https://dockerhub.xxxx.internal/artifactory/dockerhub-prod/repository.catalog": x509: certificate signed by unknown authority
Incidentally, if I add wget to the yum install and then instead of ADD, I do:
RUN wget https://dockerhub.xxxx.internal/artifactory/dockerhub-prod/repository.catalog -O repository.catalog
it does work. (Note the lack of any --ca-certificate or --ca-directory options.)
So, I have a workaround, but I’d like to get this figured out. My context is that I am trying to migrate our builds
from docker build to docker buildx build and I am not the author of many of these Dockerfile’s.
Is the ==> CANCELED [2/4] COPY portion of the output telling us that perhaps the load of that layer is cancelled as opposed to the build? That would at least be less confusing about the order of things…
I created this response, but forgot to post it:
So the ca certificate is used from the trust store, but the ADD instruction still has no chance to use it or maybe wouldn’t even use it all. Apparently the ADD institution in works differently than I thought.
I am curious if the same thing happens, if the first stage is built as standalone image, and this built image is used in a standalone Dockerfile.
today’s response:
Builds are processed in parallel. It looks to me like the failed ADD instruction caused the other unprocessed instructions to get calnceled
An ADD instruction with a URL source, AFAIK, is a straightforward GET request made by the build host (in this case, buildx). All the steps taken so far add the custom certificate authority to the build stage(s), not to the host. This is why the RUN wget works - the GET happens from inside the build stage where the CA is trusted.
I don’t know if there is a way to add the CA certificate to the build host environment. For my own Dockerfiles, I prefer to use RUN wget or RUN curl for fetching any remote artifacts instead of ADD; precisely because the RUN commands give me control over proxies, ca certs etc.
The FROM instruction is not failing because the CA is added via the buildkitd.toml. However, it does not appear to apply to the ADD instruction, so clearly:
The CA is added to the build host (because FROM works)
The CA is not added to the build host in a way that lets the ADD instruction use it.