We have a pair of Docker CE instances running on Oracle Linux 7.5. One is a Jenkins build agent and the other is set up as an image container to receive images from the build agent. Therefore the image server is running a Docker registry in a container. This is set up as an insecure registry.
When we try to push images from the build agent to the image registry, it fails.
We’re seeing that the push command does various retries when trying to submit the layer files, then ultimately bombs out with “digest invalid: provided digest did not match uploaded content”. The docker logs from the registry on the other machine showed that it was able to connect, but it’s mainly getting errors. What we initially see is the 404 "err.message=“blob unknown to registry” errors, which we now understand to mean that the layer isn’t already in the registry and it’ll need to receive it from the client. However, when it receives the layer we see more errors such as "error msg=“response completed with error” err.code=“digest invalid” err.detail=“invalid digest for referenced layer” and error msg=“canonical digest does match provided digest”.
If the image we push is based on commands only (with no files being copied in), then it works okay. It’s only when it has to PUT a layer that the digest or the SHA256 calculation fails?
The version of Docker on the build agent was originally considerably out of date compared to the image server, so we’d suspected that if there was a different in how the SHA256 or digests were constructed, that could have been the problem but an update failed to fix it. However, is it possible that there’s a dependant component that’s still out of date?
Curiously though, the image registry/server works fine for receiving images from other machines, including Windows Docker installations. It’s only the build agent that’s going wrong. The errors we’re getting aren’t providing us with any clues at this stage, so any help or enlightenment will be very useful. Thanks.
