Docker Community Forums

Share and learn in the Docker community.

Docker push not working from one particular machine

We have a pair of Docker CE instances running on Oracle Linux 7.5. One is a Jenkins build agent and the other is set up as an image container to receive images from the build agent. Therefore the image server is running a Docker registry in a container. This is set up as an insecure registry.

When we try to push images from the build agent to the image registry, it fails.

We’re seeing that the push command does various retries when trying to submit the layer files, then ultimately bombs out with “digest invalid: provided digest did not match uploaded content”. The docker logs from the registry on the other machine showed that it was able to connect, but it’s mainly getting errors. What we initially see is the 404 "err.message=“blob unknown to registry” errors, which we now understand to mean that the layer isn’t already in the registry and it’ll need to receive it from the client. However, when it receives the layer we see more errors such as "error msg=“response completed with error” err.code=“digest invalid” err.detail=“invalid digest for referenced layer” and error msg=“canonical digest does match provided digest”.

If the image we push is based on commands only (with no files being copied in), then it works okay. It’s only when it has to PUT a layer that the digest or the SHA256 calculation fails?

The version of Docker on the build agent was originally considerably out of date compared to the image server, so we’d suspected that if there was a different in how the SHA256 or digests were constructed, that could have been the problem but an update failed to fix it. However, is it possible that there’s a dependant component that’s still out of date?

Curiously though, the image registry/server works fine for receiving images from other machines, including Windows Docker installations. It’s only the build agent that’s going wrong. The errors we’re getting aren’t providing us with any clues at this stage, so any help or enlightenment will be very useful. Thanks.

Also… if we spin up a registry on the build agent, it can add an image to its own registry fine. ie

docker push localhost:5000/… etc…

We’ve now figured this out. :smiley:

It turned out to be because Docker was configured to use a lightweight proxy server (tinyproxy v1.8.3), which presumably has some issues? It didn’t actually need a proxy so when it was configured to go direct, it worked okay. Information from here https://docs.docker.com/registry/spec/api/#content-digests helped us to understand the logs and the point at which it was failing. It seems that Docker push uses HTTP PATCH to send the layers over in segments and we suspect that our proxy either reordered, truncated or otherwise corrupted the segments such that at the other end the SHA256 digest did not match.