Unexpected missing content-length header for request

Hi,

There were no changes in the deployment and about 7 hours ago I started receiving messages like:
missing content-length header for request: https://registry-1.docker.io/v2/***/[.....]/blobs/sha256:[.......]
Sometimes this error occurs on the first image I push, sometimes on a later images (like 4th or 5th). There is no particular image this occurs on.

Is this related to the new limitations on docker hub?

8 Likes

Got the same issue. No changes in the pipeline, and about an hour ago, the development pipeline ran successfully, but the production pipeline suddenly failed. Now, both development and production pipelines are failing with the same error:
missing content-length header for request

Got some issue when using jib-maven-plugin to build docker images. There is no change in our jib implementation but suddenly, this error occurs:

Failed to execute goal com.google.cloud.tools:jib-maven-plugin:3.4.0:build (jib-build) on project xxx: Tried to check BLOB exists for registry-1.docker.io/***/xxx with digest sha256:ā€¦ but failed because: Did not receive Content-Length header

Same here :confused:
missing content-length header for request: https://registry-1.docker.io/v2/myrepo/my-image/blobs/sha256:

Today we noticed the same problem.

Same thing in our CI, started failing around 6pm UTC on 6 Jan. Pushing multiple images fails on 1st, 2nd, 3rd, or 4th image with:

> docker push omniops/example:1234567

The push refers to repository [docker.io/omniops/example]
64b5f3ff9a89: Preparing
c4e8e502d85a: Preparing
89e9a2307d34: Preparing
89e9a2307d34: Mounted from omniops/example
64b5f3ff9a89: Pushed
c4e8e502d85a: Pushed
missing content-length header for request: https://registry-1.docker.io/v2/omniops/example/blobs/sha256:1096af2f9db68416edb7b5366a3b49ad15d18e6285b0fef9ecde71223630ca09

I tried switching from serviceaccount login to personal access token, this didnā€™t work.

Getting the same error this morning, managed to get it to work after re-running our build pipeline a few times. We use GitLab.

If the content-length header is missing, which should be set by the client, can you all share the docker version you are using?

Iā€™m not sure why it works after trying multiple times though.

Iā€™ve got the same issue, our CI server is using Docker version 26.1.3, build b72abbb. Iā€™m about to try downgrading to see if that works

same here, no changes in code, cicd production build crashed due to the same error:
missing content-length header for request

ran again after a couple of minutes and it happened on another image but the same image that had the error before had passed the push

ran again after another couple of minutes and the push succeeded for all images

seems like a random issue with docker hub

Had the same issue, updating our docker images in our CI seems to have fixed it for us (even multiple retries were not working).

Old version: docker:24.0.5 & docker:24.0.5-dind

New version: docker:27.4.1 & docker:27.4.1-dind

2 Likes

Upgrading solved the issue for us, too. Thanks @plustimeit :ok_hand:

The upgrade to v27 has worked for us, thanks a lot!

The issue persists but occurs less frequently with Docker version 27.4.1, build b9d17ea. We observed it happening just a minute ago.

Experiencig the same issue on Travis CI builds, no pipeline code has been changed, seems like something got broken on docker hub itself. And it falls randomly, sometimes it works OK, sometimes failed with the same errorā€¦ Worst part here is that we do not manage docker version directly, so workaround with newer version does not look like a good solutionā€¦ If it is possible, just make it work great again :slight_smile:

Have any of you had this issue locally, not in a CI/CD pipeline?

I got news from Docker that it was related to a change that they rolled back while they are investigating. So it shouldnā€™t happen now and hopefully they can find a permanent solution soon. Please, share if you still experience the issue.

Stilll experiencing it in CI/CD pipelinesā€¦

Hi,
Iā€™m CI/CD Tech Lead in CDQ AG.

Yesterday I spent almost the whole day debugging this issue. Our companyā€™s CI/CD was down. Developers couldnā€™t work.
As a paid Docker customer, I want to emphasize how unprofessional this is. And this is not the first time when Docker team is rolling a breaking change, without earlier notice. Previously it was related to changes in Docker Hub REST API.
To be honest I have enough of this. I cannot deliver stable CI/CD environment when I have no guarantee on Docker side that they will not change something out of the blue.

This is why I decided that our company will be leaving Docker Hub as a registry provider.

Br,
Marek

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.