Docker Push to Docker.io hub hangs as "waiting" with no error or timeout

insightindustry · April 13, 2016, 11:39pm

I’m attempting to push an image to my private Docker Hub account using the docker push command.

In the past, this has worked just fine (even for previous versions of this image).

However, it is now hanging in perpetual “waiting” mode as shown in the screenshot below:

It would seem the daemon has frozen in this “Waiting” state for the past 10+ minutes, with no timeout or error message reported. This is a problem that has appeared in the past and then “magically” gone away later. This time, the problem has persisted for some time.

ETA: Here is my docker version info:

Client:
Version: 1.10.3
API version: 1.22
Go version: go1.5.3
Git commit: 20f81dd
Built: Thu Mar 10 21:49:11 2016
OS/Arch: windows/amd64

Server:
Version: 1.10.3
API version: 1.22
Go version: go1.5.3
Git commit: 20f81dd
Built: Thu Mar 10 21:49:11 2016
OS/Arch: linux/amd64

Two questions:

Is there some way for me to fix this problem so that the Docker Hub Registry is actually functional for my purposes?
Is it possible for the Docker daemon to throw some sort of error when this problem occurs, so that I’m able to capture it and apply the fix, as needed?

Thanks,
Chris

insightindustry · April 14, 2016, 12:14am

FYI: Restarting the docker-machine (VirtualBox) seems to have provided some progress. In particular, now the docker push command hangs on a particular layer: “Pushing 1.407 MB/12.14 MB” (see screenshot below):

All the best,
Chris

programmerq · April 14, 2016, 3:03am

Do you have anything interesting in your docker.log file?

Run this and then do a docker push in another terminal:

$ docker-machine ssh default
$ tail -f /var/log/docker.log

insightindustry · April 14, 2016, 4:22am

Here’s the initial output of the docker.log (prior to executing the new docker push):

time=“2016-04-14T00:06:00.566660007Z” level=info msg=“API listen on /var/run/doc
ker.sock”
time=“2016-04-14T00:11:29.338225942Z” level=debug msg=“Calling GET /v1.22/info”

time=“2016-04-14T00:11:29.338286356Z” level=debug msg=“GET /v1.22/info”
time=“2016-04-14T00:11:29.452014636Z” level=debug msg=“Calling POST /v1.22/image
s/insightindustry/dev-tools/push”
time=“2016-04-14T00:11:29.452059643Z” level=debug msg=“POST /v1.22/images/insigh
tindustry/dev-tools/push?tag=”
time=“2016-04-14T00:11:29.452293986Z” level=debug msg=“Trying to push docker.io/
insightindustry/dev-tools to https://registry-1.docker.io v2”
time=“2016-04-14T00:11:30.024493824Z” level=debug msg=“Pushing repository: insig
htindustry/dev-tools:latest”
time=“2016-04-14T00:11:30.060828995Z” level=debug msg=“Pushing layer: sha256:27d
17492a53eeb15a77853efadea0a89bea48440c2d5819fcb8b3aa7ff894585”
time=“2016-04-14T00:11:30.680582971Z” level=debug msg=“Increasing token expirati
on to: 0 seconds”
time=“2016-04-14T00:11:31.361442352Z” level=debug msg=“Assembling tar data for 0
d4d82f692d482bb50722a6484af642432626dd1396da6ff7c13727e5d4177a3 from /mnt/sda1/v
ar/lib/docker/aufs/diff/0d4d82f692d482bb50722a6484af642432626dd1396da6ff7c13727e
5d4177a3”

And here’s the output following a new docker push:

time=“2016-04-14T04:20:15.989144961Z” level=debug msg=“GET /v1.22/info”
time=“2016-04-14T04:20:16.027590659Z” level=debug msg=“Calling POST /v1.22/image
s/insightindustry/dev-tools/push”
time=“2016-04-14T04:20:16.027636729Z” level=debug msg=“POST /v1.22/images/insigh
tindustry/dev-tools/push?tag=”
time=“2016-04-14T04:20:16.027754868Z” level=debug msg=“Trying to push docker.io/
insightindustry/dev-tools to https://registry-1.docker.io v2”
time=“2016-04-14T04:20:16.347133520Z” level=debug msg=“Pushing repository: insig
htindustry/dev-tools:latest”
time=“2016-04-14T04:20:16.915397226Z” level=debug msg=“Increasing token expirati
on to: 0 seconds”

So as it stands, I can’t see anything weird jumping out to me other than the “Increasing token expiration to: 0 seconds” line.

ETA: I didn’t mention it before, but the behavior of the new docker push is identical to the previous one, hanging up on the same exact layer with Pushing 1.407 MB/12.14 MB.

All the best,
Chris

programmerq · April 14, 2016, 5:47pm

Does this happen only when pushing this repository? What happens if you do a push to a blank one?

insightindustry · April 14, 2016, 6:25pm

In the past, these symptoms have cropped up intermittently when pushing to a blank repository as well.

Incidentally, yesterday after this problem appeared I also deleted the repository and pushed to newly-created private repo as well - all to no avail.

Chris

learningstaircase · April 27, 2016, 2:00am

I am having similar issue, it doesn’t happen all the time for me, but I get a list of different layers, all saying waiting, then no pushing. I just redo the build/push and it goes through, but no error is getting sent out for the fact it didn’t push.

mabuzagu · June 26, 2016, 4:52pm

I am having similar issue. Push to public repo just hangs. I am on Windows 7. No luck with redo

taisph · July 17, 2016, 11:07pm

I am seeing this as well. Ubuntu 16.04, docker 1.11.2, public repo. Seems random how far it gets before it hangs. Sometimes stops with an HTTP error 408.

moigagoo · July 29, 2016, 6:36pm

Hi!

I’m having the issue with Docker 1.12.0 on Windows 10 (the latest Insider build).

Checked the logs—nothing interesting there:

time="2016-07-29T18:34:52.346614642Z" level=debug msg="Assembling tar data for 710271407e4fd2bc3f0201967727eee34a5f206f74cf0f36bc33bab62b298070"
time="2016-07-29T18:34:52.398633677Z" level=debug msg="Assembling tar data for 34ff364e2f1be5db53f99be326de5e8264e4efcbb7962890d845a6b9a2cab100"
time="2016-07-29T18:34:52.705860241Z" level=debug msg="Pushing layer: sha256:78887f53843c5dc256f4c0c17bc7343990aba943f3ef9078058a148a90f2d34a"
time="2016-07-29T18:34:53.159739010Z" level=debug msg="Pushing layer: sha256:aea97fe987ebc9b64f6dd8dc2e0fa156e5bffe81b69e54e7e936f459434b4ef2"
time="2016-07-29T18:34:53.160257804Z" level=error msg="Not continuing with push after error: context canceled"
time="2016-07-29T18:34:53.712838297Z" level=debug msg="Assembling tar data for 0ff659325014c2f452236ab6147bd542728c730be394146ab497f8279efc0f83"
time="2016-07-29T18:34:54.066550451Z" level=debug msg="Assembling tar data for ceae7d95f2e7b592b70fb9ad4d6712f3cefd566530529e7740d5109e79988c74"
time="2016-07-29T18:34:59.875226976Z" level=debug msg="Pushing layer: sha256:aea97fe987ebc9b64f6dd8dc2e0fa156e5bffe81b69e54e7e936f459434b4ef2"
time="2016-07-29T18:34:59.876789459Z" level=error msg="Not continuing with push after error: context canceled"
time="2016-07-29T18:35:00.780837361Z" level=debug msg="Assembling tar data for ceae7d95f2e7b592b70fb9ad4d6712f3cefd566530529e7740d5109e79988c74"

Did anyone come up with any workaround yet?

ricard0javier · September 21, 2016, 3:14pm

I’m having the same issue in OS X

$ docker version

Client:
Version: 1.12.1
API version: 1.24
Go version: go1.7.1
Git commit: 6f9534c
Built: Thu Sep 8 10:31:18 2016
OS/Arch: darwin/amd64

Server:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built: Thu Aug 18 17:52:38 2016
OS/Arch: linux/amd64

chakrar27 · October 8, 2016, 11:54pm

I am having issues while pushing local docker images to registry. I am using docker 1.12 on windows.

The docker logs show the following output:
docker-machine ssh default
## .
## ## ## ==
## ## ## ## ## ===
/"""""""""""""""""_/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ / ===- ~~~
____ o /
\ \ /
_______/

| |__ ___ ___ | ||__ \ | | ___ | | _____ _ __
| ’ \ / _ \ / _ | __| __) / ` |/ _ \ / | |/ / _ \ '|
| |) | () | () | | / / (| | () | (| < / |
|_./ _/ _/ _|_____,|_/ _||____|_|
Boot2Docker version 1.12.0, build HEAD : e030bab - Fri Jul 29 00:29:14 UTC 2016
Docker version 1.12.0, build 8eab29e
docker@default:~$ tail -f /var/log/docker.log
time=“2016-10-08T23:46:37.060617364Z” level=debug msg="Increasing token expiration to: 60 seconds"
time=“2016-10-08T23:46:40.082222786Z” level=debug msg="uploaded layer sha256:78ff13900d615402568a24175b5ff1d7582c12f3ba4903f99a73752c6d5f597e (sha256:04c996abc2442fb0534f018543f0352647de3a65a3b321628b8cd1ceecedc3f6), 65702977 bytes"
time=“2016-10-08T23:47:48.102006253Z” level=error msg="Not continuing with push after error: context canceled"
time=“2016-10-08T23:47:54.676401043Z” level=debug msg="Calling GET /v1.24/info"
time=“2016-10-08T23:47:54.687198318Z” level=debug msg="Calling POST /v1.24/images/chakrar27/couchbase_new_10_08_2016/push?tag="
time=“2016-10-08T23:47:54.687545119Z” level=debug msg="Trying to push docker.io/chakrar27/couchbase_new_10_08_2016 to https://registry-1.docker.io v2"
time=“2016-10-08T23:47:55.186330356Z” level=debug msg="Pushing repository: chakrar27/couchbase_new_10_08_2016:latest"
time=“2016-10-08T23:47:55.739246099Z” level=debug msg="Increasing token expiration to: 60 seconds"
time=“2016-10-08T23:47:56.608027584Z” level=debug msg="Pushing layer: sha256:f8f734d0eeb7aa1de490bba304b14efb59fe2eb9ecb8e066aab61ee87104be38"
time=“2016-10-08T23:47:57.300409189Z” level=debug msg="Assembling tar data for d10a8842bdaf6c8eae2c287663e95fe3183eb33df2174c5ee8619d9675a432f8"
time=“2016-10-08T23:53:14.196613332Z” level=debug msg="Increasing token expiration to: 60 seconds"
time=“2016-10-08T23:53:16.081267282Z” level=debug msg=“uploaded layer sha256:f8f734d0eeb7aa1de490bba304b14efb59fe2eb9ecb8e066aab61ee87104be38 (sha256:a22ce94d08c28da7d0354d46028d0e03aeffb29c35bba99021fea8c7197acbd2), 131574298 bytes”

The docker push just hangs and waits indefinitely. It never completes.

gioppoluca · April 11, 2017, 9:03am

Same problem here.
I’m on CentOS 7
Docker version 17.04.0-ce, build 4845c56

The push command just hangs there doing nothing, no progress no messages no clue.
Just this:
86c4534765bb: Pushing [==> ] 655.9kB/11.03MB
and waiting.
Is there any logs apart journal that does not says anything useful to look for?
Any known workaround?

lrkirven · May 21, 2017, 5:44am

Same issue here as well. I am on GNU/Linux. Push command just hangs with the following display:

62197bea5fec: Pushing [============================================> ] 343.3 MB/388.6 MB
42d261a184f3: Pushing [============================================> ] 388.6 MB/432 MB
62197bea5fec: Pushing [==================================================>] 390 MB
42d261a184f3: Pushing [=============================================> ] 396.2 MB/432 MB
42d261a184f3: Pushing [===============================================> ] 412.3 MB/432 MB

42d261a184f3: Pushed

Appreciate any help, advice or workaround.

willdenzhu · July 10, 2017, 6:25am

Same issue here on docker for windows :

PS D:\www3\Paper> docker push willdenzhu/han
The push refers to a repository [docker.io/willdenzhu/han]
bd2590719480: Layer already exists
8a33f70d7f27: Layer already exists
c4799d497f4d: Layer already exists
0e0af71ed238: Layer already exists
bd159a140ca7: Layer already exists
b78ae116ec45: Layer already exists
6a7a54017432: Layer already exists
ba4b481aadc4: Layer already exists
e5580892769f: Layer already exists
8108b96ed344: Layer already exists
acd74657641a: Layer already exists
753f5851a4ea: Layer already exists
98ac6c32b9bf: Layer already exists
8c31eae7fe91: Layer already exists
958c46160919: Waiting

How do you guys solve this problem at the end?

boonzaai · December 11, 2018, 10:36am

Had exactly the same problem. Restarting docker fixed it.

nachmangeva · February 11, 2019, 9:54am

Yep - restart of docker “fixed” it…

simonmiller99 · June 26, 2019, 11:52am

I had this and it proved to be an incorrect specification of the nexus repo url in the noproxy variable in docker https-proxy.conf- I’d used a wildcard but it did not seem to evaluate it when pushing. Details here https://github.com/docker/distribution/issues/2147 at this post where he was kind enough to post a permanent solution

chrisbabayans · February 12, 2021, 9:22pm

Such a simple solution. I closed my laptop lid and when I reopened it, I quit the “docker push”, ran it again, and it was stuck at the same upload amount. I’m using the M1 preview

douglas0n · January 28, 2022, 3:13pm

I had a similar problem pushing to a private docker repository, and the problem was wrong credentials.

Try to read the logs:

Get-EventLog -LogName Application -Source Docker -After (Get-Date).AddMinutes(-5) | Sort-Object Time

ref: Troubleshooting Windows Containers | Microsoft Docs