Docker Community Forums

Share and learn in the Docker community.

Docker Push returns 500 Internal Server Error after freeing up disk space

docker
(Limeade) #1

We run repository:2 to store our build images on. This afternoon, the machine this container runs on ran out of disk space, so any docker push command would just retry a bunch of times and give up. So, I stopped the container, did a docker rm -f repository to delete the image, then ran a docker system prune -a --volume --force to pretty much clear out everything, which freed up about 150 gigs. Next, I pulled the container down and ran it again (same way since I have a script that starts that container). Now, when I do a docker push, I get the following:

66a3a7fb3292: Pushed
dfc462700612: Pushed
b5752b56608f: Pushed
4f11e6c7d729: Pushed
5786c3ba2e0b: Pushed
6a6c42c9e6ca: Pushed
1c1c7f94dd33: Pushed
bf5bcdf908be: Pushed
73676e63b8a9: Pushed
77fee8b28217: Pushed
66820bef07e2: Pushed
e34d021482bd: Pushed
5a54807842af: Pushed
beddfd1cfc9f: Pushed
f5856416f664: Pushed
a0519c409c1d: Pushed
d9e066c569be: Pushed
5561dd586013: Pushed
b4e376d18ad3: Pushed
b15dca70a64b: Pushed
4ba0c3e137a6: Pushed
f11ccd9aa0da: Pushed
ba3fef3cafcf: Pushed
5c9d8eb8792e: Pushed
5511599ef865: Pushed
427a52df0e50: Pushed
5670c24bacb2: Pushed
0266cdc1840a: Pushed
6dcb5341138b: Pushed
45f8949007cb: Pushed
b35d95ca2064: Pushed
6537cbfba06f: Pushed
b0e8a9a91e06: Pushed
272abef7cc5d: Pushed
f7342e1fbb9b: Pushed
e69d410f03bf: Pushed
24770c391e95: Pushed
e7d9b1b6b528: Pushed
6aab3951d60e: Pushed
dd206df6e1d4: Pushed
37dcb06a3cfd: Pushed
b7616fcb3e64: Pushed
e5cc5667e429: Skipped foreign layer
f358be10862c: Skipped foreign layer
received unexpected HTTP status: 500 Internal Server Error

I get the same error from any machine I run the docker push from, so it’s not that machine. The repository stores all the images in a Docker volume which is at /var/snap/docker/common/var-lib-docker/volumes/registry/_data. If I look in this directory, I see the images I pushed in /docker/registry/v2/repositories so it seems like they’ve been pushed. However, if I run a docker pull from another machine, it tells me the image isn’t found.

Any idea what broke?

Update: I believe these are logs from the container.

 time="2019-02-01T23:58:58.63253274Z" level=error msg="response completed with error" err.code=unknown err.message="unknown error" go.version=g   ¦
¦   o1.11.2 http.request.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.request.host=hqdocker.xxxxxxxx.com http.request.   ¦
¦   id=c99a311d-3e24-40ed-9088-100b6908e0be http.request.method=PUT http.request.remoteaddr="172.16.1.123:50910" http.request.uri="/v2/web/manifes   ¦
¦   ts/staging" http.request.useragent="docker/18.09.1 go/go1.10.6 git-commit/4c52b90 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.1 \   ¦
¦   (windows\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.224772ms http.response.status=500 http.respo   ¦
¦   nse.written=523 vars.name=web vars.reference=staging                                                                                             ¦
¦   time="2019-02-01T23:58:58.632690415Z" level=error msg="response completed with error" err.code=unknown err.message="invalid URL on layer" go.v   ¦
¦   ersion=go1.11.2 http.request.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.request.host=hqdocker.xxxxxxxx.com http.   ¦
¦   request.id=c99a311d-3e24-40ed-9088-100b6908e0be http.request.method=PUT http.request.remoteaddr="172.16.1.123:50910" http.request.uri="/v2/web   ¦
¦   /manifests/staging" http.request.useragent="docker/18.09.1 go/go1.10.6 git-commit/4c52b90 os/windows arch/amd64 UpstreamClient(Docker-Client/1   ¦
¦   8.09.1 \(windows\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.383189ms http.response.status=500 ht   ¦
¦   tp.response.written=523 vars.name=web vars.reference=staging                                                                                     ¦
¦   time="2019-02-01T23:58:58.632782359Z" level=error msg="response completed with error" err.code="manifest blob unknown" err.detail=sha256:3889b   ¦
¦   b8d808bbae6fa5a33e07093e65c31371bcf9e4c38c21be6b9af52ad1548 err.message="blob unknown to registry" go.version=go1.11.2 http.request.contenttyp   ¦
¦   e="application/vnd.docker.distribution.manifest.v2+json" http.request.host=hqdocker.xxxxxxxx.com http.request.id=c99a311d-3e24-40ed-9088-100b   ¦
¦   6908e0be http.request.method=PUT http.request.remoteaddr="172.16.1.123:50910" http.request.uri="/v2/web/manifests/staging" http.request.userag   ¦
¦   ent="docker/18.09.1 go/go1.10.6 git-commit/4c52b90 os/windows arch/amd64 UpstreamClient(Docker-Client/18.09.1 \(windows\))" http.response.cont   ¦
¦   enttype="application/json; charset=utf-8" http.response.duration=1.474407ms http.response.status=500 http.response.written=523 vars.name=web v   ¦
¦   ars.reference=staging