Docker push not working, but pull does. Private registry

Docker Engine
1

When I try to push using docker 1.8.1 on Ubuntu 14.04 to a private registry (self signed key) I get this message:

Username (user): user
WARNING: login credentials saved in /home/MyUser/.docker/config.json
Login Succeeded
The push refers to a repository [private.repository:443/test/my-image] (len: 1)
59d9302fffa1: Pushing [==================================================>]     32 B/32 B
unauthorized: access to the requested resource is not authorized

I use a private registry and a self signed key (–insecure option for the url in /etc/default/docker).

This worked with 1.7.1 but no 1.8.1 and the ubuntu repos only have the newest version so no idé to downgrade.

Pull works fine, it’s only push that complains.

Any suggestions?

2

Is this for the Docker Hub or for a private registry via self-hosted docker/distribution (or is it DTR)? Should we move this to https://forums.docker.com/c/opensrcreg ?

3

It’s via self-hosted docker/distribution. Version 2.0 so I will try to update it to 2.1.1 and see if the problem persists.

4

The problem still persists on registry 2.1.1.

I have a Nginx proxy (followed the guide at docker registry 2)

These are the logs from the registry after a push-try (some names are changed for security reasons):

time="2015-08-20T06:27:40Z" level=info msg="response completed" http.request.host=registryUrl.dns.se http.request.id=7dd7fd8d-6a5c-47c8-bfa6-2bd4d34e36ed http.request.method=POST http.request.remoteaddr=131.116.79.99 http.request.uri="/v2/registryUrl/eap/blobs/uploads/" http.request.useragent="docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.13.0-62-generic os/linux arch/amd64" http.response.duration=14.619201ms http.response.status=202 http.response.written=0 instance.id=ffc9fe11-53fd-459b-81e7-b405c926cbcb version=v2.1.1 
172.17.0.4 - - [20/Aug/2015:06:27:40 +0000] "POST /v2/registryUrl/eap/blobs/uploads/ HTTP/1.0" 202 0 "" "docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.13.0-62-generic os/linux arch/amd64"
time="2015-08-20T06:28:09Z" level=info msg="response completed" http.request.host=registryUrl.dns.se http.request.id=30081122-b22d-4fc9-8110-60c57d187f6e http.request.method=GET http.request.remoteaddr=131.116.79.99 http.request.uri="/v2/" http.request.useragent="docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.13.0-62-generic os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=4.105522ms http.response.status=200 http.response.written=2 instance.id=ffc9fe11-53fd-459b-81e7-b405c926cbcb version=v2.1.1 
172.17.0.4 - - [20/Aug/2015:06:28:09 +0000] "GET /v2/ HTTP/1.0" 200 2 "" "docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.13.0-62-generic os/linux arch/amd64"
time="2015-08-20T06:30:04Z" level=info msg="response completed" http.request.host=registryUrl.dns.se http.request.id=8f03f998-4d40-422d-a874-e0974e8e570f http.request.method=POST http.request.remoteaddr=131.116.79.99 http.request.uri="/v2/registryUrl/eap/blobs/uploads/" http.request.useragent="docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.13.0-62-generic os/linux arch/amd64" http.response.duration=9.095604ms http.response.status=202 http.response.written=0 instance.id=ffc9fe11-53fd-459b-81e7-b405c926cbcb version=v2.1.1 
172.17.0.4 - - [20/Aug/2015:06:30:04 +0000] "POST /v2/registryUrl/eap/blobs/uploads/ HTTP/1.0" 202 0 "" "docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.13.0-62-generic os/linux arch/amd64"

Downgrading to 1.7.1 worked, so the problem has appeared in 1.8.x

5

I’m experiencing a similar issue using v1.8.2 of Docker installed as part of the latest OSX dockertoolbox installation.

In my case, pulls and pushes to our internal, private, insecure registry are no longer working. As this is currently an insecure registry we don’t even have the added complexity of having an nginx frontend.

The previous installation (boot2docker using docker v1.7.1) worked fine on my mac. VMs in our cloud running Ubuntu 14.04 with docker 1.8.1 work perfectly. In each case, the docker daemon is configured to observe our proxies as well as being configured to recognize our internal insecure proxy.

The private registry is the distribution/registry:2.1.1 container from docker hub and is configured to use our internal cloud’s blog store.

I’ve collected the following logs. First from a working Ubuntu 14.04/Docker 1.8.1 combo:

> DEBU[0185] Calling POST /images/create                  
> INFO[0185] POST /v1.20/images/create?fromImage=xx.xx.xx.xx%2Fmsi%2Fttt%3Alatest 
> DEBU[0185] Trying to pull hub.cc.mot-solutions.com/msi/ttt from https://xx.xx.xx.xx v2 
> DEBU[0185] Error getting v2 registry: Get https://xx.xx.xx.xx/v2/: dial tcp x.x.x.x:443: connection refused
> DEBU[0185] Trying to pull xx.xx.xx.xx/msi/ttt from https://hub.cc.mot-solutions.com v1 
> DEBU[0185] attempting v2 ping for registry endpoint https://xx.xx.xx.xx/v2/ 
> DEBU[0185] attempting v1 ping for registry endpoint https://xx.xx.xx.xx/v1/ 
> DEBU[0185] Fallback from error: unable to ping registry endpoint https://xx.xx.xx.xx/v0/
> v2 ping attempt failed with error: Get https://xx.xx.xx.xx/v2/: dial tcp x.x.x.x:443: connection refused
>  v1 ping attempt failed with error: Get https://xx.xx.xx.xx/v1/_ping: dial tcp x.x.x.x:443: connection refused 
> DEBU[0185] Trying to pull xx.xx.xx.xx/msi/ttt from http://xx.xx.xx.xx v2 
> DEBU[0185] Pulling tag from V2 registry: "latest"       
> DEBU[0185] Verification failed for /msi/ttt using key DO3I:UPSJ:MUOG:XNTO:3MAT:CB5H:CWD2:EKNO:FMO3:WXC5:45IX:OFJ6 
> DEBU[0185] Key check result: not verified               
> DEBU[0185] Image already exists: 902b87aaaec929e80541486828959f14fa061f529ad7f37ab300d4ef9f3a0dbf 
> DEBU[0185] Image already exists: 9a61b6b1315e6b457c31a03346ab94486a2f5397f4a82219bee01eead1c34c2e 
> DEBU[0185] Image already exists: 1ff9f26f09fb1bc7b5955c269b1042429e86d7891c653f52f3e48f1e0365d7df 
> DEBU[0185] Image already exists: 607e965985c11e6a23270feec487908aeaa9af763d24a2986866a41537770c8c 
> DEBU[0185] pulling blob "sha256:412dfc97bdb313dd58d80b3599e24e7481e0a82a4a6317b0dc552e439b29c94a" to 9213e81cb0f2a81133c82bc9
> 3394b600f532866e3aaf78d9c5dee79bedc64704 
> DEBU[0185] pulling blob "sha256:f179f115203c68564c1dab6314a7a1e763e4583e17cc0f41fc25c6f2b53184a8" to 6ad0799af6bd83f609ac1e13
> c73ed8c0a5152ce7f45f13a9a6e0042831901eac 
> DEBU[0185] pulling blob "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to e0919a8b95a85a196bb65e8c
> fe39ea958f938b75012103065cea9823938041a4 
> DEBU[0185] pulling blob "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to 3383909e8f95e058d1c56d69
> 02963fc1ea269044d1f556f7251c48f14f3e20fc 
> DEBU[0185] pulling blob "sha256:d643093d6bf27f87b72aa158446d533b088b2bbf00d1c679168930ef53484835" to 8b74d7a75802a08262803358
> cc14241ed1872ad0049ad641379c6bfcd9b4c922 
> DEBU[0185] pulling blob "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to 9e20baae42c83cc306eb34dc
> 87bf3661e240a1ed8428e95b8376e1c9a04c16e0 
> DEBU[0185] pulling blob "sha256:0fbd7ceb1413de24fffa0ca02876071c72b62ced2b5db1c2b2fe66af79d53239" to 10d57d1f991d28d771d67414
> 6f7de2a3fcd7938d82534d862b80c58fe8114e7c 
> DEBU[0185] pulling blob "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to 9605e1520da477bf1f438351
> c3d8cc0b372dc629cffb62362e29cf6171771672 
> DEBU[0185] pulling blob "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to ceebe4c43d37d30ce4c3ef89
> d23370e4b3b613b868b0df4ee6caafa6cea936b9 
> DEBU[0185] pulling blob "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to 6fca045f3c05d4d88b9b8795
> 2fba4d3bd35a50fbd58da6a8bf589f2c995ab8ea 
> DEBU[0185] pulling blob "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to a4377ebe6678bae30da0b038
> 4f6c260bbf2ed9eb658d958f01ff9867c091a6be 
> DEBU[0185] pulling blob "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to b278f036bf4694ffaf4d5020
> 1eec9d926f8a39fe2743de17c92ceccb8904aba4 
> DEBU[0185] pulling blob "sha256:bc06005d18a45199c970c3c5aeef00bfeeedf4d90ab9297af849037e990055df" to d72441fb995ee4ee9afd2b96
> f7cd385e9ee5ae2d7f58bed8dd9bf4f4221d227c 
> DEBU[0185] pulling blob "sha256:1aa0a5e5cfaf4c1cd587d76fa085704347d8d81ec0687a687db110bdb18b8811" to b40a386420f6f7e2303f98d1
> b5b69f9dc90ca4221efba9f03c4e9b49b8686f81 
> DEBU[0185] pulling blob "sha256:1c274d12c8d85a5319aa5e1986bf30f189f48bb0525193200c3cf14c9647dd7a" to 631f7a4d6531e23d0d1a2986
> 9930ec02005323466f86b88552b04786a2043d12 
> DEBU[0185] pulling blob "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" to c2dbd396d644c1ab1236bb3b
> 260746e2a0ad516be92ce120ebc09236a835992b 
> DEBU[0186] Downloaded 6fca045f3c05d4d88b9b87952fba4d3bd35a50fbd58da6a8bf589f2c995ab8ea to tempfile /var/lib/docker/tmp/GetIma
> geBlob379842359 
> DEBU[0186] Downloaded 6ad0799af6bd83f609ac1e13c73ed8c0a5152ce7f45f13a9a6e0042831901eac to tempfile /var/lib/docker/tmp/GetIma
> geBlob018267055 
> DEBU[0186] Downloaded b278f036bf4694ffaf4d50201eec9d926f8a39fe2743de17c92ceccb8904aba4 to tempfile /var/lib/docker/tmp/GetIma
> geBlob454729345 
> DEBU[0186] Downloaded ceebe4c43d37d30ce4c3ef89d23370e4b3b613b868b0df4ee6caafa6cea936b9 to tempfile /var/lib/docker/tmp/GetIma
> geBlob716172088 
> DEBU[0186] Downloaded a4377ebe6678bae30da0b0384f6c260bbf2ed9eb658d958f01ff9867c091a6be to tempfile /var/lib/docker/tmp/GetIma
> geBlob110342442 
> DEBU[0186] Downloaded 10d57d1f991d28d771d674146f7de2a3fcd7938d82534d862b80c58fe8114e7c to tempfile /var/lib/docker/tmp/GetIma
> geBlob036686230 
> DEBU[0186] Downloaded e0919a8b95a85a196bb65e8cfe39ea958f938b75012103065cea9823938041a4 to tempfile /var/lib/docker/tmp/GetIma
> geBlob658207554 
> DEBU[0186] Downloaded 3383909e8f95e058d1c56d6902963fc1ea269044d1f556f7251c48f14f3e20fc to tempfile /var/lib/docker/tmp/GetIma
> geBlob569690041 
> DEBU[0186] Downloaded d72441fb995ee4ee9afd2b96f7cd385e9ee5ae2d7f58bed8dd9bf4f4221d227c to tempfile /var/lib/docker/tmp/GetIma
> geBlob369382892 
> DEBU[0186] Downloaded 9e20baae42c83cc306eb34dc87bf3661e240a1ed8428e95b8376e1c9a04c16e0 to tempfile /var/lib/docker/tmp/GetIma
> geBlob286943315 
> DEBU[0186] Downloaded 631f7a4d6531e23d0d1a29869930ec02005323466f86b88552b04786a2043d12 to tempfile /var/lib/docker/tmp/GetIma
> geBlob990302718 
> DEBU[0186] Downloaded 9605e1520da477bf1f438351c3d8cc0b372dc629cffb62362e29cf6171771672 to tempfile /var/lib/docker/tmp/GetIma
> geBlob150381053 
> DEBU[0186] Downloaded c2dbd396d644c1ab1236bb3b260746e2a0ad516be92ce120ebc09236a835992b to tempfile /var/lib/docker/tmp/GetIma
> geBlob341216069 
> DEBU[0186] Downloaded b40a386420f6f7e2303f98d1b5b69f9dc90ca4221efba9f03c4e9b49b8686f81 to tempfile /var/lib/docker/tmp/GetIma
> geBlob402985563 
> DEBU[0187] Downloaded 8b74d7a75802a08262803358cc14241ed1872ad0049ad641379c6bfcd9b4c922 to tempfile /var/lib/docker/tmp/GetIma
> geBlob643953604 
> DEBU[0189] Downloaded 9213e81cb0f2a81133c82bc93394b600f532866e3aaf78d9c5dee79bedc64704 to tempfile /var/lib/docker/tmp/GetIma
> geBlob130237840...

And then from the non-working Docker-machine/docker 1.8.2 installation:

> time="2015-09-17T19:22:33.258204000Z" level=debug msg="Calling POST /images/create" 
> time="2015-09-17T19:22:33.258295705Z" level=info msg="POST /v1.20/images/create?fromImage=hub.cc.mot-solutions.com%2Fmsi%2Fttt%3Alatest" 
> time="2015-09-17T19:22:33.258467974Z" level=debug msg="Trying to pull xx.xx.xx.xx/msi/ttt from https://xx.xx.xx.xx v2" 
> time="2015-09-17T19:22:38.260460847Z" level=debug msg="Error getting v2 registry: Get https://xx.xx.xx.xx/v2/: net/http: request canceled while waiting for connection" 
> time="2015-09-17T19:22:38.261639387Z" level=debug msg="Trying to pull xx.xx.xx.xx/msi/ttt from https://xx.xx.xx.xx v1" 
> time="2015-09-17T19:22:38.261737722Z" level=debug msg="attempting v2 ping for registry endpoint https://xx.xx.xx.xx/v2/" 
> time="2015-09-17T19:22:43.270482621Z" level=debug msg="attempting v1 ping for registry endpoint https://xx.xx.xx.xx/v1/" 
> time="2015-09-17T19:22:53.285156845Z" level=debug msg="Fallback from error: unable to ping registry endpoint https://xx.xx.xx.xx/v0/\nv2 ping attempt failed with error: Get https://xx.xx.xx.xx/v2/: dial tcp x.x.x.x:443: connection refused\n v1 ping attempt failed with error: Get https://xx.xx.xx.xx/v1/_ping: dial tcp x.x.x.x:443: connection refused" 
> time="2015-09-17T19:22:53.285252399Z" level=debug msg="Trying to pull xx.xx.xx.xx/msi/ttt from http://xx.xx.xx.xx v2" 
> time="2015-09-17T19:22:58.285946512Z" level=debug msg="Error getting v2 registry: Get http://xx.xx.xx.xx/v2/: net/http: request canceled while waiting for connection" 
> time="2015-09-17T19:22:58.287609380Z" level=debug msg="Trying to pull xx.xx.xx.xx/msi/ttt from http://xx.xx.xx.xx v1" 
> time="2015-09-17T19:22:58.287893806Z" level=debug msg="[registry] Calling GET http://xx.xx.xx.xx/v1/repositories/msi/ttt/images" 
> time="2015-09-17T19:23:03.297844289Z" level=debug msg="Not continuing with error: Error: image msi/ttt:latest not found"

Unfortunately, I don’t have logs from the registry itself (yet). All log messages above are from the client’s perspective.

Any suggestions?

1 Like