More Trouble with Local Registries

Worked this problem for a couple of days now. Just updated (and synced client/server versions) to 1.17, hoping to fix the problem but no luck.

Here’s the story: I have inherited a local Docker registry and after setting up a new client machine, I am unable to run existing Dockerfiles because I can’t pull from the local repo.

I’m new at this, so not sure how to troubleshoot. What’s more scary is that, although I see similar problems, I don’t see that anyone is running into this one. Please Help. How can I see why the registry is giving me 403???

Started the registry as per (https://docs.docker.com/registry/deploying/))

I have specified the --insecure-registry as evidences in the CLIENT docker info. I think this is the correct configuration.

When I attempt to build a Dockerfile with it’s first line being:
FROM IP.OF.LOCAL.REPO:5000/consul
Pulling repository IP.OF.LOCAL.REPO:5000/consul
Error: Status 403 trying to pull repository consul: <HEAD><TITLE>Access Y BGCOLOR=\"white\" FGCOLOR=\"black\"><H1>Access Denied</H1><HR>\n<FONT F\nDescription: You are not allowed to access the document at location <em2\" cols=\"75\">http://IP.OF.LOCAL.REPO:5000/v1/repositories/consul/images<n<HR>\n<!-- default \"Access Denied\" response (403) -->\n</BODY>\n

I tried the following from the documentation:
docker pull ubuntu
docker tag ubuntu IP.OF.MY.REGISTRY:5000/ubuntu
docker push IP.OF.MY.REGISTRY:5000/ubuntu

and received the same error:
56827159aa8b: Preparing
440e02c3dcde: Preparing
29660d0e5bb2: Preparing
85782553e37a: Preparing
745f5be9952c: Preparing

Error: Status 403 trying to push repository ubuntu: <HEAD><TITLE>Access Denied</TITLE></HEAD>\n<BODY BGCOLOR=\"white\" FGCOLOR=\"black\"><H1>Access Denied</H1><HR>\n<FONT FACE=\"Helvetica,Arial\"><B>\nDescription: You are not allowed to access the document at location <em><TEXTAREA READONLY rows=\"2\" cols=\"75\">http://IP.OF.MY.REGISTRY:5000/v1/repositories/ubuntu/</TEXTAREA></em></B></FONT>\n<HR>\n<!-- default \"Access Denied\" response (403) -->\n</BODY>\n

Server hosting the repo:
~]$ ps -aef | grep docker
root 1563 1 2 15:50 ? 00:01:52 /usr/bin/dockerd
root 1582 1563 0 15:50 ? 00:00:04 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc
root 5717 1563 0 17:09 ? 00:00:00 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 5000 -container-ip 10.0.18.2 -container-port 5000
root 5722 1582 0 17:09 ? 00:00:00 docker-containerd-shim c332e4ccaddb992ddf51b38b65c20171571ab7c80ccc16631d7dfd8368201f45 /var/run/docker/libcontainerd/c332e4ccaddb992ddf51b38b65c20171571ab7c80ccc16631d7dfd8368201f45 docker-runc
root 5739 5722 4 17:09 ? 00:00:00 registry serve /etc/docker/registry/config.yml
jshanno+ 5764 3384 0 17:09 pts/0 00:00:00 grep --color=auto docker

Note: I have not created the /etc/docker/registry/config.yml file. Do I need to override some defaults???

On the server:
~]$ docker info
Containers: 2
Running: 1
Paused: 0
Stopped: 1
Images: 124
Server Version: 17.03.1-ce
Storage Driver: devicemapper
Pool Name: docker-253:0-103563254-pool
Pool Blocksize: 65.54 kB
Base Device Size: 107.4 GB
Backing Filesystem: ext4
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 2.984 GB
Data Space Total: 107.4 GB
Data Space Available: 6.922 GB
Metadata Space Used: 6.279 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.141 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use --storage-opt dm.thinpooldev to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-229.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.954 GiB
Name: dtcrevere3
ID: NJFU:XVBW:KHTD:G6RT:XWFV:3Q3L:F4FX:YUVL:2IOY:VOKX:JH7P:IBGO
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Http Proxy: MY.PROXY.IP:80/
Registry: ‘https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

On the Client:
$ docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 29
Server Version: 17.03.1-ce
Storage Driver: devicemapper
Pool Name: docker-253:0-134399500-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 937.6 MB
Data Space Total: 107.4 GB
Data Space Available: 47.35 GB
Metadata Space Used: 1.749 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.146 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use --storage-opt dm.thinpooldev to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-514.10.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.64 GiB
Name: centos7-MYNAME
ID: ZAHL:BW2A:JKTP:WBDJ:IJWB:H3O5:ASPV:5GXR:AXQ6:VROJ:K2X6:GPTW
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Http Proxy: IP.OF.MY.PROXY:80/
Registry: ‘https://index.docker.io/v1/
Experimental: false
Insecure Registries:
IP.OF.REMOTE.REGISTRY:5000
127.0.0.0/8
Live Restore Enabled: false

Any ideas on this? How does the Docker registry enforce authorization???

I’m facing the same issue.

Have you managed to fix this?

We have not. We had to give up for a time and resort to moving the images to the machines where they would run. We have, however, picked this task back up and a member of my team is working on configuring the “secure solution” using certificates in hopes of being able to pull from remote private registries. Hopefully, we’ll get it solved in a day or two.