Registry 2, forbidden with daemon 1.13.1 but works with 18.09-CE

Open Source Projects Open Source Registry
1

Hi,

I have setup a docker registry in kubernetes. In first step i want to access it via NodePort which is effectively doing a NAT like if we expose it directly in docker.

I added a self signed certificate to the registry.

  env:
            - name: REGISTRY_HTTP_ADDR
              value: "0.0.0.0:443"
            - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
              value: "/var/lib/registry"
            - name: REGISTRY_HTTP_TLS_CERTIFICATE
              value: "/certs/kubernetes01_intermediate.pem"
            - name: REGISTRY_HTTP_TLS_KEY
              value: "/certs/kubernetes01.key"

the file “/certs/kubernetes01_intermediate.pem” is a concatenation of 1. host-certificate and 2. root-ca-certificate.

I am able to access (push /pull is working) from docker-CE 18.09 after I added the root-ca to trust store by doing the following:

sudo cp ./ca-root.pem /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust
sudo systemctl restart docker

When I do the same steps on docker which comes directly from centos (version 1.13.1), I get following errors:

Jun 12 11:36:34 kubernetes02 dockerd-current[32228]: time="2019-06-12T11:36:34.742485854+02:00" level=warning msg="Error getting v2 registry: Get https://kubernetes01:31001/v2/: Forbidden"
Jun 12 11:36:34 kubernetes02 dockerd-current[32228]: time="2019-06-12T11:36:34.742544561+02:00" level=error msg="Attempting next endpoint for pull after error: Get https://kubernetes01:31001/v2/: Forbidden"
Jun 12 11:36:34 kubernetes02 dockerd-current[32228]: time="2019-06-12T11:36:34.748766030+02:00" level=error msg="Attempting next endpoint for pull after error: Get https://kubernetes01:31001/v1/_ping: Forbidden"

Access via curl does not show any issues. So the self signed ca should be trusted systemwide.

Any help is appreciated.

Thanks, Andreas

2

not sure if version really have impact. Now I have also one docker CE node, which has same issues.
I will try do diff configs, but on first glimpse they look identically.

3

found the issue. To access the internet I need to use a proxy.
in /etc/environment I successfully set up http_proxy, https_proxy and no_proxy variables. So curl was fine.

For docker daemon I declared the proxy in /etc/systemd/system/docker.service.d/http-proxy.conf.
On the working machine no_proxy was specified, on the non working machine no_proxy was missing.

[Service]
Environment="HTTPS_PROXY=http://myproxy:8080/"  "NO_PROXY=localhost,127.0.0.1,server1,server2,docker-registry"
4

I’m glad setting up your proxy environment variables resolved the immediate problem.

One item i did note that is not directly related to your problem is that you had been using a stock engine of version 1.13.x. That version is very old, is missing a lot of features, and is also missing a lot of bug fixtures, that would be available in most newer versions. I’m not sure, but I’m guessing that the standard CentOS repositories only carry the older version of the engine.

Since you’ve been able to compare and contrast using a more modern version of the engine, I’m guessing you’ve figured out how to fetch and install them.

But, for those folks who might encounter this thread and wonder how to install those more recent versions: To install the latest version of the engine, you can visit the documentation for installing the daemon on our docs page. There’s a bit of extra work required to set that up, but it’d be worth it to gain the latest functionality.

5

yes, unfortunately centos and rhel are shipping this old package within their standard repo.

Will dockerCE in Version 18.09. run smooth with kubernetes on top? Because most install manuals just say sudo yum install docker and do not mention the current docker-ce.