Push to docker registry inside kubernetes doesn't even attempt conection

rafaeltab · June 2, 2021, 10:08pm

Dear humans of the docker community,

I have been attempting to host a docker registry inside local docker-desktop Kubernetes for the past few hours. I’ve got the pod running, can open the catalog inside my browser on the host machine but got a lot of problems when trying to push images.

When I try to push my image with the following command: docker push registry.me:5000/rafaeltab/kubetest/authentication-service I get the following error:

Using default tag: latest
The push refers to repository [registry.me:5000/rafaeltab/kubetest/authentication-service]
Get http://registry.me:5000/v2/: dial tcp 127.0.0.1:5000: connect: connection refused

Registry.me is set as hostname that refers to 127.0.0.1.
I have set up TLS certificates for the registry to try and fix the issue to no avail. (I tried with insecure registries before, which gave the same error).

I run a port forward to the pod using kubectl port-forward kube-registry-v0-rlqdj --address 127.0.0.1 5000:5000 and I see that it is handling a connection when I load the catalog in my browser but not when I do the push. Which leads me to believe it isn’t even attempting a connection. I see no logs created inside the pod when doing the push, I do when opening it in my browser.

If you need any more info just ask ill post it asap.

Thanks, Rafael

rafaeltab · June 3, 2021, 11:57am

I figured it out! I figured it out!

What I had to do was: change the DNS for registry.me to my PC’s IP for the LAN. 192.168.178.130 in this case. Delete the Daemon Set I had and add SSL. And that worked! It took a while but I made it. Finally.

P.S. no clue if you can close discussions so if it is possible please either do or tell me how.

rafaeltab · June 3, 2021, 2:05pm

Since I’ve got some time on my hands, I’ll explain the exact steps that I had to take to make it work.

Step 1

First get your local LAN’s IP address:
ipconfig on Windows
ip addr on Linux

This should contain an IP in any of the following ranges:

Address range	Subnet mask	Provides	Addresses per LAN
10.0.0.0 – 10.255.255.255.255	255.0.0.0	1 class A LAN	16,777,216
172.16.0.0 – 172.31.255.255	255.255.0.0	16 class B LANs	65,536
192.168.0.0 – 192.168.255.255	25.255.255.0	256 class C LANs	256

Remember this address or write it down somewhere.

You need a domain name in your hosts file. On Windows, this file is located in
C:\Windows\System32\Drivers\Etc\hosts
on Linux, in
/etc/hosts

Add a line to this file in the following format without the < and >

<ip-address from earlier> <domain name>

an example is

192.168.178.130 registry.me

Step 2
Choose a folder where you’re going to put all the configuration. For me this was inside a kubeconfig folder I’d created for my project

Step 3
Create an openssl.cnf:

[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no

[req_distinguished_name]
C = <Country>
ST = <Province>
L = <City>
O = <Orginization>
CN = <The domain name>

[v3_req]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
basicConstraints = CA:TRUE
subjectAltName = @alt_names

[alt_names]
DNS.1 = <The domain name>

Step 4 A
Create SSL certificates inside a certs folder. I ran these 2 commands on my WSL Ubuntu client:

openssl genrsa -out mysite.key 3072 -nodes
openssl req -new -x509 -key certs/tls.key -sha256 -config openssl.cnf -out certs/tls.crt -days 730

Step 4 B

Linux : Copy the tls.crt file to /etc/docker/certs.d/<domain name>:5000/ca.crt on every Docker host. You do not need to restart Docker.
Windows :
1. Open Windows Explorer, right-click the tls.crt file, and choose Install certificate. When prompted, select the following options:
  Local Machine Next
  Place all certificates in the following store
2. Click Browser and select Trusted Root Certificate Authorities or Trusted Root Certification Authorities.
3. Click Finish. Restart Docker.

Step 5
Create a secret inside Kubernetes containing the certificate files:
kubectl create secret tls certs-secret --cert=certs/tls.crt --key=certs/tls.key

Step 6
Create a kube-registry.yaml file with the following content:

apiVersion: v1
kind: ReplicationController
metadata:
    name: kube-registry-v0
    labels:
        k8s-app: kube-registry
        version: v0
spec:
    replicas: 1
    selector:
        k8s-app: kube-registry
        version: v0
    template:
        metadata:
            labels:
                k8s-app: kube-registry
                version: v0
        spec:
            containers:
                - name: registry
                  image: registry:latest
                  resources:
                      # keep request = limit to keep this container in guaranteed class
                      limits:
                          cpu: 500m
                          memory: 500Mi
                      requests:
                          cpu: 500m
                          memory: 500Mi
                  env:
                      - name: REGISTRY_HTTP_ADDR
                        value: 0.0.0.0:5000
                      - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
                        value: /var/lib/registry
                      - name: REGISTRY_HTTP_TLS_CERTIFICATE
                        value: "/certs/tls.crt"
                      - name: REGISTRY_HTTP_TLS_KEY
                        value: "/certs/tls.key"
                  volumeMounts:
                      - name: image-store
                        mountPath: /var/lib/registry
                      - name: certs-vol
                        mountPath: "/certs"
                        readOnly: true
                  ports:
                      - containerPort: 5000
                        name: registry
                        protocol: TCP
            volumes:
                - name: image-store
                  hostPath:
                      path: /data/registry/
                - name: certs-vol
                  secret:
                      secretName: certs-secret

---
apiVersion: v1
kind: Service
metadata:
    name: kube-registry
    labels:
        k8s-app: kube-registry
spec:
    selector:
        k8s-app: kube-registry
    type: LoadBalancer
    ports:
        - name: registry
          port: 5000
          targetPort: registry
          protocol: TCP

The memory and CPU limitations were arbitrarily chosen by me, these might be able to change.

Step 7
apply the file:
kubectl apply -f ./kube-registry.yaml

and wait for everything to be up and running by checking the dashboard or by running docker get pods

Step 8
Verify that you can access the registry through your browser by going to
https://<domain name here>:5000/v2/_catalog
this should return an empty JSON list

Step 9
Build your image and tag it as follows: <domain name>:5000/<rest of name>
push your image using docker push <domain name>:5000/<rest of name>

You can also do both in one step using docker buildx build --push -t <domain name>:5000/<rest of name> .

Step 10
Enjoy!

I think this should be everything. If you have any problems, you can let me know. I won’t guarantee that I answer as I am a very busy person, sorry . Someone else might!