Push to docker registry inside kubernetes doesn't even attempt conection

Dear humans of the docker community,

I have been attempting to host a docker registry inside local docker-desktop Kubernetes for the past few hours. I’ve got the pod running, can open the catalog inside my browser on the host machine but got a lot of problems when trying to push images.

When I try to push my image with the following command: docker push registry.me:5000/rafaeltab/kubetest/authentication-service I get the following error:

Using default tag: latest
The push refers to repository [registry.me:5000/rafaeltab/kubetest/authentication-service]
Get http://registry.me:5000/v2/: dial tcp 127.0.0.1:5000: connect: connection refused

Registry.me is set as hostname that refers to 127.0.0.1.
I have set up TLS certificates for the registry to try and fix the issue to no avail. (I tried with insecure registries before, which gave the same error).

I run a port forward to the pod using kubectl port-forward kube-registry-v0-rlqdj --address 127.0.0.1 5000:5000 and I see that it is handling a connection when I load the catalog in my browser but not when I do the push. Which leads me to believe it isn’t even attempting a connection. I see no logs created inside the pod when doing the push, I do when opening it in my browser.

If you need any more info just ask ill post it asap.

Thanks, Rafael

I figured it out! I figured it out!

What I had to do was: change the DNS for registry.me to my PC’s IP for the LAN. 192.168.178.130 in this case. Delete the Daemon Set I had and add SSL. And that worked! It took a while but I made it. Finally.

P.S. no clue if you can close discussions so if it is possible please either do or tell me how.

Since I’ve got some time on my hands, I’ll explain the exact steps that I had to take to make it work.

Step 1

First get your local LAN’s IP address:
ipconfig on Windows
ip addr on Linux

This should contain an IP in any of the following ranges:

Address range Subnet mask Provides Addresses per LAN
10.0.0.0 – 10.255.255.255.255 255.0.0.0 1 class A LAN 16,777,216
172.16.0.0 – 172.31.255.255 255.255.0.0 16 class B LANs 65,536
192.168.0.0 – 192.168.255.255 25.255.255.0 256 class C LANs 256

Remember this address or write it down somewhere.

You need a domain name in your hosts file. On Windows, this file is located in
C:\Windows\System32\Drivers\Etc\hosts
on Linux, in
/etc/hosts

Add a line to this file in the following format without the < and >

<ip-address from earlier> <domain name>

an example is

192.168.178.130 registry.me

Step 2
Choose a folder where you’re going to put all the configuration. For me this was inside a kubeconfig folder I’d created for my project

Step 3
Create an openssl.cnf:

[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no

[req_distinguished_name]
C = <Country>
ST = <Province>
L = <City>
O = <Orginization>
CN = <The domain name>

[v3_req]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
basicConstraints = CA:TRUE
subjectAltName = @alt_names

[alt_names]
DNS.1 = <The domain name>

Step 4 A
Create SSL certificates inside a certs folder. I ran these 2 commands on my WSL Ubuntu client:

openssl genrsa -out mysite.key 3072 -nodes
openssl req -new -x509 -key certs/tls.key -sha256 -config openssl.cnf -out certs/tls.crt -days 730

Step 4 B

  • Linux : Copy the tls.crt file to /etc/docker/certs.d/<domain name>:5000/ca.crt on every Docker host. You do not need to restart Docker.
  • Windows :
    1. Open Windows Explorer, right-click the tls.crt file, and choose Install certificate. When prompted, select the following options:
      Local Machine Next
      Place all certificates in the following store
    2. Click Browser and select Trusted Root Certificate Authorities or Trusted Root Certification Authorities.
    3. Click Finish. Restart Docker.

Step 5
Create a secret inside Kubernetes containing the certificate files:
kubectl create secret tls certs-secret --cert=certs/tls.crt --key=certs/tls.key

Step 6
Create a kube-registry.yaml file with the following content:

apiVersion: v1
kind: ReplicationController
metadata:
    name: kube-registry-v0
    labels:
        k8s-app: kube-registry
        version: v0
spec:
    replicas: 1
    selector:
        k8s-app: kube-registry
        version: v0
    template:
        metadata:
            labels:
                k8s-app: kube-registry
                version: v0
        spec:
            containers:
                - name: registry
                  image: registry:latest
                  resources:
                      # keep request = limit to keep this container in guaranteed class
                      limits:
                          cpu: 500m
                          memory: 500Mi
                      requests:
                          cpu: 500m
                          memory: 500Mi
                  env:
                      - name: REGISTRY_HTTP_ADDR
                        value: 0.0.0.0:5000
                      - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
                        value: /var/lib/registry
                      - name: REGISTRY_HTTP_TLS_CERTIFICATE
                        value: "/certs/tls.crt"
                      - name: REGISTRY_HTTP_TLS_KEY
                        value: "/certs/tls.key"
                  volumeMounts:
                      - name: image-store
                        mountPath: /var/lib/registry
                      - name: certs-vol
                        mountPath: "/certs"
                        readOnly: true
                  ports:
                      - containerPort: 5000
                        name: registry
                        protocol: TCP
            volumes:
                - name: image-store
                  hostPath:
                      path: /data/registry/
                - name: certs-vol
                  secret:
                      secretName: certs-secret

---
apiVersion: v1
kind: Service
metadata:
    name: kube-registry
    labels:
        k8s-app: kube-registry
spec:
    selector:
        k8s-app: kube-registry
    type: LoadBalancer
    ports:
        - name: registry
          port: 5000
          targetPort: registry
          protocol: TCP

The memory and CPU limitations were arbitrarily chosen by me, these might be able to change.

Step 7
apply the file:
kubectl apply -f ./kube-registry.yaml

and wait for everything to be up and running by checking the dashboard or by running docker get pods

Step 8
Verify that you can access the registry through your browser by going to
https://<domain name here>:5000/v2/_catalog
this should return an empty JSON list

Step 9
Build your image and tag it as follows: <domain name>:5000/<rest of name>
push your image using docker push <domain name>:5000/<rest of name>

You can also do both in one step using docker buildx build --push -t <domain name>:5000/<rest of name> .

Step 10
Enjoy!

I think this should be everything. If you have any problems, you can let me know. I won’t guarantee that I answer as I am a very busy person, sorry :frowning:. Someone else might!

1 Like