I’ve an issue using “FROM” in my Dockerfile. I will explain.
My environment is Linux.
I try to create a Kubernetes pod with Harbor registry on it, using “docker in docker”.
Install “docker in docker” and Harbor works without issue. Certificates are generated, used by docker and Harbor.
For Harbor hostname I use “kube-registry.my_local.domain” and use Harbor port “AAAAA”. To access to Harbor pod I created a service with “port BBBBB to port AAAAA”.
Certificates are in /etc/docker/certs.d/kube-registry.my_local.domain:BBBBB/.
I can access to Harbor webui and create projects.
I can do a “docker login kube-registry.my_local.domain:BBBBB” without issue.
After that I try to create an image (test:v1) using Dockerfile (“simple” image using rockylinux) and push it on the pod registry. It works, image is on registry.
But when I want to use image “test:v1” (which is on pod registry) with another Dockerfile to create a second image (test1:v1) using “FROM kube-registry.my_local.domain:BBBBB//test:v1” the build stop with an TLS error.
If I test a “docker pull kube-registry.my_local.domain:BBBBB//test:v1” it works.
I try to create /etc/docker/daemon.json with “insecure” entry but always KO.
Have you tried adding the CA certificate of your domain to Linux system as well? Images are built with buildkit which don’t use Docker containers for the temporary build containers. But I’m not sure if it is needed. I never used not globally accepted TLS for aregistry domain and it was years ago when I last used insecure registries probably before the current builder.
(Sorry for the short replies, I only have time for these at the moment)
I don’t think it’s needed to add CA certificate on Linux system.
Actually I use Harbor registry installed on a physical server, using let’sencrypt certificates, and I didn’t needed to add CA certificate on Linux system or to put certificates in /etc/docker/<registry_hostname>:<registry_port>. It’s why I don’t understand where is the problem.
configure Harbor using hostname “kube-registry.my_external.domain”, port “BBBBB” and my_external.domain let’s encrypt certificates (personal domain)
don’t push any certificates on Linux system and in /etc/docker/certs.d/kube-registry.my_external.domain:BBBBB
Résults:
docker login: works
docker push: works
docker pull: works
create image with Dockerfile (FROM kube-registry.my_external.domain//:): works
But my goal is to have only internal registry access with kube-registry.my_local.domain.
Hostname “kube-registry” is a cname of my Kubernetes master (saved on my personal DNS “.my_local.domain”) because of using a Kubernetes Service.
Certificates needed for docker and Harbor are generated manually (openssl etc…).
The next test I will do is to use “kube-registry” without domain specified, with and without putting manual certificates in /etc/docker/certs.d/.
Please share the manifest of the service (of course obfuscate external ips if visible). Do cluster external and cluster internal traffic both communicate directly with the service (nodePort?), or does external traffic enter the cluster through ingress? Where does TLS offloading happen? In ingress? Or in the pod itself?
This also answers the question of where TLS termination happens. It must be in the pod.
Now one thing confuses me.
external traffic resolves the domain to the ip of one or more k8s nodes and use the node port 30700
internal traffic resolves the domain to the cluster and must be using the port 40701
Which port is BBBBB in your posts?
Why didn’t you just use kubevip or metallb, so you could create a sevice of type LoadBalancer, make sure the hostname resolves to the loadbalancer ip, so the ip and port would remain the same for internal and external access?
I’m a beginner in Kubernetes. Since 1 year I train myself in Kubernetes, after 2 years in Openshift professionally.
I managed to create a kubernetes cluster where application pods like mariaDB, nextcloud, zabbix and some personal site urls are running. But I know I’m missing a lot of things (like ingress) to improve my skills and I’m trying to learn them at my own pace.
About ports:
when I use URL https://kube-registry.my_external.domain/ I can access to Harbor webui (a Nginx proxy redirects port 40701 to port 30700 on my Kubernetes master (so access pod by the Nodeport service)
when I use URL kube-registry:30700 for “docker login” or “docker push” or “docker pull” all works
only using “FROM kube-registry:30700” to a Dockerfile doesn’t work
The error message :
ERROR: failed to build: failed to solve: failed to fetch oauth token: Post “https://kube-registry:30700/service/token”: tls: failed to verify certificate: x509: certificate signed by unknown authority
After some checks using curl -v that didn’t work I tried specifying CA certificate (curl -v --cacert) and it works.
So I have to copy the CA certificate on Linux system (/etc/pki/ca-trust/source/anchors/ on Rocky Linux in my case), do an “update-ca-trust extract”. And now, “FROM” in my Dockerfile works.
I was convinced that it was not useful despite rimelek’s suggestion. But he was right.
