Pushing image to microk8s

Hello there,
I’m trying to push an image localhost:32000/pain-webapp:registry to the built-in microk8s registry.
microk8s and docker are running on the same machine (Ubuntu 24.04)
docker info prints this:

Client: Docker Engine - Community
 Version:    27.1.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 27.1.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  rootless
  cgroupns
 Kernel Version: 6.8.0-40-generic
 Operating System: Ubuntu 24.04 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.22GiB
 Name: Intel-Silent
 ID: 94ea1c16-68ed-493b-ae02-57dc8a9efab1
 Docker Root Dir: /home/fsajeva/.local/share/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  localhost:32000
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No cpuset support
WARNING: No io.weight support
WARNING: No io.weight (per device) support
WARNING: No io.max (rbps) support
WARNING: No io.max (wbps) support

when I run the push command
docker push localhost:32000/pain-webapp:registry
I get this

The push refers to repository [localhost:32000/pain-webapp]
Get "http://localhost:32000/v2/": dial tcp 127.0.0.1:32000: connect: connection refused

And this happens regardless I’m setting insecure-registries in daemon.json or not.

If I run the same commands (build and push) as rootful docker everything works fine.
The same if I use podman (rootless).

Am I missing something in configuration?
Many thanks

It says “connection refused”. Why do you think it should not happen? Can you use curl to access the API?

curl -vvv localhost:3200

How did you deploy the registry? Is it some kind of built-in, or did you have to deploy it?

This is the response from curl -vs http://localhost:32000/v2/_catalog
repositories array is empty. When the push succedes (in rootful mode) repositories array contains the pushed image

* Host localhost:32000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:32000...
* connect to ::1 port 32000 from ::1 port 44848 failed: Connection refused
*   Trying 127.0.0.1:32000...
* Connected to localhost (127.0.0.1) port 32000
> GET /v2/_catalog HTTP/1.1
> Host: localhost:32000
> User-Agent: curl/8.5.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
< X-Content-Type-Options: nosniff
< Date: Mon, 12 Aug 2024 09:15:50 GMT
< Content-Length: 20
< 
{"repositories":[]}
* Connection #0 to host localhost left intact

The command used in microk8s is sudo microk8s enable registry

If I build and push the same image in rootful mode this is the response

The push refers to repository [localhost:32000/pain-webapp]
13fa864ddfd3: Pushed 
2271c4e44a9f: Pushed 
registry: digest: sha256:baec8362a9456941b8e059c56362924173f31b8c7c51d7ca64c4e11e3ecb3c84 size: 741

And this is the response from the get api of the microk8s built-in registry

* Host localhost:32000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:32000...
* connect to ::1 port 32000 from ::1 port 50236 failed: Connection refused
*   Trying 127.0.0.1:32000...
* Connected to localhost (127.0.0.1) port 32000
> GET /v2/_catalog HTTP/1.1
> Host: localhost:32000
> User-Agent: curl/8.5.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
< X-Content-Type-Options: nosniff
< Date: Mon, 12 Aug 2024 09:25:50 GMT
< Content-Length: 33
< 
{"repositories":["pain-webapp"]}
* Connection #0 to host localhost left intact

As you can see the repository is there. And when I create a deployment on microk8s the image is actually there

REF                                                                                                 TYPE                                                 DIGEST                                                                  SIZE     PLATFORMS   LABELS                          
localhost:32000/pain-webapp:registry                                                                application/vnd.docker.distribution.manifest.v2+json sha256:baec8362a9456941b8e059c56362924173f31b8c7c51d7ca64c4e11e3ecb3c84 78.5 MiB linux/amd64 io.cri-containerd.image=managed 
localhost:32000/pain-webapp@sha256:baec8362a9456941b8e059c56362924173f31b8c7c51d7ca64c4e11e3ecb3c84 application/vnd.docker.distribution.manifest.v2+json sha256:baec8362a9456941b8e059c56362924173f31b8c7c51d7ca64c4e11e3ecb3c84 78.5 MiB linux/amd64 io.cri-containerd.image=managed

I guess something is missing in rootless configuration

Your observation exactly match this issue: https://github.com/canonical/microk8s/issues/3112

Though it provides no other solution than running docker rootful.

Thank you! Got it! It’s weird that, even when docker is running in rootless mode, pushing image is working fine when microk8s is running inside multipass. The only difference is in tagging docker image using the IP showed by multipass list and not localhost.
Since I’m developing on a linux machine and since I’m still in dev mode I find stupid to have the overhead of a VM, even though I have enough RAM. Multipass will be useful in a next stage when testing CLOUD INIT
Thank you again

I was wondering why this could happen, as localhost should be available in both mode, but I realized that rootless Docker is using rootlesskit, which isolates itself from the host. So even when you run something like:

docker run --rm -it --net host nicolaka/netshoot curl localhost:32000

it will not be able to access the API. If the docker daemon is in such an isolated environment, it will not be able to access the registry either.

If I just try

rootlesskit curl localhost:32000

that works, but not from a container.

I installed podman just for a test. Podman is rootless by default and it works. It’s been just for test since I don’t like it. Compose doesn’t work like docker and I find many intermediate images when running native build that I don’t see in docker

Podman works differently. It doesn’t have a daemon as far as I know. The Docker daemon is running in the namespace created by rootlesskit. I just tested it. I knew about it, but didn’t remember.

dockerd_pid=$(rootlesskit pidof dockerd | tr ' ' '\n' | sort | head -n1)
sudo nsenter --all -t "$dockerd_pid" ip a

And the output is something like this:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc fq_codel state UP group default qlen 1000
    link/ether 6a:e4:56:0b:31:7c brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.100/24 scope global tap0
       valid_lft forever preferred_lft forever
    inet6 fe80::68e4:56ff:fe0b:317c/64 scope link
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:ec:ab:40:49 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:ecff:feab:4049/64 scope link
       valid_lft forever preferred_lft forever

I have much more interfaces on the host. So indeed, the docker daemon has no access to the localhost of the real host.

EDIT:

My example might not work on your machine and the rootlesskit command was not really needed as the process IDs are the same with or without rootlesskit.

1 Like