How does imagePull call [ docker.io/library/image ] get resolved?

Docker Desktop
,
1

I’m running Docker Desktop with built-in / bundled Kuberentes in WSL. Every time a POD pulling a container image, the call to docker . io/library/< image:tag > fails for ImagePullErr . These images are standard ubuntu:latest or alpine: from any public registry lets say.

I workaround this failure by pulling the image locally. Subsequently i redeploy or rollout restart the pod in K8S. POD successfully creates the container.

What is happening behind the scenes? is docker dot io always meant to resolve to my local docker? how do i make the PODs in K8S (running inside the docker containers) to pull the images from public registry.

I have tried the same with KIND cluster running on top of my local docker. same behavior for container pull .

Ideally i wanted the K8S POD to pull automatically from the global public registry wherever the image exists eg. ubuntu:latest or busybox:latest rather than me locally pulling the image in my docker desktop

2

If you are trying official public images , Kubernetes should be able to pull. When you say you tried with “kind”, do you mean you enabled “kind” in Docker Desktop for Kubernetes?

What is your Docker Desktop version and what is the exact error message in addition to “imagePullErr”?

I know that Docker Desktop could have issues with tokens and sometimes logging out is required to update the tokens even for public images, but I don’t remember cases when docker pull worked but pulling images in Kubernetes didn’t..

3

Docker Desktop 4.43.2
Engine 28.3.2
Kubernetes 1.32.2 ( inbuilt kubernetes with KIND nodes instead of kubeadm)

Below is applicable to any image

command >
kubectl run -it --rm tester --image=alpine:latest --restart=Never – sh

POD failure from events log (adjusted spaces in url for readability…) >
Failed to pull image “alpine:latest”: failed to pull and unpack image “docker . io/library/alpine : latest”: failed to read expected number of bytes: unexpected EOF’

Image TAG doesnt matter. same behavior with specifying an explicit tag as well.

My Q : why cant Kubernetes go beyond the local docker and hit the docker public registry (or whichever public registry providing that image) like when we docker run --rm alpine:latest

My workaround: i have to pull the images outside of kubernetes and stage it in my docker (seen in docker desktop as well). Its a pain to keep doing this prep work for kubernetes

4

after further digging i see

  1. the containers spun up as nodes for kubernetes (spun by Docker Desktop) run containerd process
  2. the images pulled are stored in containerd registry on these nodes. verified by ctr and crictl commands.

I was able to see the logs for kubelet and containerd showing ImagePullErr when a pod deployment requesting the image pull .

from the K8S node (container itself spun by Docker desktop) i manually ran “ctr --namespace k8s .io image pull docker .io/library/alpine:latest” (extra spaces for readability here) … and it did pull the image from public docker hub registry

so im not sure why K8S Pod requesting the same container would fail.

5

Just use code blocks please, as described in our formatting guide and as you would on GitHub or any markdow-based editor. Then your commands and error messages can be copied as well easily to reproduce issues or search for errors: How to format your forum posts

Yes, this is how Kind works (Kubernetes in Docker)

I’m not sure either. Maybe the problem is accessing the containerd api if anything was corrupted in your Docker Desktop.

If you want to understand the networking of Docker Desktop, you can read this blog post which is old, but it should be something similar today.

What I can say is what I said before. Kubernetes should have no problem with pulling the image. I tried to reproduce the issue on my Mac. I trid the kubeadm-based and kind-based Kubernetes and both worked.

In general, proxy settings or VPN affected the network in Docker Desktop before, but I would expect it to fail with ctr too..

What is your host operating system and version? Cany ou show the docker info output? You can remove any private IP or username that you wouldn’t share before posting. And please, use code blocks.

6

I’m on Windows WSL2. May be its containerd but those are baked / brought in by KIND cluster spun by Docker Desktop. Individually tested the containerd image pull with

containerd image pull :

ctr --namespace k8s.io image pull docker.io/library/alpine:latest 
docker.io/library/alpine:latest:                                                  resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:eafc1edb577d2e9b458664a15f23ea1c370214193226069eb22921169fc7e43f: done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:9824c27679d3b27c5e1cb00a73adb6f4f8d556994111c12db3c5d61a0c843df8:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:9234e8fb04c47cfe0f49931e4ac7eb76fa904e33b7f8576aec0501c085f02516:   done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 1.7 s                                                                    total:  3.0 Mi (1.8 MiB/s)                                       
unpacking linux/amd64 sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1...
done: 104.507499ms
8 
docker info
Client:
 Version:    28.3.2
 Context:    default
 Debug Mode: false
 Plugins:
  ai: Docker AI Agent - Ask Gordon (Docker Inc.)
    Version:  v1.6.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-ai
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.25.0-desktop.1
    Path:     /usr/local/lib/docker/cli-plugins/docker-buildx
  cloud: Docker Cloud (Docker Inc.)
    Version:  v0.4.2
    Path:     /usr/local/lib/docker/cli-plugins/docker-cloud
  compose: Docker Compose (Docker Inc.)
    Version:  v2.38.2-desktop.1
    Path:     /usr/local/lib/docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.41
    Path:     /usr/local/lib/docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Docker Inc.)
    Version:  v0.1.11
    Path:     /usr/local/lib/docker/cli-plugins/docker-desktop
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.29
    Path:     /usr/local/lib/docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-init
  mcp: Docker MCP Plugin (Docker Inc.)
    Version:  v0.9.9
    Path:     /usr/local/lib/docker/cli-plugins/docker-mcp
  model: Docker Model Runner (EXPERIMENTAL) (Docker Inc.)
    Version:  v0.1.33
    Path:     /usr/local/lib/docker/cli-plugins/docker-model
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.18.1
    Path:     /usr/local/lib/docker/cli-plugins/docker-scout
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-dev" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-dev: no such file or directory
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-feedback" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-feedback: no such file or directory

Server:
 Containers: 29
  Running: 10
  Paused: 0
  Stopped: 19
 Images: 42
 Server Version: 28.3.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Discovered Devices:
  cdi: docker.com/gpu=webgpu
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 nvidia runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
 runc version: v1.2.5-0-g59923ef
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.6.87.2-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 22
 Total Memory: 15.34GiB
 Name: docker-desktop
 ID: d2842524-a9ec-47f6-875a-68fad64b9e4e
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///var/run/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
9

Sorry for the slow response. Your outputs look correct. I don’t know why it happened to you and not to me. And indeed, if ctr worked, I don’t see why Kubernetes could not pull images unless it is not a network issue but something between Kubernetes and containerd.

Maybe you could share the issue on GitHub if you haven’t done it yet

So someone at Docker with deeper knowledge about the Docker Desktop internals can try to reproduce it.

10

ok, will file an issue.

But in meantime, i uncovered / understood few other things .

  1. in Each of the Containerd running on the nodes, there’s a config.toml file. I noticed an entry with registry.mirror:1273

    1. after KIND create cluster, there’s a container kind-registry-mirror ( image: docker/desktop-containerd-registry-mirror:v0.0.2) . this container resolves to following DNS

      ``
      "DNSNames": [ "kind-registry-mirror", "registry-mirror", "231d8e967984" ]

So its evident that docker desktop or KIND is doing a registry mirror. That explains why docker image pull ( outside the kubernetes cluster) resolves the image pull in a pod.

1 Like
11

I have the same problem to pull alpine image when I use kind cluster version 1.32.5 with 2 nodes (I didn’t try an other version). I don’t have any problen with kubeadm

Docker-Desktop version : v4.44.3