Docker stuck on Pulling fs layer after downloading is started

General
1

Issue

docker pull problem

Platform

Linux 6.12.69+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.69-1 (2026-02-08) x86_64 GNU/Linux

Application


Client: Docker Engine - Community
 Version:    29.2.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.31.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.0.2
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
Server:
 Containers: 3
  Running: 1
  Paused: 0
  Stopped: 2
 Images: 15
 Server Version: 29.2.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: dea7da592f5d1d2b7755e3a161be07f43fad8f75
 runc version: v1.3.4-0-gd6d73eb8
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.12.69+deb13-amd64
 Operating System: Debian GNU/Linux 13 (trixie)
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 14.92GiB
 Name: gyronavdebian
 ID: 77de6ba7-1c5b-4ff8-a82b-a592e661b44f
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 31
  Goroutines: 49
  System Time: 2026-02-15T07:58:21.626989491+03:00
  EventsListeners: 0
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Firewall Backend: iptables


Client: Docker Engine - Community
 Version:           29.2.1
 API version:       1.53
 Go version:        go1.25.6
 Git commit:        a5c7197
 Built:             Mon Feb  2 17:17:31 2026
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          29.2.1
  API version:      1.53 (minimum version 1.44)
  Go version:       go1.25.6
  Git commit:       6bc6209
  Built:            Mon Feb  2 17:17:31 2026
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.2.1
  GitCommit:        dea7da592f5d1d2b7755e3a161be07f43fad8f75
 runc:
  Version:          1.3.4
  GitCommit:        v1.3.4-0-gd6d73eb8
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Installed packeges

ii  docker                    1.5-2                       all          transitional package
ii  docker-buildx-plugin      0.31.1-1~debian.13~trixie   amd64        Docker Buildx plugin extends build capabilities with BuildKit.
ii  docker-ce                 5:29.2.1-1~debian.13~trixie amd64        Docker: the open-source application container engine
ii  docker-ce-cli             5:29.2.1-1~debian.13~trixie amd64        Docker CLI: the open-source application container engine
ii  docker-ce-rootless-extras 5:29.2.1-1~debian.13~trixie amd64        Rootless support for Docker.
ii  docker-compose-plugin     5.0.2-1~debian.13~trixie    amd64        Docker Compose (V2) plugin for the Docker CLI.

Installation guide: Install using the apt repository

Description

I see a strange behaviour on my server machine. Docker hangs on fs layer pulling. The most relevant request for that problem says that I have some network missconsistency. But the strange thing is that docker stucks after donloading actually started…

log after timeout:

sudo docker pull busybox
Using default tag: latest
latest: Pulling from library/busybox
61dfb50712f5: Downloading [==================================================>]  2.156MB/2.156MB
read tcp 192.168.0.200:34826->172.64.66.1:443: read: connection timed out

I didn’t believe in that log at first. I thought it some kind of a bug. After some tries to fix my network I lunched tcpdump. But tcpdump show that downloading actually is going on:

last transfers

14:02:46.716819 IP 192.168.0.200.60682 > 172.64.66.1.443: Flags [.], ack 6415, win 619, options [nop,nop,TS val 3958289255 ecr 647921782], length 0
14:02:46.716907 IP 172.64.66.1.443 > 192.168.0.200.60682: Flags [.], seq 7806:9246, ack 2461, win 17, options [nop,nop,TS val 647921782 ecr 3958289065], length 1440
14:02:46.716927 IP 172.64.66.1.443 > 192.168.0.200.60682: Flags [P.], seq 9246:10588, ack 2461, win 17, options [nop,nop,TS val 647921782 ecr 3958289065], length 1342
14:02:46.716934 IP 192.168.0.200.60682 > 172.64.66.1.443: Flags [.], ack 9246, win 654, options [nop,nop,TS val 3958289255 ecr 647921782], length 0
14:02:46.717040 IP 172.64.66.1.443 > 192.168.0.200.60682: Flags [P.], seq 10588:11979, ack 2461, win 17, options [nop,nop,TS val 647921782 ecr 3958289065], length 1391
14:02:46.717155 IP 172.64.66.1.443 > 192.168.0.200.60682: Flags [P.], seq 11979:13370, ack 2461, win 17, options [nop,nop,TS val 647921782 ecr 3958289065], length 1391
14:02:46.717241 IP 192.168.0.200.60682 > 172.64.66.1.443: Flags [.], ack 13370, win 637, options [nop,nop,TS val 3958289255 ecr 647921782], length 0
14:02:46.717323 IP 172.64.66.1.443 > 192.168.0.200.60682: Flags [.], seq 13370:14810, ack 2461, win 17, options [nop,nop,TS val 647921782 ecr 3958289065], length 1440
14:02:46.717378 IP 172.64.66.1.443 > 192.168.0.200.60682: Flags [P.], seq 14810:16152, ack 2461, win 17, options [nop,nop,TS val 647921782 ecr 3958289065], length 1342
14:02:46.717412 IP 192.168.0.200.60682 > 172.64.66.1.443: Flags [.], ack 16152, win 666, options [nop,nop,TS val 3958289255 ecr 647921782], length 0
14:02:46.717546 IP 172.64.66.1.443 > 192.168.0.200.60682: Flags [.], seq 16152:17592, ack 2461, win 17, options [nop,nop,TS val 647921782 ecr 3958289065], length 1440
14:02:46.717646 IP 172.64.66.1.443 > 192.168.0.200.60682: Flags [P.], seq 17592:18934, ack 2461, win 17, options [nop,nop,TS val 647921782 ecr 3958289065], length 1342
14:02:46.717717 IP 172.64.66.1.443 > 192.168.0.200.60694: Flags [P.], seq 4584:5480, ack 2461, win 16, options [nop,nop,TS val 3819606592 ecr 3958289069], length 896
14:02:46.717717 IP 172.64.66.1.443 > 192.168.0.200.60694: Flags [P.], seq 5480:5504, ack 2461, win 16, options [nop,nop,TS val 3819606592 ecr 3958289069], length 24
14:02:46.717717 IP 172.64.66.1.443 > 192.168.0.200.60694: Flags [F.], seq 5504, ack 2461, win 16, options [nop,nop,TS val 3819606592 ecr 3958289069], length 0
14:02:46.717730 IP 192.168.0.200.60682 > 172.64.66.1.443: Flags [.], ack 18934, win 666, options [nop,nop,TS val 3958289255 ecr 647921782], length 0
14:02:46.717755 IP 192.168.0.200.60694 > 172.64.66.1.443: Flags [.], ack 5505, win 597, options [nop,nop,TS val 3958289255 ecr 3819606592], length 0
14:02:46.717878 IP 192.168.0.200.60694 > 172.64.66.1.443: Flags [P.], seq 2461:2485, ack 5505, win 597, options [nop,nop,TS val 3958289256 ecr 3819606592], length 24
14:02:46.717921 IP 192.168.0.200.60694 > 172.64.66.1.443: Flags [F.], seq 2485, ack 5505, win 597, options [nop,nop,TS val 3958289256 ecr 3819606592], length 0
14:02:46.761751 IP 172.64.66.1.443 > 192.168.0.200.60694: Flags [.], ack 2486, win 16, options [nop,nop,TS val 3819606637 ecr 3958289256], length 0
14:02:47.638684 IP 192.168.0.200.33608 > 95.100.107.169.443: Flags [P.], seq 3610:3649, ack 926, win 501, options [nop,nop,TS val 3884298490 ecr 44646352], length 39
14:02:47.689204 IP 95.100.107.169.443 > 192.168.0.200.33608: Flags [.], ack 3649, win 567, options [nop,nop,TS val 44704674 ecr 3884298490], length 0
14:02:47.694769 IP 95.100.107.169.443 > 192.168.0.200.33608: Flags [P.], seq 926:965, ack 3649, win 567, options [nop,nop,TS val 44704679 ecr 3884298490], length 39
14:02:47.694823 IP 192.168.0.200.33608 > 95.100.107.169.443: Flags [.], ack 965, win 501, options [nop,nop,TS val 3884298547 ecr 44704679], length 0
14:02:50.494459 IP 95.100.107.169.443 > 192.168.0.200.33608: Flags [P.], seq 965:989, ack 3649, win 567, options [nop,nop,TS val 44707479 ecr 3884298547], length 24
14:02:50.494460 IP 95.100.107.169.443 > 192.168.0.200.33608: Flags [F.], seq 989, ack 3649, win 567, options [nop,nop,TS val 44707479 ecr 3884298547], length 0

Docker debug log:

Feb 15 07:46:32 dockerd[2575]: time="2026-02-15T07:46:32.132184742+03:00" level=debug msg="handling HEAD request" method=HEAD module=api request-url=/_ping vars="map[]"
Feb 15 07:46:32 dockerd[2575]: time="2026-02-15T07:46:32.132493585+03:00" level=debug msg="handling POST request" method=POST module=api request-url="/v1.53/images/create?fromImage=docker.io%2Flibrary%2Fbusybox&tag=latest" vars="map[version:1.53]"
Feb 15 07:46:32 dockerd[2575]: time="2026-02-15T07:46:32.134751325+03:00" level=debug msg="Trying to pull busybox from https://registry-1.docker.io"
Feb 15 07:46:32 dockerd[2575]: time="2026-02-15T07:46:32.134769559+03:00" level=debug msg="Trying to pull busybox from https://registry-1.docker.io"
Feb 15 07:46:33 dockerd[2575]: time="2026-02-15T07:46:33.549220753+03:00" level=debug msg="Pulling ref from V2 registry: busybox:latest" digest="sha256:b3255e7dfbcd10cb367af0d409747d511aeb66dfac98cf30e97e87e4207dd76f" remote="docker.io/library/busybox:latest"
Feb 15 07:46:33 dockerd[2575]: time="2026-02-15T07:46:33.549247453+03:00" level=debug msg="docker.io/library/busybox:latest resolved to a manifestList object with 17 entries; looking for a unknown match" digest="sha256:b3255e7dfbcd10cb367af0d409747d511aeb66dfac98cf30e97e87e4207dd76f" remote="docker.io/library/busybox:latest"
Feb 15 07:46:33 dockerd[2575]: time="2026-02-15T07:46:33.549260687+03:00" level=debug msg="found match for linux/amd64/v3 with media type application/vnd.oci.image.manifest.v1+json, digest sha256:70ce0a747f09cd7c09c2d6eaeab69d60adb0398f569296e8c0e844599388ebd6"
Feb 15 07:46:33 dockerd[2575]: time="2026-02-15T07:46:33.549267580+03:00" level=debug msg="found match for linux/amd64/v3 with media type application/vnd.oci.image.manifest.v1+json, digest sha256:4630ec42856ce4bb9e3c3c28974b2e5352e5ddb83f30f6dff593af6bd61a04fc"
Feb 15 07:46:33 dockerd[2575]: time="2026-02-15T07:46:33.554673203+03:00" level=debug msg="pulling blob \"sha256:61dfb50712f5ff92c880813210257a42169ff0937896ae95dab763582cc380e2\""
Feb 15 07:46:41 dockerd[2575]: time="2026-02-15T07:46:41.640164152+03:00" level=info msg="Not continuing with pull after error" error="context canceled"

I can ping that IP (172.64.66.1.443).

I can pull from another network from same IP (172.64.66.1.443).So the problem obviously is in my network. But where exactly?

Restarting and rebooting host and router does not help. Setting other dns ip does not help (it is resolved fine)
How can I debug that issue?

2

We usually need the following information to understand the issue:

  1. What platform are you using? Windows, Linux or macOS? Which version of the operating systems? In case of Linux, which distribution?

  2. How did you install Docker? Sharing the platform almost answers it, but only almost. Direct links to the followed guide can be useful.

  3. On debian based Linux, the following commands can give us some idea and recognize incorrectly installed Docker:

    docker info
docker version

    Review the output before sharing and remove confidential data if any appears (public IP for example)

    dpkg -l 'docker*' | grep '^ii'
snap list docker

    When you share the outputs, always format your posts according to the following guide: How to format your forum posts

3

Thank you. I edited my post

4

uhm, this isn’t a valid ip. I assume you took this from the tcpdump output, where the last element looks like the port.

Your logs show that the engine is successfully able to communicate with the api using registry-1.docker.io (which is hosted on aws), but then fails to download a blob. The cli command shows that the blob fails to be loaded from the ip you see (which belongs to cloudflare).

Looks like you already enabled debugging in your docker engine:
https://docs.docker.com/engine/daemon/logs/#enable-debugging

Can you check if sudo journalctl -xu containerd.service has entries between
Feb 15 07:46:33 and Feb 15 07:46:41 that contain the missing pieces?

5

containerd.service log

Feb 15 18:39:11 systemd[1]: containerd.service: Found left-over process 12952 (containerd-shim) in control group while starting unit. Ignoring.
Feb 15 18:39:11 systemd[1]: containerd.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 15 18:39:11 systemd[1]: Starting containerd.service - containerd container runtime...
░░ Subject: A start job for unit containerd.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit containerd.service has begun execution.
░░ 
░░ The job identifier is 8793.
Feb 15 18:39:11 systemd[1]: containerd.service: Found left-over process 12952 (containerd-shim) in control group while starting unit. Ignoring.
Feb 15 18:39:11 systemd[1]: containerd.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.422651092+03:00" level=info msg="starting containerd" revision=dea7da592f5d1d2b7755e3a161be07f43fad8f75 version=v2.2.1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.427690968+03:00" level=info msg="loading plugin" id=io.containerd.content.v1.content type=io.containerd.content.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.427848891+03:00" level=info msg="loading plugin" id=io.containerd.image-verifier.v1.bindir type=io.containerd.image-verifier.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.427860442+03:00" level=info msg="loading plugin" id=io.containerd.internal.v1.opt type=io.containerd.internal.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.427911026+03:00" level=info msg="loading plugin" id=io.containerd.warning.v1.deprecations type=io.containerd.warning.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.427928459+03:00" level=info msg="loading plugin" id=io.containerd.mount-handler.v1.erofs type=io.containerd.mount-handler.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.427936494+03:00" level=info msg="loading plugin" id=io.containerd.snapshotter.v1.blockfile type=io.containerd.snapshotter.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.427952644+03:00" level=info msg="skip loading plugin" error="no scratch file generator: skip plugin" id=io.containerd.snapshotter.v1.blockfile type=io.containerd.snapshotter.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.427961159+03:00" level=info msg="loading plugin" id=io.containerd.snapshotter.v1.btrfs type=io.containerd.snapshotter.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.428150260+03:00" level=info msg="skip loading plugin" error="path /var/lib/containerd/io.containerd.snapshotter.v1.btrfs (ext4) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" id=io.containerd.snapshotter.v1.btrfs type=io.containerd.snapshotter.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.428176569+03:00" level=info msg="loading plugin" id=io.containerd.snapshotter.v1.devmapper type=io.containerd.snapshotter.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.428191968+03:00" level=info msg="skip loading plugin" error="devmapper not configured: skip plugin" id=io.containerd.snapshotter.v1.devmapper type=io.containerd.snapshotter.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.428204020+03:00" level=info msg="loading plugin" id=io.containerd.snapshotter.v1.erofs type=io.containerd.snapshotter.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.429063185+03:00" level=info msg="skip loading plugin" error="EROFS unsupported, please `modprobe erofs`: skip plugin" id=io.containerd.snapshotter.v1.erofs type=io.containerd.snapshotter.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.429272093+03:00" level=info msg="loading plugin" id=io.containerd.snapshotter.v1.native type=io.containerd.snapshotter.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.429333798+03:00" level=info msg="loading plugin" id=io.containerd.snapshotter.v1.overlayfs type=io.containerd.snapshotter.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.429437841+03:00" level=info msg="loading plugin" id=io.containerd.snapshotter.v1.zfs type=io.containerd.snapshotter.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.429462076+03:00" level=info msg="skip loading plugin" error="lstat /var/lib/containerd/io.containerd.snapshotter.v1.zfs: no such file or directory: skip plugin" id=io.containerd.snapshotter.v1.zfs type=io.containerd.snapshotter.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.429468057+03:00" level=info msg="loading plugin" id=io.containerd.event.v1.exchange type=io.containerd.event.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.429486691+03:00" level=info msg="loading plugin" id=io.containerd.monitor.task.v1.cgroups type=io.containerd.monitor.task.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.429616853+03:00" level=info msg="loading plugin" id=io.containerd.metadata.v1.bolt type=io.containerd.metadata.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.429631771+03:00" level=info msg="metadata content store policy set" policy=shared
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.429838525+03:00" level=info msg="loading plugin" id=io.containerd.gc.v1.scheduler type=io.containerd.gc.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.429857660+03:00" level=info msg="loading plugin" id=io.containerd.nri.v1.nri type=io.containerd.nri.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.429874331+03:00" level=info msg="built-in NRI default validator is disabled"
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.429878569+03:00" level=info msg="runtime interface created"
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.429881805+03:00" level=info msg="created NRI interface"
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.429887205+03:00" level=info msg="loading plugin" id=io.containerd.differ.v1.erofs type=io.containerd.differ.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.429908725+03:00" level=info msg="skip loading plugin" error="failed to check mkfs.erofs availability: failed to run mkfs.erofs --help: exec: \"mkfs.erofs\": executable file not found in $PATH: skip plugin" id=io.containerd.differ.v1.erofs type=io.containerd.differ.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.429914846+03:00" level=info msg="loading plugin" id=io.containerd.differ.v1.walking type=io.containerd.differ.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.429921499+03:00" level=info msg="loading plugin" id=io.containerd.lease.v1.manager type=io.containerd.lease.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.429927820+03:00" level=info msg="loading plugin" id=io.containerd.mount-manager.v1.bolt type=io.containerd.mount-manager.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.429979807+03:00" level=info msg="loading plugin" id=io.containerd.service.v1.containers-service type=io.containerd.service.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.430001527+03:00" level=info msg="loading plugin" id=io.containerd.service.v1.content-service type=io.containerd.service.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.430009041+03:00" level=info msg="loading plugin" id=io.containerd.service.v1.diff-service type=io.containerd.service.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.430015443+03:00" level=info msg="loading plugin" id=io.containerd.service.v1.images-service type=io.containerd.service.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.430022216+03:00" level=info msg="loading plugin" id=io.containerd.service.v1.introspection-service type=io.containerd.service.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.430033096+03:00" level=info msg="loading plugin" id=io.containerd.service.v1.namespaces-service type=io.containerd.service.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.430039267+03:00" level=info msg="loading plugin" id=io.containerd.service.v1.snapshots-service type=io.containerd.service.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.430044697+03:00" level=info msg="loading plugin" id=io.containerd.shim.v1.manager type=io.containerd.shim.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.430051540+03:00" level=info msg="loading plugin" id=io.containerd.runtime.v2.task type=io.containerd.runtime.v2
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.430263253+03:00" level=info msg="connecting to shim 8cddfe2cb9acc20a2d7d497059145fb3fcfd413740e12d35613226d03a565863" address="unix:///run/containerd/s/fd0e25066a7f33e3841f82ff62a05647fbf12694906afe02d62d764fcfee1ed7" namespace=moby protocol=ttrpc version=3
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435195179+03:00" level=info msg="loading plugin" id=io.containerd.service.v1.tasks-service type=io.containerd.service.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435233059+03:00" level=info msg="loading plugin" id=io.containerd.grpc.v1.containers type=io.containerd.grpc.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435254298+03:00" level=info msg="loading plugin" id=io.containerd.grpc.v1.content type=io.containerd.grpc.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435266772+03:00" level=info msg="loading plugin" id=io.containerd.grpc.v1.diff type=io.containerd.grpc.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435278994+03:00" level=info msg="loading plugin" id=io.containerd.grpc.v1.events type=io.containerd.grpc.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435290155+03:00" level=info msg="loading plugin" id=io.containerd.grpc.v1.images type=io.containerd.grpc.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435308158+03:00" level=info msg="loading plugin" id=io.containerd.grpc.v1.introspection type=io.containerd.grpc.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435318798+03:00" level=info msg="loading plugin" id=io.containerd.grpc.v1.leases type=io.containerd.grpc.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435332093+03:00" level=info msg="loading plugin" id=io.containerd.grpc.v1.mounts type=io.containerd.grpc.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435343444+03:00" level=info msg="loading plugin" id=io.containerd.grpc.v1.namespaces type=io.containerd.grpc.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435355677+03:00" level=info msg="loading plugin" id=io.containerd.sandbox.store.v1.local type=io.containerd.sandbox.store.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435366617+03:00" level=info msg="loading plugin" id=io.containerd.transfer.v1.local type=io.containerd.transfer.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435391703+03:00" level=info msg="loading plugin" id=io.containerd.cri.v1.images type=io.containerd.cri.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435431557+03:00" level=info msg="Get image filesystem path \"/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs\" for snapshotter \"overlayfs\""
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435443229+03:00" level=info msg="Start snapshots syncer"
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435463376+03:00" level=info msg="loading plugin" id=io.containerd.cri.v1.runtime type=io.containerd.cri.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435637099+03:00" level=info msg="starting cri plugin" config="{\"containerd\":{\"defaultRuntimeName\":\"runc\",\"runtimes\":{\"runc\":{\"runtimeType\":\"io.containerd.runc.v2\",\"runtimePath\":\"\",\"PodAnnotations\":null,\"ContainerAnnotations\":null,\"options\":{\"BinaryName\":\"\",\"CriuImagePath\":\"\",\"CriuWorkPath\":\"\",\"IoGid\":0,\"IoUid\":0,\"NoNewKeyring\":false,\"Root\":\"\",\"ShimCgroup\":\"\",\"SystemdCgroup\":false},\"privileged_without_host_devices\":false,\"privileged_without_host_devices_all_devices_allowed\":false,\"cgroupWritable\":false,\"baseRuntimeSpec\":\"\",\"cniConfDir\":\"\",\"cniMaxConfNum\":0,\"snapshotter\":\"\",\"sandboxer\":\"podsandbox\",\"io_type\":\"\"}},\"ignoreBlockIONotEnabledErrors\":false,\"ignoreRdtNotEnabledErrors\":false},\"cni\":{\"binDir\":\"\",\"binDirs\":[\"/opt/cni/bin\"],\"confDir\":\"/etc/cni/net.d\",\"maxConfNum\":1,\"setupSerially\":false,\"confTemplate\":\"\",\"ipPref\":\"\",\"useInternalLoopback\":false},\"enableSelinux\":false,\"selinuxCategoryRange\":1024,\"maxContainerLogLineSize\":16384,\"disableApparmor\":false,\"restrictOOMScoreAdj\":false,\"disableProcMount\":false,\"unsetSeccompProfile\":\"\",\"tolerateMissingHugetlbController\":true,\"disableHugetlbController\":true,\"device_ownership_from_security_context\":false,\"ignoreImageDefinedVolumes\":false,\"netnsMountsUnderStateDir\":false,\"enableUnprivilegedPorts\":true,\"enableUnprivilegedICMP\":true,\"enableCDI\":true,\"cdiSpecDirs\":[\"/etc/cdi\",\"/var/run/cdi\"],\"drainExecSyncIOTimeout\":\"0s\",\"ignoreDeprecationWarnings\":null,\"containerdRootDir\":\"/var/lib/containerd\",\"containerdEndpoint\":\"/run/containerd/containerd.sock\",\"rootDir\":\"/var/lib/containerd/io.containerd.grpc.v1.cri\",\"stateDir\":\"/run/containerd/io.containerd.grpc.v1.cri\"}"
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435712469+03:00" level=info msg="loading plugin" id=io.containerd.podsandbox.controller.v1.podsandbox type=io.containerd.podsandbox.controller.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435759396+03:00" level=info msg="loading plugin" id=io.containerd.sandbox.controller.v1.shim type=io.containerd.sandbox.controller.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435812515+03:00" level=info msg="loading plugin" id=io.containerd.grpc.v1.sandbox-controllers type=io.containerd.grpc.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435830949+03:00" level=info msg="loading plugin" id=io.containerd.grpc.v1.sandboxes type=io.containerd.grpc.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435842300+03:00" level=info msg="loading plugin" id=io.containerd.grpc.v1.snapshots type=io.containerd.grpc.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435852759+03:00" level=info msg="loading plugin" id=io.containerd.streaming.v1.manager type=io.containerd.streaming.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435864391+03:00" level=info msg="loading plugin" id=io.containerd.grpc.v1.streaming type=io.containerd.grpc.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435876463+03:00" level=info msg="loading plugin" id=io.containerd.grpc.v1.tasks type=io.containerd.grpc.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435887764+03:00" level=info msg="loading plugin" id=io.containerd.grpc.v1.transfer type=io.containerd.grpc.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435899556+03:00" level=info msg="loading plugin" id=io.containerd.grpc.v1.version type=io.containerd.grpc.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435911629+03:00" level=info msg="loading plugin" id=io.containerd.monitor.container.v1.restart type=io.containerd.monitor.container.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435945832+03:00" level=info msg="loading plugin" id=io.containerd.tracing.processor.v1.otlp type=io.containerd.tracing.processor.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435959467+03:00" level=info msg="skip loading plugin" error="skip plugin: tracing endpoint not configured" id=io.containerd.tracing.processor.v1.otlp type=io.containerd.tracing.processor.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435969175+03:00" level=info msg="loading plugin" id=io.containerd.internal.v1.tracing type=io.containerd.internal.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435979234+03:00" level=info msg="skip loading plugin" error="skip plugin: tracing endpoint not configured" id=io.containerd.internal.v1.tracing type=io.containerd.internal.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435988231+03:00" level=info msg="loading plugin" id=io.containerd.ttrpc.v1.otelttrpc type=io.containerd.ttrpc.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.435998500+03:00" level=info msg="loading plugin" id=io.containerd.grpc.v1.healthcheck type=io.containerd.grpc.v1
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.436208009+03:00" level=info msg=serving... address=/run/containerd/containerd.sock.ttrpc
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.436265395+03:00" level=info msg=serving... address=/run/containerd/containerd.sock
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.436325767+03:00" level=info msg="containerd successfully booted in 0.014016s"
Feb 15 18:39:11 systemd[1]: Started containerd.service - containerd container runtime.
░░ Subject: A start job for unit containerd.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit containerd.service has finished successfully.
░░ 
░░ The job identifier is 8793.
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.525552618+03:00" level=info msg="shim disconnected" id=8cddfe2cb9acc20a2d7d497059145fb3fcfd413740e12d35613226d03a565863 namespace=moby
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.525590398+03:00" level=info msg="cleaning up after shim disconnected" id=8cddfe2cb9acc20a2d7d497059145fb3fcfd413740e12d35613226d03a565863 namespace=moby
Feb 15 18:39:11 containerd[20039]: time="2026-02-15T18:39:11.525599264+03:00" level=info msg="cleaning up dead shim" id=8cddfe2cb9acc20a2d7d497059145fb3fcfd413740e12d35613226d03a565863 namespace=moby
Feb 15 18:39:12 containerd[20039]: time="2026-02-15T18:39:12.890510280+03:00" level=info msg="connecting to shim 8cddfe2cb9acc20a2d7d497059145fb3fcfd413740e12d35613226d03a565863" address="unix:///run/containerd/s/fd0e25066a7f33e3841f82ff62a05647fbf12694906afe02d62d764fcfee1ed7" namespace=moby protocol=ttrpc version=3
6

The Log from tcpdump of “working” network contain the same IP but it have much more transactions (I will not spam it here).

7

I specifically asked for logs that correlate with the last two lines of your docker logs. I hope to see the reason it fails in the containerd logs.

So please try to pull again and send the containerd logs that occur in the time between those to entries:

Feb 15 07:46:33 dockerd[2575]: time="2026-02-15T07:46:33.554673203+03:00" level=debug msg="pulling blob \"sha256:61dfb50712f5ff92c880813210257a42169ff0937896ae95dab763582cc380e2\""
Feb 15 07:46:41 dockerd[2575]: time="2026-02-15T07:46:41.640164152+03:00" level=info msg="Not continuing with pull after error" error="context canceled"

Of course within the time frame of your new attempt.

8

Nothing happen in cotainerd logs in that period of time. I cheked many times. Restarted serviceis and reinitiated pulling several times.
In “working” environment I indeed see some events during image pulling.
What does it mean?

9

On fresh Docker v29 installations containerd is used for images and the container file system.
I am not sure if docker fetches the layers, or already delegates it to containerd. I was hoping to see hints about why the problem occurs.

A Google search showed that people in Spain are affected, due to ISP blocks of cloudflare ips. Though, from what I understand it works from other machines in the same network, so it can’t be caused by your ISP.

Without any further information, I have no idea what could cause such a problem. It doesn’t seem to be dns (which is a usual suspect). Other suspects could be tls inspection, or a http proxy for outgoing http(s) connections.

10

I found it.
It was nfqws-keenetic module on router (keenetic hopper se).
All our routers with such filtering by default, but probably this one is buggy…
I think it just was breaking tcp tunnel with it’s fake packets.
Well, nfqws now is r.i.p and we keep on living with that knowladge.))

Thank you very much for your help and support.