In `docker:dind`, `docker pull` succeeds but `docker run` fails, or `docker:dind-rootless` cannot be executed on Rootless Docker host

hytdsh · October 12, 2023, 3:40am

Hi there,

I am facing an error trying to run the dind-rootless image on a Rootless Docker host.
I would like to achieve pattern 2 in the following.

Please let me know if there is any information I am missing.

dind or dind-rootless on Rootless Docker host

rootless@rootless:~$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

rootless@rootless:~$ docker version
Client: Docker Engine - Community
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:32:10 2023
 OS/Arch:           linux/amd64
 Context:           rootless

Server: Docker Engine - Community
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:32:10 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.24
  GitCommit:        61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfc
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 rootlesskit:
  Version:          1.1.1
  ApiVersion:       1.1.1
  NetworkDriver:    slirp4netns
  PortDriver:       builtin
  StateDir:         /tmp/rootlesskit1591481074
 slirp4netns:
  Version:          1.2.0
  GitCommit:        656041d45cfca7a4176f6b7eed9e4fe6c11e8383

rootless@rootless:~$ docker info
Client: Docker Engine - Community
 Version:    24.0.6
 Context:    rootless
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 2
  Running: 0
  Paused: 0
  Stopped: 2
 Images: 3
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc version: v1.1.9-0-gccaecfc
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  rootless
  cgroupns
 Kernel Version: 6.1.0-13-amd64
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 1.921GiB
 Name: rootless
 ID: 34a061b9-ae74-41b6-8211-bed2fd9e4f8f
 Docker Root Dir: /home/rootless/.local/share/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No cpuset support
WARNING: No io.weight support
WARNING: No io.weight (per device) support
WARNING: No io.max (rbps) support
WARNING: No io.max (wbps) support
WARNING: No io.max (riops) support
WARNING: No io.max (wiops) support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

rootless@rootless:~$ ls -l /var/run/user/1001/docker.sock
srw-rw---T 1 rootless 166532 0 Oct  9 18:38 /var/run/user/1001/docker.sock

rootless@rootless:~$ docker run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.
...(snip)...
For more examples and ideas, visit:
 https://docs.docker.com/get-started/

Pattern 1: dind on Rootless Docker host

docker pull succeeds but docker run fails.

rootless@rootless:~/dind$ cat compose.yml
version: "3"

services:
  dind:
    image: docker:24.0.6-dind-alpine3.18
    privileged: true

rootless@rootless:~/dind$ docker compose up -d
[+] Running 2/2
 ✔ Network dind_default   Created                                                                                  0.1s
 ✔ Container dind-dind-1  Started                                                                                  0.1s

rootless@rootless:~/dind$ docker compose exec dind sh
/ #

/ # docker pull ubuntu:latest
latest: Pulling from library/ubuntu
37aaf24cf781: Pull complete
Digest: sha256:9b8dec3bf938bc80fbe758d856e96fdfab5f56c39d44b0cff351e847bb1b01ea
Status: Downloaded newer image for ubuntu:latest
docker.io/library/ubuntu:latest

/ # docker run --rm -it ubuntu:latest bash
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't get final child's PID from pipe: EOF: unknown.
ERRO[0000] error waiting for container:

Pattern 2: dind-rootless on Rootless Docker host

dind-rootless cannot be executed.

rootless@rootless:~/dind-rootless$ cat compose.yml
version: "3"

services:
  dind-rootless:
    image: docker:24.0.6-dind-rootless
    privileged: true
    environment:
      - DOCKER_HOST=unix:///var/run/user/1000/docker.sock

rootless@rootless:~/dind-rootless$ docker compose up -d
[+] Running 2/2
 ✔ Network dind-rootless_default            Created                                                                0.2s
 ✔ Container dind-rootless-dind-rootless-1  Started                                                                0.1s

rootless@rootless:~/dind-rootless$ docker compose ps
NAME      IMAGE     COMMAND   SERVICE   CREATED   STATUS    PORTS

rootless@rootless:~/dind-rootless$ docker compose logs dind-rootless
dind-rootless-dind-rootless-1  | Certificate request self-signature ok
dind-rootless-dind-rootless-1  | subject=CN = docker:dind server
dind-rootless-dind-rootless-1  | /certs/server/cert.pem: OK
dind-rootless-dind-rootless-1  | Certificate request self-signature ok
dind-rootless-dind-rootless-1  | subject=CN = docker:dind client
dind-rootless-dind-rootless-1  | /certs/client/cert.pem: OK
dind-rootless-dind-rootless-1  | Device "ip_tables" does not exist.
dind-rootless-dind-rootless-1  | ip_tables              36864  2 iptable_nat,iptable_filter
dind-rootless-dind-rootless-1  | x_tables               61440  9 iptable_nat,iptable_filter,xt_nat,xt_tcpudp,xt_conntrack,xt_MASQUERADE,xt_addrtype,nft_compat,ip_tables
dind-rootless-dind-rootless-1  | modprobe: can't change directory to '/lib/modules': No such file or directory
dind-rootless-dind-rootless-1  | [rootlesskit:parent] error: failed to setup UID/GID map: newuidmap 56 [0 1000 1 1 100000 65536] failed: newuidmap: write to uid_map failed: Operation not permitted
dind-rootless-dind-rootless-1  | : exit status 1

dind or dind-rootless on Rootfull Docker host (as a comparison)

root@rootfull:~# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

root@rootfull:~# docker version
Client: Docker Engine - Community
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:32:10 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:32:10 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.24
  GitCommit:        61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfc
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

root@rootfull:~# docker info
Client: Docker Engine - Community
 Version:    24.0.6
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 3
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc version: v1.1.9-0-gccaecfc
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.0-13-amd64
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 1.921GiB
 Name: rootfull
 ID: b84f2e9b-21f5-48a2-91b6-d5945b4803f4
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

root@rootfull:~# ls -l /var/run/docker.sock
srw-rw---- 1 root docker 0 Oct  9 18:47 /var/run/docker.sock

root@rootfull:~# docker run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.
...(snip)...
For more examples and ideas, visit:
 https://docs.docker.com/get-started/

Pattern 3: dind on Rootfull Docker host

Success.

root@rootfull:~/dind# cat compose.yml
version: "3"

services:
  dind:
    image: docker:24.0.6-dind-alpine3.18
    privileged: true

root@rootfull:~/dind# docker compose up -d
[+] Running 2/2
 ✔ Network dind_default   Created                                                                                  0.2s
 ✔ Container dind-dind-1  Started                                                                                  0.1s

root@rootfull:~/dind# docker compose exec dind sh
/ #

/ # docker pull ubuntu:latest
latest: Pulling from library/ubuntu
37aaf24cf781: Pull complete
Digest: sha256:9b8dec3bf938bc80fbe758d856e96fdfab5f56c39d44b0cff351e847bb1b01ea
Status: Downloaded newer image for ubuntu:latest
docker.io/library/ubuntu:latest

/ # docker run --rm -it ubuntu:latest bash
root@d5856a278403:/#

root@d5856a278403:/# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Pattern 4: dind-rootless on Rootfull Docker host

Success with DOCKER_HOST env.

root@rootfull:~/dind-rootless# cat compose.yml
version: "3"

services:
  dind-rootless:
    image: docker:24.0.6-dind-rootless
    privileged: true
    environment:
      - DOCKER_HOST=unix:///var/run/user/1000/docker.sock

root@rootfull:~/dind-rootless# docker compose up -d
[+] Running 2/2
 ✔ Network dind-rootless_default            Created                                                                0.2s
 ✔ Container dind-rootless-dind-rootless-1  Started                                                                0.1s

root@rootfull:~/dind-rootless# docker compose exec dind-rootless sh
/ $

/ $ docker pull ubuntu:latest
latest: Pulling from library/ubuntu
37aaf24cf781: Pull complete
Digest: sha256:9b8dec3bf938bc80fbe758d856e96fdfab5f56c39d44b0cff351e847bb1b01ea
Status: Downloaded newer image for ubuntu:latest
docker.io/library/ubuntu:latest

/ $ docker run --rm -it ubuntu:latest bash
root@8203c3367b95:/#

root@8203c3367b95:/# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Topic		Replies	Views
CAP_SETUID and CAP_SETGID appear to be missing General docker	0	644	October 23, 2023
Rootless docker: i/o timeout with docker pull General	32	4438	October 23, 2023
"docker container top" fails with cryptic error regarding runc termination Docker Hub	1	2899	August 17, 2022
Doesn't work when host is a virtual machine General docker	2	2910	July 28, 2022
Docker rootless on raspberry pi General raspberrypi	0	2501	February 15, 2022