Troubleshoot while running 2 docker daemon on the some VM

[rubzlebg (RuBeeN-cmd)]

Hi !

My Plan:
I got a VM with regular docker working properly. A Debian user named test wants to create some containers and manage them. Since I can’t allow him to interact with the others containers I wanted to create a 2nd docker daemon and bind it to the test-docker group. With this setup, the user test in the group test-docker could manage his containers without sudo and without any permissions to the others containers. Maybe ?

My problem is when I try to run the hello-world docker image for example, docker failed to create the container start task, because it cant find /hello file in the container. Maybe I got something misconfigured…

I create config file for my 2nd docker daemon:
/etc/docker/test/daemon.json :

{
        "userns-remap": "test",
        "group": "test-docker",
        "hosts": ["unix:///run/docker-test.sock"],
        "pidfile": "/var/run/docker-test.pid",
        "data-root": "/var/lib/docker-test",
        "containerd": "/run/containerd/containerd.sock",
        "storage-driver": "overlay2",
        "debug": true
}

And created the systemctl service:
/etc/systemd/system/docker-test.service :

[Unit]
Description=Docker Application Container Engine (Test)
Documentation=https://docs.docker.com
Wants=network-online.target
After=network-online.target nss-lookup.target firewalld.service time-set.target docker-test.socket containerd.service

[Service]
Type=notify
ExecStart=/usr/bin/dockerd --config-file=/etc/docker/test/daemon.json
MountFlags=slave
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
TimeoutStartSec=0

[Install]
WantedBy=multi-user.target

/etc/systemd/system/docker-test.socket :

[Unit]
Description=Docker Socket for the API (Test)
PartOf=docker-test.service

[Socket]
ListenStream=/run/docker-test.sock
SocketMode=660
SocketUser=root
SocketGroup=test-docker

[Install]
WantedBy=sockets.target

$ ls -l /run/docker-test.sock
srw-rw---- 1 root test-docker 0  4 juin  11:13 /run/docker-test.sock

/etc/subuid :

user1:100000:65536
test:165536:65536
test:100000:65536

/etc/subgid :

user1:100000:65536
test:165536:65536
test:100000:65536

At this point, docker run hello-world returns :

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: exec: "/hello": stat /hello: no such file or directory: unknown

Run 'docker run --help' for more information

Here’s dockerd logs:

level=debug msg="handling HEAD request" method=HEAD module=api request-url=/_ping vars="map[]"
level=debug msg="handling POST request" form-data="{\"AttachStderr\":true,\"AttachStdin\":false,\"AttachStdout\":true,\"Cmd\":null,\"Domainname\":\"\",\"Entrypoint\":null,\"Env\":null,\"HostConfig\":{\"AutoRemove\":false,\"Binds\":null,\"BlkioDeviceReadBps\":[],\"BlkioDeviceReadIOps\":[],\"BlkioDeviceWriteBps\":[],\"BlkioDeviceWriteIOps\":[],\"BlkioWeight\":0,\"BlkioWeightDevice\":[],\"CapAdd\":null,\"CapDrop\":null,\"Cgroup\":\"\",\"CgroupParent\":\"\",\"CgroupnsMode\":\"\",\"ConsoleSize\":[47,185],\"ContainerIDFile\":\"\",\"CpuCount\":0,\"CpuPercent\":0,\"CpuPeriod\":0,\"CpuQuota\":0,\"CpuRealtimePeriod\":0,\"CpuRealtimeRuntime\":0,\"CpuShares\":0,\"CpusetCpus\":\"\",\"CpusetMems\":\"\",\"DeviceCgroupRules\":null,\"DeviceRequests\":null,\"Devices\":[],\"Dns\":[],\"DnsOptions\":[],\"DnsSearch\":[],\"ExtraHosts\":null,\"GroupAdd\":null,\"IOMaximumBandwidth\":0,\"IOMaximumIOps\":0,\"IpcMode\":\"\",\"Isolation\":\"\",\"Links\":null,\"LogConfig\":{\"Config\":{},\"Type\":\"\"},\"MaskedPaths\":null,\"Memory\":0,\"MemoryReservation\":0,\"MemorySwap\":0,\"MemorySwappiness\":-1,\"NanoCpus\":0,\"NetworkMode\":\"default\",\"OomKillDisable\":false,\"OomScoreAdj\":0,\"PidMode\":\"\",\"PidsLimit\":0,\"PortBindings\":{},\"Privileged\":false,\"PublishAllPorts\":false,\"ReadonlyPaths\":null,\"ReadonlyRootfs\":false,\"RestartPolicy\":{\"MaximumRetryCount\":0,\"Name\":\"no\"},\"SecurityOpt\":null,\"ShmSize\":0,\"UTSMode\":\"\",\"Ulimits\":[],\"UsernsMode\":\"\",\"VolumeDriver\":\"\",\"VolumesFrom\":null},\"Hostname\":\"\",\"Image\":\"hello-world\",\"Labels\":{},\"NetworkingConfig\":{\"EndpointsConfig\":{\"default\":{\"Aliases\":null,\"DNSNames\":null,\"DriverOpts\":null,\"EndpointID\":\"\",\"Gateway\":\"\",\"GlobalIPv6Address\":\"\",\"GlobalIPv6PrefixLen\":0,\"GwPriority\":0,\"IPAMConfig\":null,\"IPAddress\":\"\",\"IPPrefixLen\":0,\"IPv6Gateway\":\"\",\"Links\":null,\"MacAddress\":\"\",\"NetworkID\":\"\"}}},\"OnBuild\":null,\"OpenStdin\":false,\"StdinOnce\":false,\"Tty\":false,\"User\":\"\",\"Volumes\":{},\"WorkingDir\":\"\"}" method=POST module=api request-url=/v1.49/containers/create vars="map[version:1.49]"
level=debug msg="container mounted via layerStore" container=e33cad233667e3f93a88d8749daf7f3936e1b16f7e9b704d7d5ed3e72747a6d6 root=/var/lib/docker-test/165536.165536/overlay2/0314a33b708fa609e74c0f2f9851144632a313e710004b43b60d74b990c15a31/merged storage-driver=overlay2
level=debug msg="handling POST request" method=POST module=api request-url="/v1.49/containers/e33cad233667e3f93a88d8749daf7f3936e1b16f7e9b704d7d5ed3e72747a6d6/attach?stderr=1&stdout=1&stream=1" vars="map[name:e33cad233667e3f93a88d8749daf7f3936e1b16f7e9b704d7d5ed3e72747a6d6 version:1.49]"
level=debug msg="attach: stdout: begin"
level=debug msg="attach: stderr: begin"
level=debug msg="handling POST request" method=POST module=api request-url="/v1.49/containers/e33cad233667e3f93a88d8749daf7f3936e1b16f7e9b704d7d5ed3e72747a6d6/wait?condition=next-exit" vars="map[name:e33cad233667e3f93a88d8749daf7f3936e1b16f7e9b704d7d5ed3e72747a6d6 version:1.49]"
level=debug msg="handling POST request" method=POST module=api request-url=/v1.49/containers/e33cad233667e3f93a88d8749daf7f3936e1b16f7e9b704d7d5ed3e72747a6d6/start vars="map[name:e33cad233667e3f93a88d8749daf7f3936e1b16f7e9b704d7d5ed3e72747a6d6 version:1.49]"
level=debug msg="container mounted via layerStore" container=e33cad233667e3f93a88d8749daf7f3936e1b16f7e9b704d7d5ed3e72747a6d6 root=/var/lib/docker-test/165536.165536/overlay2/0314a33b708fa609e74c0f2f9851144632a313e710004b43b60d74b990c15a31/merged storage-driver=overlay2
level=debug msg="createSpec: cgroupsPath: system.slice:docker:e33cad233667e3f93a88d8749daf7f3936e1b16f7e9b704d7d5ed3e72747a6d6"
level=debug msg="bundle dir created" bundle=/var/run/docker/containerd/e33cad233667e3f93a88d8749daf7f3936e1b16f7e9b704d7d5ed3e72747a6d6 module=libcontainerd namespace=moby-165536.165536 root=/var/lib/docker-test/165536.165536/overlay2/0314a33b708fa609e74c0f2f9851144632a313e710004b43b60d74b990c15a31/merged
level=error msg="copy stream failed" error="reading from a closed fifo" stream=stdout
level=error msg="copy stream failed" error="reading from a closed fifo" stream=stderr
level=debug msg="attach: stdout: end"
level=debug msg="attach: stderr: end"
level=debug msg="attach done"
level=debug msg="Failed to remove 0314a33b708fa609e74c0f2f9851144632a313e710004b43b60d74b990c15a31 overlay: directory not empty" storage-driver=overlay2
level=debug msg="error response for POST request" error-response="failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: exec: \"/hello\": stat /hello: no such file or directory: unknown" method=POST module=api request-url=/v1.49/containers/e33cad233667e3f93a88d8749daf7f3936e1b16f7e9b704d7d5ed3e72747a6d6/start status=400 vars="map[name:e33cad233667e3f93a88d8749daf7f3936e1b16f7e9b704d7d5ed3e72747a6d6 version:1.49]"

and containerd logs:

level=debug msg="shim bootstrap parameters" address="unix:///run/containerd/s/608bc2357fe83b7de0aacb10c239ef0208090e0c9eb06a216f95ddb91695aed7" namespace=moby-165536.165536 protocol=ttrpc
level=debug msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
level=debug msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
level=debug msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
level=debug msg="registering ttrpc service" id=io.containerd.ttrpc.v1.task
level=debug msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
level=debug msg="registering ttrpc service" id=io.containerd.ttrpc.v1.pause
level=debug msg="serving api on socket" socket="[inherited from parent]"
level=debug msg="starting signal loop" namespace=moby-165536.165536 path=/run/containerd/io.containerd.runtime.v2.task/moby-165536.165536/e33cad233667e3f93a88d8749daf7f3936e1b16f7e9b704d7d5ed3e72747a6d6 pid=3750618 runtime=io.containerd.runc.v2
level=debug msg="failed to delete task" error="rpc error: code = NotFound desc = container not created: not found" id=e33cad233667e3f93a88d8749daf7f3936e1b16f7e9b704d7d5ed3e72747a6d6
level=info msg="shim disconnected" id=e33cad233667e3f93a88d8749daf7f3936e1b16f7e9b704d7d5ed3e72747a6d6 namespace=moby-165536.165536
level=warning msg="cleaning up after shim disconnected" id=e33cad233667e3f93a88d8749daf7f3936e1b16f7e9b704d7d5ed3e72747a6d6 namespace=moby-165536.165536
level=info msg="cleaning up dead shim" namespace=moby-165536.165536
level=warning msg="cleanup warnings time=\"2025-06-04T11:51:10+02:00\" level=debug msg=\"starting signal loop\" namespace=moby-165536.165536 pid=3750649 runtime=io.containerd.runc.v2\ntime=\"2025-06-04T11:51:10+02:00\" level=warning msg=\"failed to read init pid file\" error=\"open /run/containerd/io.containerd.runtime.v2.task/moby-165536.165536/e33cad233667e3f93a88d8749daf7f3936e1b16f7e9b704d7d5ed3e72747a6d6/init.pid: no such file or directory\" runtime=io.containerd.runc.v2\n" namespace=moby-165536.165536
level=error msg="copy shim log" error="read /proc/self/fd/147: file already closed" namespace=moby-165536.165536

I can see the namespace created in /var/lib/docker-test/165536.165536 or from ctr command :

$ sudo ls -la /var/lib/docker-test/
total 12
drwx--x---  3 root 165536 4096  4 juin  11:12 .
drwxr-xr-x 31 root root   4096  3 juin  09:16 ..
drwx--x--- 12 root 165536 4096  4 juin  11:13 165536.165536

$ sudo ctr ns ls
NAME               LABELS
moby
moby-165536.165536
plugins.moby

I think I’m stuck, I really need help to debug it ! PLZZZ !

Thx for your investisment !

[rimelek (Ákos Takács)]

I’m not sure what misconfiguration happened. It may be in the logs that I could not recognize, but I did’t spend much time on it, to be honest.
It looks like the binary in the container is not found. I don’t know why, but I would like to point out that if your goal is

@rubzlebg

With this setup, the user test in the group test-docker could manage his containers without sudo and without any permissions to the others containers. Maybe ?

As long as the docker daemon is running as root, the user with access to the Docker socket pratcically has root access to the host. The user namespace remapping is to avoid processes in the container accessing something on the host, but you can disable the remapping by using --userns=host and nothing stops you from running a privileged container and using nsenter to run a shell on the host as root. So if you just want make it harder for a user to accidentally do something wrong, userns remapping might work, but if you really want to make sure the user has no access to the host and the other docker daemon, you either need to filter requests sent to the API or run the docker daemon as a non-root user. This is what “rootless mode” is.

Rootless mode

It has limitations though.