Docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc did not terminate successfully: exit status 134: unknown

Hi there,
I am having problems launching multiple Docker containers. I have moved from a RaspberryPi to an external Linux server with Ubuntu 22.04. I wanted to move all my containers there bit by bit, but when I try to start more than five containers, I get an error message - regardless of the image.

Additionally, I should say that I am not a Docker expert and I may be missing something very simple.

I would really appreciate an answer as it would be extremely important for me to be able to continue running all my containers.

The problem:

I currently have portainer, mariadb, nextcloud, traefik and mealie running. If I now try to start a fifth container, I get the following output:

➜  ~ docker run hello-world
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc did not terminate successfully: exit status 134: unknown.
ERRO[0001] error waiting for container:   

Moreover, there are following syslog entries:

May 19 09:10:42 XXXXXXXX systemd[1]: var-lib-docker-overlay2-3eac1ad979bfb73a27fd83de3c586356cd7f15b5c022a7d4d7adf1be20c91c1c\x2dinit-merged.mount: Deactivated successfully.
May 19 09:10:42 XXXXXXXX systemd[1]: var-lib-docker-overlay2-3eac1ad979bfb73a27fd83de3c586356cd7f15b5c022a7d4d7adf1be20c91c1c-merged.mount: Deactivated successfully.
May 19 09:10:43 XXXXXXXX systemd-udevd[170114]: Using default interface naming scheme 'v249'.
May 19 09:10:43 XXXXXXXX systemd-udevd[170115]: Using default interface naming scheme 'v249'.
May 19 09:10:43 XXXXXXXX systemd-networkd[166]: veth8c05cd3: Link UP
May 19 09:10:43 XXXXXXXX containerd[436]: time="2023-05-19T09:10:43.173087547+02:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
May 19 09:10:43 XXXXXXXX containerd[436]: time="2023-05-19T09:10:43.173164512+02:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
May 19 09:10:43 XXXXXXXX containerd[436]: time="2023-05-19T09:10:43.173807484+02:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
May 19 09:10:43 XXXXXXXX containerd[436]: time="2023-05-19T09:10:43.174155149+02:00" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/bc5ac8c2f298f5e0d9ca28ca145f278c5f22ea6bd882d7cdfdaaba60f3c53305 pid=170136 runtime=io.containerd.runc.v2
May 19 09:10:43 XXXXXXXX containerd[436]: time="2023-05-19T09:10:43.412682608+02:00" level=info msg="shim disconnected" id=bc5ac8c2f298f5e0d9ca28ca145f278c5f22ea6bd882d7cdfdaaba60f3c53305
May 19 09:10:43 XXXXXXXX containerd[436]: time="2023-05-19T09:10:43.412758341+02:00" level=warning msg="cleaning up after shim disconnected" id=bc5ac8c2f298f5e0d9ca28ca145f278c5f22ea6bd882d7cdfdaaba60f3c53305 namespace=moby
May 19 09:10:43 XXXXXXXX containerd[436]: time="2023-05-19T09:10:43.412768610+02:00" level=info msg="cleaning up dead shim"
May 19 09:10:43 XXXXXXXX containerd[436]: time="2023-05-19T09:10:43.443980350+02:00" level=warning msg="cleanup warnings time=\"2023-05-19T09:10:43+02:00\" level=info msg=\"starting signal loop\" namespace=moby pid=170194 runtime=io.containerd.runc.v2\ntime=\"2023-05-19T09:10:43+02:00\" level=warning msg=\"failed to read init pid file\" error=\"open /run/containerd/io.containerd.runtime.v2.task/moby/bc5ac8c2f298f5e0d9ca28ca145f278c5f22ea6bd882d7cdfdaaba60f3c53305/init.pid: no such file or directory\" runtime=io.containerd.runc.v2\n"
May 19 09:10:43 XXXXXXXX containerd[436]: time="2023-05-19T09:10:43.444327344+02:00" level=error msg="copy shim log" error="read /proc/self/fd/32: file already closed"
May 19 09:10:43 XXXXXXXX dockerd[166495]: time="2023-05-19T09:10:43.449325605+02:00" level=error msg="stream copy error: reading from a closed fifo"
May 19 09:10:43 XXXXXXXX dockerd[166495]: time="2023-05-19T09:10:43.449348238+02:00" level=error msg="stream copy error: reading from a closed fifo"
May 19 09:10:43 XXXXXXXX systemd-networkd[166]: veth8c05cd3: Link DOWN
May 19 09:10:44 XXXXXXXX systemd[1]: var-lib-docker-overlay2-3eac1ad979bfb73a27fd83de3c586356cd7f15b5c022a7d4d7adf1be20c91c1c-merged.mount: Deactivated successfully.
May 19 09:10:44 XXXXXXXX dockerd[166495]: time="2023-05-19T09:10:44.383491478+02:00" level=error msg="Handler for POST /v1.43/containers/bc5ac8c2f298f5e0d9ca28ca145f278c5f22ea6bd882d7cdfdaaba60f3c53305/start returned error: failed to create task for container: failed to create shim task: OCI runtime create failed: runc did not terminate successfully: exit status 2: unknown"

What I have tried so far:

  • Restart docker (systemctl restart docker)
  • Restart the system (reboot now)
  • Reinstalled the OS

Moreover, I have found this post with a similar issue. The first difference to my problem is, that I can start exactly five containers before the problem appears and I didn’t used an preinstalled docker version, but I have used the convenience script.

Further system information:

➜  ~ uname -a
Linux XXXXXXXX.stratoserver.net 5.2.0 #1 SMP Thu Dec 15 20:31:06 MSK 2022 x86_64 x86_64 x86_64 GNU/Linux
➜  ~ docker version        
Client: Docker Engine - Community
 Version:           24.0.0
 API version:       1.43
 Go version:        go1.20.4
 Git commit:        98fdcd7
 Built:             Mon May 15 18:49:22 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.0
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.4
  Git commit:       1331b8c
  Built:            Mon May 15 18:49:22 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
➜  ~ whoami
root

If you need some more information, please tell me.

The problem still exists today. I have now reinstalled the OS and Docker several times and tested the setup with different containers. After all that time I assume that there is actually a problem with the resources, because I can deploy between 5 and 6 containers at a time, depending on the images. One more run command and I get the aforementioned error.

My problem is that I have no idea which resource might be the limiting one. Using htop I can see that all CPUs are idling at about 0% to 1%. When I request a website from the docker, one of the four CPUs goes up to 25%, the others are still around 1%. Also, the memory is constantly between 900 and 1000 MB, but there are 8 GB available.
This is the output of df -h:

Filesystem         Size  Used Avail Use% Mounted on
/dev/ploop31714p1  296G   41G  243G  15% /
none               4,0G     0  4,0G   0% /sys/fs/cgroup
none               4,0G     0  4,0G   0% /dev
tmpfs              4,0G     0  4,0G   0% /dev/shm
tmpfs              820M  1,5M  818M   1% /run
tmpfs              5,0M     0  5,0M   0% /run/lock
none               4,0G     0  4,0G   0% /run/shm
overlay            296G   41G  243G  15% /var/lib/docker/overlay2/0307ef0bcaf3617fa4c1de644388823a599bfd7505e6870d5c8334c35670ec07/merged
overlay            296G   41G  243G  15% /var/lib/docker/overlay2/10bbc517f1dd607b645817092a488c6419896da84b3a32e76b268a1601d329f1/merged
overlay            296G   41G  243G  15% /var/lib/docker/overlay2/38968a85fff1d00863e9821583aac834b27eb126276c8daeb840b40df3682a56/merged
overlay            296G   41G  243G  15% /var/lib/docker/overlay2/70cb3a9b59b99ea949f880be8e28e8c9216205e52e0f07016459fcd6d07c3d8d/merged
overlay            296G   41G  243G  15% /var/lib/docker/overlay2/16bd74ea98125181f15a79ade07f883fd7f27d0daecb49f6e96cee13bdfba1a7/merged
tmpfs              820M     0  820M   0% /run/user/0

Does anyone have an idea what the limiting factor could be?