Docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't get final child's PID from pipe: EOF: unknown

Greetings,

I have no hair left on my head now, since I started pulling one by one in the hope to solve this error, error which tons and tons of posts have it is listed. Yet none of them helped me solve my troubles.

Any help is greatly apprecaited:
Basic Error:

docker run hello-world

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can’t get final child’s PID from pipe: EOF: unknown.
ERRO[0000] error waiting for container:

Follow instructions thoroughly from here to install docker engine using apt: Install Docker Engine on Ubuntu | Docker Docs

docker version

Client: Docker Engine - Community
Version: 24.0.6
API version: 1.43
Go version: go1.20.7
Git commit: ed223bc
Built: Mon Sep 4 12:31:44 2023
OS/Arch: linux/amd64
Context: default

Server: Docker Engine - Community
Engine:
Version: 24.0.6
API version: 1.43 (minimum version 1.12)
Go version: go1.20.7
Git commit: 1a79695
Built: Mon Sep 4 12:31:44 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.22
GitCommit: 8165feabfdfe38c65b599c4993d227328c231fca
runc:
Version: 1.1.8
GitCommit: v1.1.8-0-g82f18fe
docker-init:
Version: 0.19.0
GitCommit: de40ad0

I started dockerd with --debug to capture following output that which says it is not able to create shim layer.
Any idea what else is going on here?

Sep 15 18:30:27 masternode kernel: device vethc8e608c entered promiscuous mode
Sep 15 18:30:27 masternode dockerd[29540]: time=“2023-09-15T18:30:27.434730488-05:00” level=debug msg=“Assigning addresses for endpoint stoic_snyder’s interface on network bridge”
Sep 15 18:30:27 masternode dockerd[29540]: time=“2023-09-15T18:30:27.436274749-05:00” level=debug msg=“Programming external connectivity on endpoint stoic_snyder (26f9344ff03c7e44be21248b9c9a378d905aad528250bd78d7aae9110436a022)”
Sep 15 18:30:27 masternode dockerd[29540]: time=“2023-09-15T18:30:27.437472643-05:00” level=debug msg=“DeleteConntrackEntries purged ipv4:0, ipv6:0”
Sep 15 18:30:27 masternode dockerd[29540]: time=“2023-09-15T18:30:27.437485163-05:00” level=debug msg=“DeleteConntrackEntriesByPort for udp ports purged ipv4:0, ipv6:0”
Sep 15 18:30:27 masternode dockerd[29540]: time=“2023-09-15T18:30:27.437685017-05:00” level=debug msg=“EnableService 34d41093026ed2c4c6fb991a83e91d3defb7ff4acf4698198e11f7423e586043 START”
Sep 15 18:30:27 masternode dockerd[29540]: time=“2023-09-15T18:30:27.437698617-05:00” level=debug msg=“EnableService 34d41093026ed2c4c6fb991a83e91d3defb7ff4acf4698198e11f7423e586043 DONE”
Sep 15 18:30:27 masternode dockerd[29540]: time=“2023-09-15T18:30:27.438685217-05:00” level=debug msg=“createSpec: cgroupsPath: system.slice:docker:34d41093026ed2c4c6fb991a83e91d3defb7ff4acf4698198e11f7423e586043”
Sep 15 18:30:27 masternode dockerd[29540]: time=“2023-09-15T18:30:27.438912432-05:00” level=debug msg=“bundle dir created” bundle=/var/run/docker/containerd/34d41093026ed2c4c6fb991a83e91d3defb7ff4acf4698198e11f7423e586043 module=libcontainerd namespace=moby root=/var/lib/docker/overlay2/764bb5ddf4e0ef4cf1c46de1727d1c2043ba1dd0c8b5c27c139b6b979303ecaf/merged
Sep 15 18:30:27 masternode systemd-udevd[29758]: Using default interface naming scheme ‘v249’.
Sep 15 18:30:27 masternode containerd[28338]: time=“2023-09-15T18:30:27.452700518-05:00” level=info msg=“loading plugin "io.containerd.event.v1.publisher"…” runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Sep 15 18:30:27 masternode containerd[28338]: time=“2023-09-15T18:30:27.452784980-05:00” level=info msg=“loading plugin "io.containerd.internal.v1.shutdown"…” runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Sep 15 18:30:27 masternode containerd[28338]: time=“2023-09-15T18:30:27.452794920-05:00” level=info msg=“loading plugin "io.containerd.ttrpc.v1.task"…” runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Sep 15 18:30:27 masternode containerd[28338]: time=“2023-09-15T18:30:27.452943613-05:00” level=info msg=“starting signal loop” namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/34d41093026ed2c4c6fb991a83e91d3defb7ff4acf4698198e11f7423e586043 pid=29772 runtime=io.containerd.runc.v2
Sep 15 18:30:27 masternode systemd[1]: Started libcontainer container 34d41093026ed2c4c6fb991a83e91d3defb7ff4acf4698198e11f7423e586043.
Sep 15 18:30:27 masternode systemd[1]: docker-34d41093026ed2c4c6fb991a83e91d3defb7ff4acf4698198e11f7423e586043.scope: Deactivated successfully.
Sep 15 18:30:27 masternode systemd[1]: Stopped libcontainer container 34d41093026ed2c4c6fb991a83e91d3defb7ff4acf4698198e11f7423e586043.
Sep 15 18:30:27 masternode containerd[28338]: time=“2023-09-15T18:30:27.563253122-05:00” level=info msg=“shim disconnected” id=34d41093026ed2c4c6fb991a83e91d3defb7ff4acf4698198e11f7423e586043
Sep 15 18:30:27 masternode containerd[28338]: time=“2023-09-15T18:30:27.563338814-05:00” level=warning msg=“cleaning up after shim disconnected” id=34d41093026ed2c4c6fb991a83e91d3defb7ff4acf4698198e11f7423e586043 namespace=moby
Sep 15 18:30:27 masternode containerd[28338]: time=“2023-09-15T18:30:27.563355475-05:00” level=info msg=“cleaning up dead shim”
Sep 15 18:30:27 masternode containerd[28338]: time=“2023-09-15T18:30:27.592124211-05:00” level=warning msg=“cleanup warnings time="2023-09-15T18:30:27-05:00" level=info msg="starting signal loop" namespace=moby pid=30057 runtime=io.containerd.runc.v2\ntime="2023-09-15T18:30:27-05:00" level=warning msg="failed to read init pid file" error="open /run/containerd/io.containerd.runtime.v2.task/moby/34d41093026ed2c4c6fb991a83e91d3defb7ff4acf4698198e11f7423e586043/init.pid: no such file or directory" runtime=io.containerd.runc.v2\n”
Sep 15 18:30:27 masternode containerd[28338]: time=“2023-09-15T18:30:27.592433398-05:00” level=error msg=“copy shim log” error=“read /proc/self/fd/16: file already closed”
Sep 15 18:30:27 masternode dockerd[29540]: time=“2023-09-15T18:30:27.593054950-05:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 15 18:30:27 masternode dockerd[29540]: time=“2023-09-15T18:30:27.593138812-05:00” level=debug msg=“attach: stdout: end”
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can’t get final child’s PID from pipe: EOF: unknown.
Sep 15 18:30:27 masternode dockerd[29540]: time=“2023-09-15T18:30:27.593105411-05:00” level=error msg=“stream copy error: reading from a closed fifo”
Sep 15 18:30:27 masternode dockerd[29540]: time=“2023-09-15T18:30:27.593158152-05:00” level=debug msg=“attach: stderr: end”
Sep 15 18:30:27 masternode dockerd[29540]: time=“2023-09-15T18:30:27.593215513-05:00” level=debug msg=“attach done”
Sep 15 18:30:27 masternode dockerd[29540]: time=“2023-09-15T18:30:27.596020289-05:00” level=debug msg=“Revoking external connectivity on endpoint stoic_snyder (26f9344ff03c7e44be21248b9c9a378d905aad528250bd78d7aae9110436a022)”
ERRO[0000] error waiting for container:
Sep 15 18:30:27 masternode dockerd[29540]: time=“2023-09-15T18:30:27.597890926-05:00” level=debug msg=“DeleteConntrackEntries purged ipv4:0, ipv6:0”
Sep 15 18:30:27 masternode dockerd[29540]: time=“2023-09-15T18:30:27.597908037-05:00” level=debug msg=“DeleteConntrackEntriesByPort for udp ports purged ipv4:0, ipv6:0”
Sep 15 18:30:27 masternode kernel: docker0: port 1(vethc8e608c) entered disabled state
Sep 15 18:30:27 masternode kernel: device vethc8e608c left promiscuous mode
Sep 15 18:30:27 masternode kernel: docker0: port 1(vethc8e608c) entered disabled state
Sep 15 18:30:27 masternode dockerd[29540]: time=“2023-09-15T18:30:27.662902489-05:00” level=debug msg=“Releasing addresses for endpoint stoic_snyder’s interface on network bridge”
Sep 15 18:30:27 masternode dockerd[29540]: time=“2023-09-15T18:30:27.662933470-05:00” level=debug msg=“ReleaseAddress(LocalDefault/172.17.0.0/16, 172.17.0.2)”
Sep 15 18:30:27 masternode dockerd[29540]: time=“2023-09-15T18:30:27.662973360-05:00” level=debug msg=“Released address Address:172.17.0.2 Sequence:Bits: 65536, Unselected: 65533, Sequence: (0xc0000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:3”
Sep 15 18:30:27 masternode systemd[1]: var-lib-docker-overlay2-764bb5ddf4e0ef4cf1c46de1727d1c2043ba1dd0c8b5c27c139b6b979303ecaf-merged.mount: Deactivated successfully.
Sep 15 18:30:27 masternode dockerd[29540]: time=“2023-09-15T18:30:27.666886869-05:00” level=error msg=“Handler for POST /v1.43/containers/34d41093026ed2c4c6fb991a83e91d3defb7ff4acf4698198e11f7423e586043/start returned error: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can’t get final child’s PID from pipe: EOF: unknown”

Best,
-amit

Please format your post according: How to format your forum posts

Furthermore: please change the title to something meaningful that describes the situation, rather than posting long error messages as title.