Error response from daemon: failed to create shim: OCI runtime create failed: runc create failed: unable to start container process: error during container init

Hi, good afternoon!

I’m getting a lot of problems trying to get into my docker containers and I don’t know where to look for some info to solve my problem. I wasted a lot of time trying to look for a solution in Google but no luck.

First of all, I have to say that I’m not an expert using docker so, maybe this question is quite stupid, I’m so sorry if that’s the case.

The problem is, I always get this error when I try to get into any docker container:

gbernaldo@gbernaldo-Vostro-3500:~$ sudo docker start a4ba5a6a6ab4
Error response from daemon: failed to create shim: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: unable to find device '99/136': scan /proc/devices line "33": EOF: unknown
Error: failed to start containers: a4ba5a6a6ab4

The container exists!

gbernaldo@gbernaldo-Vostro-3500:~$ docker image ls
REPOSITORY                            TAG       IMAGE ID       CREATED         SIZE
rhel8                                 sercom    a4ba5a6a6ab4   4 months ago    2.79GB
rhel8                                 latest    ba3aef99485e   5 months ago    2.79GB
registry.access.redhat.com/ubi8       latest    b81e86a2cb9a   6 months ago    216MB
sercom8                               clon      453161040b89   7 months ago    8.63GB
centos8                               sercom    c34d41afdfd3   8 months ago    4.25GB
centos8                               devel     19361102d9f7   8 months ago    1.16GB
centos                                8         5d0da3dc9764   10 months ago   231MB
registry.access.redhat.com/ubi8/ubi   8.3       613e5da7a934   15 months ago   205MB

But I cannot get into and I don’t know exactly why.

I’ve installed Ubuntu:

gbernaldo@gbernaldo-Vostro-3500:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04 LTS
Release:        22.04
Codename:       jammy

I’ve been using docker without any problem for two years. I’m not using anykind of virtual machine. I’m running the docker on the machine directly.

Those kind of errors began to appear today this morning when I restarted the machine. Yesterday, I had to do two things that maybe are the cause of the error:

  1. I did a full upgrade of Ubuntu (from Ubuntu 21.10 to Ubuntu 22.04 LTS).
  2. I had to bind my /dev to the /dev container because I needed to share a RS232 device plugged to the computer.
    Those are the only things I had to do.

Please, can anyone help my?
I need to recover all containers, I just can’t lose them!

Thanks in advance!

NOTE:

The version of docker I have is the next:

gbernaldo@gbernaldo-Vostro-3500:~$ docker version
Client:
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.17.3
 Git commit:        20.10.12-0ubuntu4
 Built:             Mon Mar  7 17:10:06 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.3
  Git commit:       20.10.12-0ubuntu4
  Built:            Mon Mar  7 15:57:50 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.9-0ubuntu3
  GitCommit:        
 runc:
  Version:          1.1.0-0ubuntu1
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:

Please, use </> button to share codes, terminal outputs, error messages or anything that can contain special characters which would be interpreted by the MarkDown filter. Use the preview feature to make sure your text is formatted as you would expect it and check your post after you have sent it so you can still fix it. I edited your post.

This is not getting into the container but starting an existing container which fails. Can you show how you created that container?

Then it is probably because you mounted a device file which only exists until you restart the container. This is why it is important to see how you created that container originally. If you don’t remember, please share the output of:

docker container inspect a4ba5a6a6ab4 --format '{{ json .Mounts }}'

If you have “jq” installed on your system, then please, run this command instead:

docker container inspect a4ba5a6a6ab4 --format '{{ json .Mounts }}' | jq

and use the </> button when you share it of course :slight_smile:

However, you should not create containers that you can’t remove safely and create it again. If you mounted a device which is not there now, you need to remove and recreate it the container. I think there is an other way too if it is absolutely necessary, but I don’t remember it exactly and it requires editing files in /var/lib/docker which is not really a good idea unless you are experimenting in a test environment.

When you fix the container, then you can use docker exec to get into the container.

Thank you so much for your answer, very appreciated!

I tried the command you told me:

gbernaldo@gbernaldo-Vostro-3500:~$ sudo docker container inspect a4ba5a6a6ab4 --format '{{ json .Mounts }}' | jq
Error: No such container: a4ba5a6a6ab4

But, it seems it didn’t work.

I haven’t created the container using different mount points. I mean, I haven’t changed any mount point from my linux since I created the docker container two years ago.

Again, thank you in advance for your answer, and so sorry about the </>. It was my first question and I didn’t know I could do that.

Hi, good afternoon!

Finally, I’ve deleted everything (after making a backup of /var/lib/docker), I reinstalled docker and tried with a Hello World image. Those are the steps I followed:

root@gbernaldo-Vostro-3500:/home/gbernaldo# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
2db29710123e: Pull complete 
Digest: sha256:53f1bbee2f52c39e41682ee1d388285290c5c8a76cc92b42687eecf38e0af3f0
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: failed to create shim: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: unable to find device '99/136': scan /proc/devices line "33": EOF: unknown.
ERRO[0003] error waiting for container: context canceled

It seems that it doesn’t works anyway and I don’t understand why.

Next I put you the output of the docker ps -a:

root@gbernaldo-Vostro-3500:/home/gbernaldo# docker ps -a
CONTAINER ID   IMAGE                                    COMMAND       CREATED         STATUS    PORTS     NAMES
e6c41cab8531   hello-world                              "/hello"      4 minutes ago   Created             laughing_jackson

And the output of the inspect command:

[
    {
        "Id": "e6c41cab85310ebed94b3559a14ede94b7fee1a0029c03f4b87a07e5c62006ca",
        "Created": "2022-07-28T12:28:06.47514606Z",
        "Path": "/hello",
        "Args": [],
        "State": {
            "Status": "created",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 128,
            "Error": "failed to create shim: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: unable to find device '99/136': scan /proc/devices line \"33\": EOF: unknown",
            "StartedAt": "0001-01-01T00:00:00Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412",
        "ResolvConfPath": "/var/lib/docker/containers/e6c41cab85310ebed94b3559a14ede94b7fee1a0029c03f4b87a07e5c62006ca/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/e6c41cab85310ebed94b3559a14ede94b7fee1a0029c03f4b87a07e5c62006ca/hostname",
        "HostsPath": "/var/lib/docker/containers/e6c41cab85310ebed94b3559a14ede94b7fee1a0029c03f4b87a07e5c62006ca/hosts",
        "LogPath": "/var/lib/docker/containers/e6c41cab85310ebed94b3559a14ede94b7fee1a0029c03f4b87a07e5c62006ca/e6c41cab85310ebed94b3559a14ede94b7fee1a0029c03f4b87a07e5c62006ca-json.log",
        "Name": "/laughing_jackson",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "private",
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": null,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/ff7ed52a1e507eea8ef32a97f97d7a121dcfab2113eae6e6458cfc962f8cffa7-init/diff:/var/lib/docker/overlay2/d1ff5145545c10da0d3e02ff94b8115912cf98032e122328be7373a2639dfdb2/diff",
                "MergedDir": "/var/lib/docker/overlay2/ff7ed52a1e507eea8ef32a97f97d7a121dcfab2113eae6e6458cfc962f8cffa7/merged",
                "UpperDir": "/var/lib/docker/overlay2/ff7ed52a1e507eea8ef32a97f97d7a121dcfab2113eae6e6458cfc962f8cffa7/diff",
                "WorkDir": "/var/lib/docker/overlay2/ff7ed52a1e507eea8ef32a97f97d7a121dcfab2113eae6e6458cfc962f8cffa7/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [],
        "Config": {
            "Hostname": "e6c41cab8531",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "/hello"
            ],
            "Image": "hello-world",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {}
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "7f650f546b252326b339c02dd821099b2a1dfa7fca4a3479f2f661def52a730e",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/7f650f546b25",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "802362490c30a3422add1de09717f04d331821bade1857446c1d5b25241cacc5",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
    }
]

Please, can anyone help me?
Thank you so much in advance!

Please make sure you installed docker-ce according this instructions: Install Docker Engine on Ubuntu | Docker Documentation

The output you shared from docker info looks like you are not using the docker packages from docker’s repos.

Usualy if things don’t make sense, first thing we do is check if a second installation is done using the snap package. You can check it by running this command sudo snap list docker. if you get this output “error: no matching snaps installed”, then the snap package is not installed.

which docker info? Do you mean docker inspect or docker version? I didn’t notice before, but you are probably right. This does not look like the official APT repository from Docker but an APT repository from Ubuntu and that can be from snap too.

At least it is similar to what I see when I list APT packages.

Sorry, I gave you a wrong command because I used the image ID not the container ID or container name. You need to use one of those next time.

Good catch! I ment to write docker version :smiley:

Thank you so much @rimelek and @meyay!

Finally, I have solved my problem following the instructions: Install Docker Engine on Ubuntu | Docker Documentation.

Now I can run Docker containers.

It seems the problem was the Ubuntu version I had installed althought I don’t know exactly what difference there are between both of them, if any, but, after having removed the version which comes with Ubuntu and installed the version following the link above, I can run and go into the containers without any problem.

Thank you so much!
Really, really appreciated!

You can be sure that docker from docker’s repository provides vanila docker - it is the official distribution all the documentation on the docker pages are made for.

While docker distributions from any other repository may or may not have modifications to allign the package with their own philosophy or constraints. With most distributions you might see no difference at all, or differences that require different handling that what’s found in Docker’s documentation. After all the documentation aims for the official distribution and can not anticipate modifications others have done in their distribution.