Container not found error

Docker Hub
1

Hi.
for some reason i have containers in running state but when running docker top or docker exec i get the following error:
Error response from daemon: rpc error: code = 2 desc = containerd: container not found

docker version:
sudo docker version
Client:
Version: 17.05.0-ce
API version: 1.29
Go version: go1.7.5
Git commit: 89658be
Built: Thu May 4 22:10:54 2017
OS/Arch: linux/amd64

Server:
Version: 17.05.0-ce
API version: 1.29 (minimum version 1.12)
Go version: go1.7.5
Git commit: 89658be
Built: Thu May 4 22:10:54 2017
OS/Arch: linux/amd64
Experimental: false

docker ps results:
sudo docker ps
[sudo] password for rd:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
39af5722a724 agentvi/vi-agent-appliance:build-dev-2430 “/bin/sh -c /viser…” 28 hours ago Up 10 hours vi-appliance_3
7060cd2cb483 agentvi/vi-agent-appliance:build-dev-2430 “/bin/sh -c /viser…” 28 hours ago Up 10 hours vi-appliance_2
74042199470d agentvi/vi-agent-appliance:build-dev-2430 “/bin/sh -c /viser…” 28 hours ago Up 10 hours vi-appliance_1

docker top results:
rd@ubuntu:~$ sudo docker top vi-appliance_1
Error response from daemon: rpc error: code = 2 desc = containerd: container not found
rd@ubuntu:~$ sudo docker top vi-appliance_2
Error response from daemon: rpc error: code = 2 desc = containerd: container not found
rd@ubuntu:~$ sudo docker top vi-appliance_3
Error response from daemon: rpc error: code = 2 desc = containerd: container not found

docker inspect result

[
    {
        "Id": "74042199470d2312f1c2cdafce009457d3a125e196688951087dc5b2c7e0ee19",
        "Created": "2017-05-14T13:37:54.542940046Z",
        "Path": "/bin/sh",
        "Args": [
            "-c",
            "/viserver/appliance/ViAgentAppliance"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 23915,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2017-05-15T07:44:53.392071061Z",
            "FinishedAt": "2017-05-15T07:44:43.616598638Z"
        },
        "Image": "sha256:db8846af22bf0a9480c0d2141ccbc680ea7151288fd5d3e82a9847f20cd7cd20",
        "ResolvConfPath": "/var/lib/docker/containers/74042199470d2312f1c2cdafce009457d3a125e196688951087dc5b2c7e0ee19/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/74042199470d2312f1c2cdafce009457d3a125e196688951087dc5b2c7e0ee19/hostname",
        "HostsPath": "/var/lib/docker/containers/74042199470d2312f1c2cdafce009457d3a125e196688951087dc5b2c7e0ee19/hosts",
        "LogPath": "/var/lib/docker/containers/74042199470d2312f1c2cdafce009457d3a125e196688951087dc5b2c7e0ee19/74042199470d2312f1c2cdafce009457d3a125e196688951087dc5b2c7e0ee19-json.log",
        "Name": "/vi-appliance_1",
        "RestartCount": 2,
        "Driver": "aufs",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "appliancemanager_appliance-data:/viserver/appliance/localdata:rw"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": null,
            "RestartPolicy": {
                "Name": "unless-stopped",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": [],
            "CapAdd": null,
            "CapDrop": null,
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": -1,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Data": null,
            "Name": "aufs"
        },
        "Mounts": [
            {
                "Type": "volume",
                "Name": "appliancemanager_appliance-data",
                "Source": "/var/lib/docker/volumes/appliancemanager_appliance-data/_data",
                "Destination": "/viserver/appliance/localdata",
                "Driver": "local",
                "Mode": "rw",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "74042199470d",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "Tty": true,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "AGENT_ID=1",
                "AS_URL=https://1.2.3.4",
                "PAIRING_KEY=9700253a-4f42-4d5c-abee-a745585fda57",
                "MAC_ADDRESS=00:0c:29:e7:02:83",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/viserver/appliance",
                "LD_LIBRARY_PATH=:/viserver/appliance"
            ],
            "Cmd": [
                "/bin/sh",
                "-c",
                "/viserver/appliance/ViAgentAppliance"
            ],
            "ArgsEscaped": true,
            "Image": "agentvi/vi-agent-appliance:build-dev-2430",
            "Volumes": null,
            "WorkingDir": "/viserver/appliance",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {}
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "5d93bf14e68b2187008043ca05a0a34565649d389cff7da8cbabacbcd2bf18ec",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/5d93bf14e68b",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "6aaa667f09862508aa9f0f0222112b4a11377b9d70d2667a25d7ca23ac2435c2",
            "Gateway": "172.17.0.1",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "172.17.0.2",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "MacAddress": "02:42:ac:11:00:02",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "22e548b175e43cd712c727803737565813ef0e92d0979ead51129c11c147c1d6",
                    "EndpointID": "6aaa667f09862508aa9f0f0222112b4a11377b9d70d2667a25d7ca23ac2435c2",
                    "Gateway": "172.17.0.1",
                    "IPAddress": "172.17.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:11:00:02"
                }
            }
        }
    }
]
1 Like
2

i have the same issue… You have solution?

3

I found out this is related to the TTY, AttachStdin, AttachStdout, AttachStderr properties.
i had a process (also container) starting some other containers setting these properties to true. (more specifically i am using node package dockerode)
the new containers occasionally crashed unexpectedly due to some fault in their code.
the containers wont terminate completely as long as the creating container is running and holding the streams open.
my solution was to start the child containers with those properties set to false.
now if the child containers crash, their process terminates and docker engine restarts them.

4

Our team also faced the similar problem. We are using the following version of Docker.

Client:
 Version:      17.03.1-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Mon Mar 27 17:14:09 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.1-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Mon Mar 27 17:14:09 2017
 OS/Arch:      linux/amd64
 Experimental: false
5

This issue exist in 17.05.0-ce as well.

docker exec cmd gave in error:
rpc error: code = 2 desc = containerd: container not found

6

got the same error.
after restart the docker service, then start the container again. problem fixed.

7

I am seeing this error on 17.05.0-ce. In my case this happens after the OS oom killer has run and killed one of the processes in the container.

8

Got this error after oom killer too