"docker container top" fails with cryptic error regarding runc termination

I have Docker set up, including rootless mode, on a CentOS 8 Digital Ocean droplet. I am very new to this subject matter and am following along with Docker in a Month of Lunches and am now at the point where the reader is instructed to type in:

 docker container run --interactive --tty diamol/base

Then the reader is instructed to open up another terminal, which I did, and type in:

docker container top [container id]

The command spits out the following:

Error response from daemon: runc did not terminate successfully: container_linux.go:187: getting all container pids from cgroups caused: lstat : no such file or directory
: unknown

Because I am a Docker neophyte, my ability to assist anyone reading to diagnose the problem is limited beyond providing what is typically asked for; however, I think it’s worth noting that the same thing happened when I tried the alpine:latest image, meaning it’s not specific to the book image. It also seems worth pointing out that the same thing does not happen when I repeat the same procedure above but with sudo.

Now, for the rest, as stated, I am using CentOS 8 on Digital Ocean. I used the get.docker.com script and followed the instructions for rootless mode (with the minor and most likely unimportant deviation that I tailored the instructions to fish rather than bash; $DOCKER_HOST has still been exported properly).

Specifically, here is the content of /etc/redhat-release:

CentOS Linux release 8.4.2105

Here’s the output of uname -a:

Linux metalicus 4.18.0-305.7.1.el8_4.x86_64 #1 SMP Tue Jun 29 21:55:12 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

And last but not least here’s the output of docker version:

Client: Docker Engine - Community
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        f0df350
 Built:             Wed Jun  2 11:56:24 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:54:48 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.8
  GitCommit:        7eba5930496d9bbe375fdf71603e610ad737d2b2
 runc:
  Version:          1.0.0
  GitCommit:        v1.0.0-0-g84113ee
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

I’d be perfectly happy to attempt to supply whatever other details are needed to resolve the problem.

I had to solve the same problem today. Turns out it’s all written down in the docs. I do not understand the connection, but you basically have to make sure you are using cgroup v2. This is not clearly stated, but it’s mentioned under Limiting resources.

The explanation how to configure your system to use cgroup v2 is given in Runtime metrics / Changing cgroup version:

Essentially: run

sudo grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=1"

or edit /etc/default/grub manually (and run sudo update-grub afterwards).