Problems running Docker daemon as non-root user

General
1

Hello, recently based on the official documentation to configure the rootless mode, I found some strange problems, I need help.
thank you!

System Version：CentOS 7.6

Execute /usr/bin/dockerd-rootless-setuptool.sh install and prompt systemd not detected .

Later found to execute the systemctl --user show-environment command, prompt Failed to get D-Bus connection: No such file or directory.

So give up the “With systemd” method, refer to the official website “Without systemd” method.

The environment variables have been configured, and executing “dockerd-rootless.sh” under a non-root user prompts an error of “mount: permission denied [rootlesskit:child ] error: command [/usr/bin/dockerd-rootless.sh] exited: exit status 32”

After research, it was found that “mount: only root can use “–rbind” option” , and the dockerd-rootless.sh script can only be executed by non-root users, one requires root, and the other does not need root. How to solve it?

Looking forward to your reply, thank you!

2

First of all, CentOS 7.6 has systemd, so you should be able to use the systemd approach unless you actually use a CentOS WSL2 distribution. When I last tried docker-rootless.sh, it worked. I don’t have time to test it again, but if you don’t have a standard Linux distribution without WSL2 or similar environments, you should first try to run a Virtual machine to test the installation on a “real” CentOS 7.6

As for the rbind option, --rbind is required in a very specific case:

if [ "$(stat -c %T -f /etc)" = "tmpfs" ] && [ -L "/etc/ssl" ]; then
    # Workaround for "x509: certificate signed by unknown authority" on openSUSE Tumbleweed.
    # https://github.com/rootless-containers/rootlesskit/issues/225
    realpath_etc_ssl=$(realpath /etc/ssl)
    rm -f /etc/ssl
    mount --rbind ${realpath_etc_ssl} /etc/ssl
fi

So (if I read this code right) when /etc is on tmpfs, and /etc/ssl is a link to an other path. This is something that even WSL2 would not do in a CentOS distribution, but apparently some NAS does it: /etc on tmpfs | TrueNAS Community

3

Gǎnxiè nín de huífù

volume_up

6 / 5,000

翻译结果

Thank you for your reply!

I adopted the official version 7.6 mini of CentOS, and manually upgraded the kernel to 4.20.3.

CentOS 7.6 comes with systemd. For the first time, I tried to use the systemd method recommended by the official document. When I executed the command “systemctl --user start docker”, the error “Failed to get D-Bus connection: No such file or directory” was reported.

Looking back, it is found that “/usr/bin/dockerd-rootless-setuptool.sh install” is executed, and it prompts “[INFO] systemd not detected, dockerd-rootless.sh needs to be started manually”.

Checking the “dockerd-rootless-setuptool”.sh script found a setting for SYSTEMD

if systemctl --user show-environment > /dev/null 2>&1; then
        SYSTEMD=1
fi

When I manually execute “systemctl --user show-environment” under the general user, it prompts the error message “Failed to get D-Bus connection: No such file or directory”

At this time, I considered deploying through the “Without systemd” method, but it is not very smooth.

4

I try “Without systemd” method here

  1. Add environment variables
export HOME=/home/$(id -un)
export XDG_RUNTIME_DIR=/run/user/$(id -u)
export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/docker.sock
  1. Execute
bash -x /usr/bin/dockerd-rootless.sh

This time I found a strange phenomenon,
Executing “stat -c %T -f /etc” in the script returns the value “tmpfs”, while I manually execute it in the terminal and the return value “xfs”

+ net=slirp4netns
+ '[' -z '' ']'
+ mtu=65520
+ '[' -z slirp4netns ']'
+ '[' -z 65520 ']'
+ '[' -z 1 ']'
+ '[' 1 = 1 ']'
+ rm -f /run/docker /run/containerd /run/xtables.lock
+ '[' -n 1 ']'
+ chcon system_u:object_r:iptables_var_run_t:s0 /run
++ stat -c %T -f /etc
+ '[' tmpfs = tmpfs ']'
+ '[' -L /etc/ssl ']'
++ realpath /etc/ssl
+ realpath_etc_ssl=/etc/.ro635429426/ssl
+ rm -f /etc/ssl
+ mkdir /etc/ssl
+ mount --rbind /etc/.ro635429426/ssl /etc/ssl
mount: permission denied
[rootlesskit:child ] error: command [/usr/bin/dockerd-rootless.sh] exited: exit status 32
[rootlesskit:parent] error: child exited: exit status 32
[kry@localhost ~]$ 
[kry@localhost ~]$ echo $(stat -c %T -f /etc)
xfs
5

Test shield “dockerd-rootless.sh” script code block ：
vi /usr/bin/dockerd-rootless.sh

#       if [ "$(stat -c %T -f /etc)" = "tmpfs" ] && [ -L "/etc/ssl" ]; then
#               # Workaround for "x509: certificate signed by unknown authority" on openSUSE Tumbleweed.
#               # https://github.com/rootless-containers/rootlesskit/issues/225
#               realpath_etc_ssl=$(realpath /etc/ssl)
#               rm -f /etc/ssl
#               mkdir /etc/ssl
#               mount --rbind ${realpath_etc_ssl} /etc/ssl
#       fi

Running the script here outputs the following：

+ rm -f /run/docker /run/containerd /run/xtables.lock
+ '[' -n 1 ']'
+ chcon system_u:object_r:iptables_var_run_t:s0 /run
+ exec dockerd
INFO[2022-04-14T16:36:35.371628656+08:00] Starting up                                  
WARN[2022-04-14T16:36:35.371674147+08:00] Running in rootless mode. This mode has feature limitations. 
INFO[2022-04-14T16:36:35.371678967+08:00] Running with RootlessKit integration         
INFO[2022-04-14T16:36:35.380102111+08:00] libcontainerd: started new containerd process  pid=10900
INFO[2022-04-14T16:36:35.380177152+08:00] parsed scheme: "unix"                         module=grpc
INFO[2022-04-14T16:36:35.380234630+08:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2022-04-14T16:36:35.380291412+08:00] ccResolverWrapper: sending update to cc: {[{unix:///run/user/1000/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-04-14T16:36:35.380324278+08:00] ClientConn switching balancer to "pick_first"  module=grpc
WARN[0000] deprecated version : `1`, please switch to version `2` 
INFO[2022-04-14T16:36:35.387012205+08:00] starting containerd                           revision=3df54a852345ae127d1fa3092b95168e4a88e2f8 version=1.5.11
INFO[2022-04-14T16:36:35.401470146+08:00] loading plugin "io.containerd.content.v1.content"...  type=io.containerd.content.v1
INFO[2022-04-14T16:36:35.401506555+08:00] loading plugin "io.containerd.snapshotter.v1.aufs"...  type=io.containerd.snapshotter.v1
INFO[2022-04-14T16:36:35.402780654+08:00] skip loading plugin "io.containerd.snapshotter.v1.aufs"...  error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: FATAL: Module aufs not found.\\n\"): skip plugin" type=io.containerd.snapshotter.v1
INFO[2022-04-14T16:36:35.402799741+08:00] loading plugin "io.containerd.snapshotter.v1.devmapper"...  type=io.containerd.snapshotter.v1
WARN[2022-04-14T16:36:35.402819487+08:00] failed to load plugin io.containerd.snapshotter.v1.devmapper  error="devmapper not configured"
INFO[2022-04-14T16:36:35.402844175+08:00] loading plugin "io.containerd.snapshotter.v1.native"...  type=io.containerd.snapshotter.v1
INFO[2022-04-14T16:36:35.402863686+08:00] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  type=io.containerd.snapshotter.v1
INFO[2022-04-14T16:36:35.403361991+08:00] loading plugin "io.containerd.snapshotter.v1.zfs"...  type=io.containerd.snapshotter.v1
INFO[2022-04-14T16:36:35.403518314+08:00] skip loading plugin "io.containerd.snapshotter.v1.zfs"...  error="path /home/kry/.local/share/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
INFO[2022-04-14T16:36:35.403534212+08:00] loading plugin "io.containerd.metadata.v1.bolt"...  type=io.containerd.metadata.v1
WARN[2022-04-14T16:36:35.403548605+08:00] could not use snapshotter devmapper in metadata plugin  error="devmapper not configured"
INFO[2022-04-14T16:36:35.403556640+08:00] metadata content store policy set             policy=shared
INFO[2022-04-14T16:36:35.403695053+08:00] loading plugin "io.containerd.differ.v1.walking"...  type=io.containerd.differ.v1
INFO[2022-04-14T16:36:35.403710504+08:00] loading plugin "io.containerd.gc.v1.scheduler"...  type=io.containerd.gc.v1
INFO[2022-04-14T16:36:35.403750608+08:00] loading plugin "io.containerd.service.v1.introspection-service"...  type=io.containerd.service.v1
INFO[2022-04-14T16:36:35.403772313+08:00] loading plugin "io.containerd.service.v1.containers-service"...  type=io.containerd.service.v1
INFO[2022-04-14T16:36:35.403784270+08:00] loading plugin "io.containerd.service.v1.content-service"...  type=io.containerd.service.v1
INFO[2022-04-14T16:36:35.403793455+08:00] loading plugin "io.containerd.service.v1.diff-service"...  type=io.containerd.service.v1
INFO[2022-04-14T16:36:35.403803681+08:00] loading plugin "io.containerd.service.v1.images-service"...  type=io.containerd.service.v1
INFO[2022-04-14T16:36:35.403812255+08:00] loading plugin "io.containerd.service.v1.leases-service"...  type=io.containerd.service.v1
INFO[2022-04-14T16:36:35.403822018+08:00] loading plugin "io.containerd.service.v1.namespaces-service"...  type=io.containerd.service.v1
INFO[2022-04-14T16:36:35.403829661+08:00] loading plugin "io.containerd.service.v1.snapshots-service"...  type=io.containerd.service.v1
INFO[2022-04-14T16:36:35.403841692+08:00] loading plugin "io.containerd.runtime.v1.linux"...  type=io.containerd.runtime.v1
INFO[2022-04-14T16:36:35.403888636+08:00] loading plugin "io.containerd.runtime.v2.task"...  type=io.containerd.runtime.v2
INFO[2022-04-14T16:36:35.403922478+08:00] loading plugin "io.containerd.monitor.v1.cgroups"...  type=io.containerd.monitor.v1
INFO[2022-04-14T16:36:35.404190269+08:00] loading plugin "io.containerd.service.v1.tasks-service"...  type=io.containerd.service.v1
INFO[2022-04-14T16:36:35.404211318+08:00] loading plugin "io.containerd.internal.v1.restart"...  type=io.containerd.internal.v1
INFO[2022-04-14T16:36:35.404290561+08:00] loading plugin "io.containerd.grpc.v1.containers"...  type=io.containerd.grpc.v1
INFO[2022-04-14T16:36:35.404308239+08:00] loading plugin "io.containerd.grpc.v1.content"...  type=io.containerd.grpc.v1
INFO[2022-04-14T16:36:35.404318578+08:00] loading plugin "io.containerd.grpc.v1.diff"...  type=io.containerd.grpc.v1
INFO[2022-04-14T16:36:35.404328435+08:00] loading plugin "io.containerd.grpc.v1.events"...  type=io.containerd.grpc.v1
INFO[2022-04-14T16:36:35.404346145+08:00] loading plugin "io.containerd.grpc.v1.healthcheck"...  type=io.containerd.grpc.v1
INFO[2022-04-14T16:36:35.404360635+08:00] loading plugin "io.containerd.grpc.v1.images"...  type=io.containerd.grpc.v1
INFO[2022-04-14T16:36:35.404376536+08:00] loading plugin "io.containerd.grpc.v1.leases"...  type=io.containerd.grpc.v1
INFO[2022-04-14T16:36:35.404387577+08:00] loading plugin "io.containerd.grpc.v1.namespaces"...  type=io.containerd.grpc.v1
INFO[2022-04-14T16:36:35.404397109+08:00] loading plugin "io.containerd.internal.v1.opt"...  type=io.containerd.internal.v1
WARN[2022-04-14T16:36:35.404419661+08:00] failed to load plugin io.containerd.internal.v1.opt  error="mkdir /opt/containerd: permission denied"
INFO[2022-04-14T16:36:35.404429411+08:00] loading plugin "io.containerd.grpc.v1.snapshots"...  type=io.containerd.grpc.v1
INFO[2022-04-14T16:36:35.404440723+08:00] loading plugin "io.containerd.grpc.v1.tasks"...  type=io.containerd.grpc.v1
INFO[2022-04-14T16:36:35.404463437+08:00] loading plugin "io.containerd.grpc.v1.version"...  type=io.containerd.grpc.v1
INFO[2022-04-14T16:36:35.404473852+08:00] loading plugin "io.containerd.grpc.v1.introspection"...  type=io.containerd.grpc.v1
INFO[2022-04-14T16:36:35.404623728+08:00] serving...                                    address=/run/user/1000/docker/containerd/containerd-debug.sock
INFO[2022-04-14T16:36:35.404666460+08:00] serving...                                    address=/run/user/1000/docker/containerd/containerd.sock.ttrpc
INFO[2022-04-14T16:36:35.404708111+08:00] serving...                                    address=/run/user/1000/docker/containerd/containerd.sock
INFO[2022-04-14T16:36:35.404722968+08:00] containerd successfully booted in 0.019197s  
INFO[2022-04-14T16:36:35.412090719+08:00] parsed scheme: "unix"                         module=grpc
INFO[2022-04-14T16:36:35.412116161+08:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2022-04-14T16:36:35.412132594+08:00] ccResolverWrapper: sending update to cc: {[{unix:///run/user/1000/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-04-14T16:36:35.412140372+08:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2022-04-14T16:36:35.412525144+08:00] parsed scheme: "unix"                         module=grpc
INFO[2022-04-14T16:36:35.412543475+08:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2022-04-14T16:36:35.412557066+08:00] ccResolverWrapper: sending update to cc: {[{unix:///run/user/1000/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-04-14T16:36:35.412564231+08:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2022-04-14T16:36:35.412947254+08:00] [graphdriver] using prior storage driver: fuse-overlayfs 
INFO[2022-04-14T16:36:35.414986932+08:00] Loading containers: start.                   
WARN[2022-04-14T16:36:35.416951237+08:00] Running iptables --wait -t nat -L -n failed with message: `iptables v1.4.21: can't initialize iptables table `nat': Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded.`, error: exit status 3 
INFO[2022-04-14T16:36:35.430006131+08:00] stopping event stream following graceful shutdown  error="<nil>" module=libcontainerd namespace=moby
INFO[2022-04-14T16:36:35.430272003+08:00] stopping event stream following graceful shutdown  error="context canceled" module=libcontainerd namespace=plugins.moby
INFO[2022-04-14T16:36:35.430272547+08:00] stopping healthcheck following graceful shutdown  module=libcontainerd
WARN[2022-04-14T16:36:36.431324379+08:00] grpc: addrConn.createTransport failed to connect to {unix:///run/user/1000/docker/containerd/containerd.sock  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial unix:///run/user/1000/docker/containerd/containerd.sock: timeout". Reconnecting...  module=grpc
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.4.21: can't initialize iptables table `nat': Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded.
 (exit status 3)
[rootlesskit:child ] error: command [/usr/bin/dockerd-rootless.sh] exited: exit status 1
[rootlesskit:parent] error: child exited: exit status 1

When the script is executed, calling the iptable command requires Permission denied (you must be root). That’s confusing

6

I realized I installed the rootless Docker on Ubuntu, but I get that dbus error message on my CentOS server. It looks like the user mode (or whatever the official name is) is not enabled by default.

I don’t want to play with systemd on my server, so I will try to run a VM later and follow the official documentation to see if that is wrong or incomplete. If you don’t get an answer from anyone in a day, feel free to remind me, but I will bookmark the topic :slight_smile:

7

Okay, I finally managed to install centos 7.9 on MacOS in UTM. I could not find Centos 7.6 and I could not use the minimal version either. Later it turned out my network configuration was wrong, so it is possible that the minimal version would have worked too, but I used the 10 GB DVD version.

This way I had no problem with iptables running dockerd-rootless.sh.
What I had problemw with is the overlay2 storage driver. As the documentation says:

Known limitations

  • Only the following storage drivers are supported:
    • overlay2 (only if running with kernel 5.11 or later, or Ubuntu-flavored kernel)
    • …
  • Since I haven’t upgraded my kernel, I guess it won’t work. I got permission denied, even though it worked on Ubuntu
  • You have version 4.20.3, so it would not wok for you either. I know, you didn’t mention any issue with overlay2.
  • When I ran dockerd-rootless.sh I also had an error message that br_netfilter was not enabled, so I ran:
    sudo modprobe br_netfilter
    Although, it was just a warning, so Docker was able to run. I noticed that it used vfs as storage driver, but it worked.
  • So my guess is that your kernel version (4.20) is still not compatible with your Docker environment. I don’t have time to try every scenario, so the only thing I know it works is leaving kernel version 3.10 and using the vfs storage driver with a non-root user. I don’t know if you could upgrade the kernel to version 5.11 and even if you could, I am not sure that would help.

As for tmpfs issue:

When we run dockerd-rootless.sh it re-runs itself with the rootless kit:

# Re-exec the script via RootlessKit, so as to create unprivileged {user,mount,network} namespaces.
#
# --copy-up allows removing/creating files in the directories by creating tmpfs and symlinks
# * /etc: copy-up is required so as to prevent `/etc/resolv.conf` in the
#         namespace from being unexpectedly unmounted when `/etc/resolv.conf` is recreated on the host
#         (by either systemd-networkd or NetworkManager)
# * /run: copy-up is required so that we can create /run/docker (hardcoded for plugins) in our namespace
exec $rootlesskit \
  --net=$net --mtu=$mtu \
  --slirp4netns-sandbox=$DOCKERD_ROOTLESS_ROOTLESSKIT_SLIRP4NETNS_SANDBOX \
  --slirp4netns-seccomp=$DOCKERD_ROOTLESS_ROOTLESSKIT_SLIRP4NETNS_SECCOMP \
  --disable-host-loopback --port-driver=$DOCKERD_ROOTLESS_ROOTLESSKIT_PORT_DRIVER \
  --copy-up=/etc --copy-up=/run \
  --propagation=rslave \
  $DOCKERD_ROOTLESS_ROOTLESSKIT_FLAGS \
  $0 $@

So having etc on tmpfs is normal, so I don’t understand why you had permission denied error. The script does the same for me, and I see tmpfs running in the rootless kit without having any error message.

One thing it is worth to mention is I disabled selinux because when I first run the script and got permission denied for overlay2, I came to the conclusion that selinux blocks overlay2.

If you check the documentation: Run the Docker daemon as a non-root user (Rootless mode) | Docker Docs
I mentions a bug in older Docker versions:

iptables failed: iptables -t nat -N DOCKER: Fatal: can’t open lock file /run/xtables.lock: Permission denied

You have a similar message. What is your Docker version? If you installed the latest on centos, you should have 20.10.14, however, I don’t know which version is available for CentOS 7.6. If you have the latest, it is possible that the error could happen in newer versions too, for reasons like upgraded kernel versions which is the only one I know you changed, but I am not sure that this is the cause.

8

Great, thank you so much for your test and reply. I will try it later.

Docker version:

[kry@localhost ~]$ docker  --version
Docker version 20.10.14, build a224086
9

I redeployed rootless and the conclusions are as follows:
There is no “mount: Permission denied” error.
The 4.20.3 kernel version does not support the “overlay2” storage driver.
The 5.17.3 kernel version supports the “overlay2” storage driver.
Both 4.20.3 and 5.17.3 kernel versions have directory creation errors showing permission issues.
In the 5.17.3 kernel version, when executing “docker top container”, an error about “cgroups” is prompted. This error is not difficult to understand. The official website clearly supports “Cgroup is supported only when running with cgroup v2 and systemd. See Limiting resources .”,I did not use the systemd runing docker .

System version: CentOS 7.6
Kernel version: 4.20.3 and 5.17.3 (trials separately)
Run: select “Without systemd”

# Install rely
yum -y install shadow-utils

#Prohibit selinux
setenforce 0
sed -ri '/^[^#]*SELINUX=/s#=.+$#=disabled#' /etc/selinux/config

#Set kernel parameters
echo "user.max_user_namespaces=65536" >> /etc/sysctl.conf
sudo sysctl --system

#Create user and set password
useradd kry
passwd kry

#Set subuid and subgid
echo "kry:100000:65536" >> /etc/subuid
echo "kry:100000:65536" >> /etc/subgid

#Set repo
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

#Install
yum  -y  install docker-ce-rootless-extras

#Install rootless
Additional host login：  ssh kry@ip
/usr/bin/dockerd-rootless-setuptool.sh install

Exec dockerd execute part complete information

+ exec dockerd
INFO[2022-04-18T13:53:12.571191231+08:00] Starting up                                  
WARN[2022-04-18T13:53:12.571782405+08:00] Running in rootless mode. This mode has feature limitations. 
INFO[2022-04-18T13:53:12.571788155+08:00] Running with RootlessKit integration         
INFO[2022-04-18T13:53:12.573223142+08:00] libcontainerd: started new containerd process  pid=21655
INFO[2022-04-18T13:53:12.573266682+08:00] parsed scheme: "unix"                         module=grpc
INFO[2022-04-18T13:53:12.573274787+08:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2022-04-18T13:53:12.573314942+08:00] ccResolverWrapper: sending update to cc: {[{unix:///run/user/1000/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-04-18T13:53:12.573328533+08:00] ClientConn switching balancer to "pick_first"  module=grpc
WARN[0000] deprecated version : `1`, please switch to version `2` 
INFO[2022-04-18T13:53:12.585266090+08:00] starting containerd                           revision=3df54a852345ae127d1fa3092b95168e4a88e2f8 version=1.5.11
INFO[2022-04-18T13:53:12.601769431+08:00] loading plugin "io.containerd.content.v1.content"...  type=io.containerd.content.v1
INFO[2022-04-18T13:53:12.601865665+08:00] loading plugin "io.containerd.snapshotter.v1.aufs"...  type=io.containerd.snapshotter.v1
INFO[2022-04-18T13:53:12.603895339+08:00] skip loading plugin "io.containerd.snapshotter.v1.aufs"...  error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: FATAL: Module aufs not found.\\n\"): skip plugin" type=io.containerd.snapshotter.v1
INFO[2022-04-18T13:53:12.603916666+08:00] loading plugin "io.containerd.snapshotter.v1.devmapper"...  type=io.containerd.snapshotter.v1
WARN[2022-04-18T13:53:12.603932175+08:00] failed to load plugin io.containerd.snapshotter.v1.devmapper  error="devmapper not configured"
INFO[2022-04-18T13:53:12.603941577+08:00] loading plugin "io.containerd.snapshotter.v1.native"...  type=io.containerd.snapshotter.v1
INFO[2022-04-18T13:53:12.604011517+08:00] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  type=io.containerd.snapshotter.v1
INFO[2022-04-18T13:53:12.644610084+08:00] loading plugin "io.containerd.snapshotter.v1.zfs"...  type=io.containerd.snapshotter.v1
INFO[2022-04-18T13:53:12.644807869+08:00] skip loading plugin "io.containerd.snapshotter.v1.zfs"...  error="path /home/kry/.local/share/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
INFO[2022-04-18T13:53:12.644825651+08:00] loading plugin "io.containerd.metadata.v1.bolt"...  type=io.containerd.metadata.v1
WARN[2022-04-18T13:53:12.644881499+08:00] could not use snapshotter devmapper in metadata plugin  error="devmapper not configured"
INFO[2022-04-18T13:53:12.644891033+08:00] metadata content store policy set             policy=shared
INFO[2022-04-18T13:53:12.648438514+08:00] loading plugin "io.containerd.differ.v1.walking"...  type=io.containerd.differ.v1
INFO[2022-04-18T13:53:12.648462779+08:00] loading plugin "io.containerd.gc.v1.scheduler"...  type=io.containerd.gc.v1
INFO[2022-04-18T13:53:12.648491029+08:00] loading plugin "io.containerd.service.v1.introspection-service"...  type=io.containerd.service.v1
INFO[2022-04-18T13:53:12.648513409+08:00] loading plugin "io.containerd.service.v1.containers-service"...  type=io.containerd.service.v1
INFO[2022-04-18T13:53:12.648522598+08:00] loading plugin "io.containerd.service.v1.content-service"...  type=io.containerd.service.v1
INFO[2022-04-18T13:53:12.648531248+08:00] loading plugin "io.containerd.service.v1.diff-service"...  type=io.containerd.service.v1
INFO[2022-04-18T13:53:12.648539284+08:00] loading plugin "io.containerd.service.v1.images-service"...  type=io.containerd.service.v1
INFO[2022-04-18T13:53:12.648547304+08:00] loading plugin "io.containerd.service.v1.leases-service"...  type=io.containerd.service.v1
INFO[2022-04-18T13:53:12.648555539+08:00] loading plugin "io.containerd.service.v1.namespaces-service"...  type=io.containerd.service.v1
INFO[2022-04-18T13:53:12.648562909+08:00] loading plugin "io.containerd.service.v1.snapshots-service"...  type=io.containerd.service.v1
INFO[2022-04-18T13:53:12.648571141+08:00] loading plugin "io.containerd.runtime.v1.linux"...  type=io.containerd.runtime.v1
INFO[2022-04-18T13:53:12.648665227+08:00] loading plugin "io.containerd.runtime.v2.task"...  type=io.containerd.runtime.v2
INFO[2022-04-18T13:53:12.648744526+08:00] loading plugin "io.containerd.monitor.v1.cgroups"...  type=io.containerd.monitor.v1
INFO[2022-04-18T13:53:12.649003633+08:00] loading plugin "io.containerd.service.v1.tasks-service"...  type=io.containerd.service.v1
INFO[2022-04-18T13:53:12.649027538+08:00] loading plugin "io.containerd.internal.v1.restart"...  type=io.containerd.internal.v1
INFO[2022-04-18T13:53:12.649061946+08:00] loading plugin "io.containerd.grpc.v1.containers"...  type=io.containerd.grpc.v1
INFO[2022-04-18T13:53:12.649072719+08:00] loading plugin "io.containerd.grpc.v1.content"...  type=io.containerd.grpc.v1
INFO[2022-04-18T13:53:12.649080806+08:00] loading plugin "io.containerd.grpc.v1.diff"...  type=io.containerd.grpc.v1
INFO[2022-04-18T13:53:12.649089865+08:00] loading plugin "io.containerd.grpc.v1.events"...  type=io.containerd.grpc.v1
INFO[2022-04-18T13:53:12.649098713+08:00] loading plugin "io.containerd.grpc.v1.healthcheck"...  type=io.containerd.grpc.v1
INFO[2022-04-18T13:53:12.649106761+08:00] loading plugin "io.containerd.grpc.v1.images"...  type=io.containerd.grpc.v1
INFO[2022-04-18T13:53:12.649116186+08:00] loading plugin "io.containerd.grpc.v1.leases"...  type=io.containerd.grpc.v1
INFO[2022-04-18T13:53:12.649124095+08:00] loading plugin "io.containerd.grpc.v1.namespaces"...  type=io.containerd.grpc.v1
INFO[2022-04-18T13:53:12.649132634+08:00] loading plugin "io.containerd.internal.v1.opt"...  type=io.containerd.internal.v1
WARN[2022-04-18T13:53:12.649162746+08:00] failed to load plugin io.containerd.internal.v1.opt  error="mkdir /opt/containerd: permission denied"
INFO[2022-04-18T13:53:12.649174003+08:00] loading plugin "io.containerd.grpc.v1.snapshots"...  type=io.containerd.grpc.v1
INFO[2022-04-18T13:53:12.649185398+08:00] loading plugin "io.containerd.grpc.v1.tasks"...  type=io.containerd.grpc.v1
INFO[2022-04-18T13:53:12.649194269+08:00] loading plugin "io.containerd.grpc.v1.version"...  type=io.containerd.grpc.v1
INFO[2022-04-18T13:53:12.649204219+08:00] loading plugin "io.containerd.grpc.v1.introspection"...  type=io.containerd.grpc.v1
INFO[2022-04-18T13:53:12.649330015+08:00] serving...                                    address=/run/user/1000/docker/containerd/containerd-debug.sock
INFO[2022-04-18T13:53:12.649375944+08:00] serving...                                    address=/run/user/1000/docker/containerd/containerd.sock.ttrpc
INFO[2022-04-18T13:53:12.649421722+08:00] serving...                                    address=/run/user/1000/docker/containerd/containerd.sock
INFO[2022-04-18T13:53:12.649438936+08:00] containerd successfully booted in 0.065380s  
INFO[2022-04-18T13:53:12.656812289+08:00] parsed scheme: "unix"                         module=grpc
INFO[2022-04-18T13:53:12.656834415+08:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2022-04-18T13:53:12.656852535+08:00] ccResolverWrapper: sending update to cc: {[{unix:///run/user/1000/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-04-18T13:53:12.656861047+08:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2022-04-18T13:53:12.657479694+08:00] parsed scheme: "unix"                         module=grpc
INFO[2022-04-18T13:53:12.657491941+08:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2022-04-18T13:53:12.657501847+08:00] ccResolverWrapper: sending update to cc: {[{unix:///run/user/1000/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-04-18T13:53:12.657507682+08:00] ClientConn switching balancer to "pick_first"  module=grpc
ERRO[2022-04-18T13:53:12.657977442+08:00] overlay is not supported for Rootless with SELinux  storage-driver=overlay2
INFO[2022-04-18T13:53:12.682964914+08:00] Loading containers: start.                   
INFO[2022-04-18T13:53:12.746520128+08:00] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address 
INFO[2022-04-18T13:53:12.825972213+08:00] Loading containers: done.                    
INFO[2022-04-18T13:53:12.833729808+08:00] Docker daemon                                 commit=87a90dc graphdriver(s)=fuse-overlayfs version=20.10.14
INFO[2022-04-18T13:53:12.833889917+08:00] Daemon has completed initialization          
INFO[2022-04-18T13:53:12.857584255+08:00] API listen on /run/user/1000/docker.sock

Notable information

WARN[2022-04-18T13:53:12.649162746+08:00] failed to load plugin io.containerd.internal.v1.opt  error="mkdir /opt/containerd: permission denied"
ERRO[2022-04-18T13:53:12.657977442+08:00] overlay is not supported for Rootless with SELinux  storage-driver=overlay2
INFO[2022-04-18T13:53:12.833729808+08:00] Docker daemon                                 commit=87a90dc graphdriver(s)=fuse-overlayfs version=20.10.14

You can pull the container and run the container normally.
This deployment did not reproduce the “mount: permission denied” error,Happy!

mount: permission denied
[rootlesskit:child ] error: command [/usr/bin/dockerd-rootless.sh] exited: exit status 32
[rootlesskit:parent] error: child exited: exit status 32

Now start trying to manually upgrade the kernel version to 5.17.3
upgrade process

rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
yum install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm
yum --enablerepo=elrepo-kernel -y install kernel-ml
“GRUB_DEFAULT=0” Add to the third line of the /etc/default/grub file 
grub2-mkconfig -o /boot/grub2/grub.cfg
reboot

In the kry user, execute “dockerd-rootless.sh”, the error message is as follows.

WARN[2022-04-18T14:07:34.027628641+08:00] failed to load plugin io.containerd.internal.v1.opt  error="mkdir /opt/containerd: permission denied

WARN[2022-04-18T14:07:34.118359030+08:00] Running modprobe bridge br_netfilter failed with message: modprobe: ERROR: could not insert 'br_netfilter': Operation not permitted
insmod /lib/modules/5.17.3-1.el7.elrepo.x86_64/kernel/net/bridge/br_netfilter.ko 
, error: exit status 1 

INFO[2022-04-18T14:07:34.410570315+08:00] Docker daemon                                 commit=87a90dc graphdriver(s)=fuse-overlayfs version=20.10.14

permission denied solution

mkdir /opt/containerd
chown kry:kry /opt/containerd

error br_netfilter solution

sudo modprobe br_netfilter

storage drivers solution

cat   .config/docker/daemon.json
{
  "storage-driver": "overlay2"
}

Re-execute “dockerd-rootless.sh” after solving,

+ exec dockerd
INFO[2022-04-18T14:11:19.660352383+08:00] Starting up                                  
WARN[2022-04-18T14:11:19.660390220+08:00] Running in rootless mode. This mode has feature limitations. 
INFO[2022-04-18T14:11:19.660394813+08:00] Running with RootlessKit integration         
INFO[2022-04-18T14:11:19.661041611+08:00] libcontainerd: started new containerd process  pid=7654
INFO[2022-04-18T14:11:19.661065902+08:00] parsed scheme: "unix"                         module=grpc
INFO[2022-04-18T14:11:19.661073597+08:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2022-04-18T14:11:19.661085896+08:00] ccResolverWrapper: sending update to cc: {[{unix:///run/user/1000/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-04-18T14:11:19.661100917+08:00] ClientConn switching balancer to "pick_first"  module=grpc
WARN[0000] deprecated version : `1`, please switch to version `2` 
INFO[2022-04-18T14:11:19.672132552+08:00] starting containerd                           revision=3df54a852345ae127d1fa3092b95168e4a88e2f8 version=1.5.11
INFO[2022-04-18T14:11:19.686827699+08:00] loading plugin "io.containerd.content.v1.content"...  type=io.containerd.content.v1
INFO[2022-04-18T14:11:19.686854731+08:00] loading plugin "io.containerd.snapshotter.v1.aufs"...  type=io.containerd.snapshotter.v1
INFO[2022-04-18T14:11:19.687685294+08:00] skip loading plugin "io.containerd.snapshotter.v1.aufs"...  error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: FATAL: Module aufs not found.\\n\"): skip plugin" type=io.containerd.snapshotter.v1
INFO[2022-04-18T14:11:19.687705832+08:00] loading plugin "io.containerd.snapshotter.v1.devmapper"...  type=io.containerd.snapshotter.v1
WARN[2022-04-18T14:11:19.687723013+08:00] failed to load plugin io.containerd.snapshotter.v1.devmapper  error="devmapper not configured"
INFO[2022-04-18T14:11:19.687731234+08:00] loading plugin "io.containerd.snapshotter.v1.native"...  type=io.containerd.snapshotter.v1
INFO[2022-04-18T14:11:19.687765870+08:00] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  type=io.containerd.snapshotter.v1
INFO[2022-04-18T14:11:19.689609241+08:00] loading plugin "io.containerd.snapshotter.v1.zfs"...  type=io.containerd.snapshotter.v1
INFO[2022-04-18T14:11:19.689762828+08:00] skip loading plugin "io.containerd.snapshotter.v1.zfs"...  error="path /home/kry/.local/share/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
INFO[2022-04-18T14:11:19.689784722+08:00] loading plugin "io.containerd.metadata.v1.bolt"...  type=io.containerd.metadata.v1
WARN[2022-04-18T14:11:19.689797869+08:00] could not use snapshotter devmapper in metadata plugin  error="devmapper not configured"
INFO[2022-04-18T14:11:19.689813204+08:00] metadata content store policy set             policy=shared
INFO[2022-04-18T14:11:19.689906768+08:00] loading plugin "io.containerd.differ.v1.walking"...  type=io.containerd.differ.v1
INFO[2022-04-18T14:11:19.689923935+08:00] loading plugin "io.containerd.gc.v1.scheduler"...  type=io.containerd.gc.v1
INFO[2022-04-18T14:11:19.689970659+08:00] loading plugin "io.containerd.service.v1.introspection-service"...  type=io.containerd.service.v1
INFO[2022-04-18T14:11:19.689993177+08:00] loading plugin "io.containerd.service.v1.containers-service"...  type=io.containerd.service.v1
INFO[2022-04-18T14:11:19.690019064+08:00] loading plugin "io.containerd.service.v1.content-service"...  type=io.containerd.service.v1
INFO[2022-04-18T14:11:19.690032992+08:00] loading plugin "io.containerd.service.v1.diff-service"...  type=io.containerd.service.v1
INFO[2022-04-18T14:11:19.690050893+08:00] loading plugin "io.containerd.service.v1.images-service"...  type=io.containerd.service.v1
INFO[2022-04-18T14:11:19.690066835+08:00] loading plugin "io.containerd.service.v1.leases-service"...  type=io.containerd.service.v1
INFO[2022-04-18T14:11:19.690082186+08:00] loading plugin "io.containerd.service.v1.namespaces-service"...  type=io.containerd.service.v1
INFO[2022-04-18T14:11:19.690099013+08:00] loading plugin "io.containerd.service.v1.snapshots-service"...  type=io.containerd.service.v1
INFO[2022-04-18T14:11:19.690108479+08:00] loading plugin "io.containerd.runtime.v1.linux"...  type=io.containerd.runtime.v1
INFO[2022-04-18T14:11:19.690168980+08:00] loading plugin "io.containerd.runtime.v2.task"...  type=io.containerd.runtime.v2
INFO[2022-04-18T14:11:19.690215695+08:00] loading plugin "io.containerd.monitor.v1.cgroups"...  type=io.containerd.monitor.v1
INFO[2022-04-18T14:11:19.690496993+08:00] loading plugin "io.containerd.service.v1.tasks-service"...  type=io.containerd.service.v1
INFO[2022-04-18T14:11:19.690528855+08:00] loading plugin "io.containerd.internal.v1.restart"...  type=io.containerd.internal.v1
INFO[2022-04-18T14:11:19.690577617+08:00] loading plugin "io.containerd.grpc.v1.containers"...  type=io.containerd.grpc.v1
INFO[2022-04-18T14:11:19.690590646+08:00] loading plugin "io.containerd.grpc.v1.content"...  type=io.containerd.grpc.v1
INFO[2022-04-18T14:11:19.690607517+08:00] loading plugin "io.containerd.grpc.v1.diff"...  type=io.containerd.grpc.v1
INFO[2022-04-18T14:11:19.690616698+08:00] loading plugin "io.containerd.grpc.v1.events"...  type=io.containerd.grpc.v1
INFO[2022-04-18T14:11:19.690626244+08:00] loading plugin "io.containerd.grpc.v1.healthcheck"...  type=io.containerd.grpc.v1
INFO[2022-04-18T14:11:19.690636486+08:00] loading plugin "io.containerd.grpc.v1.images"...  type=io.containerd.grpc.v1
INFO[2022-04-18T14:11:19.690643616+08:00] loading plugin "io.containerd.grpc.v1.leases"...  type=io.containerd.grpc.v1
INFO[2022-04-18T14:11:19.690660056+08:00] loading plugin "io.containerd.grpc.v1.namespaces"...  type=io.containerd.grpc.v1
INFO[2022-04-18T14:11:19.690667895+08:00] loading plugin "io.containerd.internal.v1.opt"...  type=io.containerd.internal.v1
INFO[2022-04-18T14:11:19.690770503+08:00] loading plugin "io.containerd.grpc.v1.snapshots"...  type=io.containerd.grpc.v1
INFO[2022-04-18T14:11:19.690786964+08:00] loading plugin "io.containerd.grpc.v1.tasks"...  type=io.containerd.grpc.v1
INFO[2022-04-18T14:11:19.690797276+08:00] loading plugin "io.containerd.grpc.v1.version"...  type=io.containerd.grpc.v1
INFO[2022-04-18T14:11:19.690814074+08:00] loading plugin "io.containerd.grpc.v1.introspection"...  type=io.containerd.grpc.v1
INFO[2022-04-18T14:11:19.690944433+08:00] serving...                                    address=/run/user/1000/docker/containerd/containerd-debug.sock
INFO[2022-04-18T14:11:19.690997188+08:00] serving...                                    address=/run/user/1000/docker/containerd/containerd.sock.ttrpc
INFO[2022-04-18T14:11:19.691049009+08:00] serving...                                    address=/run/user/1000/docker/containerd/containerd.sock
INFO[2022-04-18T14:11:19.691062123+08:00] containerd successfully booted in 0.019711s  
INFO[2022-04-18T14:11:19.697636173+08:00] parsed scheme: "unix"                         module=grpc
INFO[2022-04-18T14:11:19.697666173+08:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2022-04-18T14:11:19.697688704+08:00] ccResolverWrapper: sending update to cc: {[{unix:///run/user/1000/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-04-18T14:11:19.697697166+08:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2022-04-18T14:11:19.698084626+08:00] parsed scheme: "unix"                         module=grpc
INFO[2022-04-18T14:11:19.698114562+08:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2022-04-18T14:11:19.698125929+08:00] ccResolverWrapper: sending update to cc: {[{unix:///run/user/1000/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-04-18T14:11:19.698130757+08:00] ClientConn switching balancer to "pick_first"  module=grpc
WARN[2022-04-18T14:11:19.700972853+08:00] Your kernel does not support cgroup blkio weight 
WARN[2022-04-18T14:11:19.700987496+08:00] Your kernel does not support cgroup blkio weight_device 
INFO[2022-04-18T14:11:19.701099475+08:00] Loading containers: start.                   
INFO[2022-04-18T14:11:19.764264361+08:00] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address 
INFO[2022-04-18T14:11:19.786181372+08:00] Loading containers: done.                    
WARN[2022-04-18T14:11:19.791622006+08:00] Not using native diff for overlay2, this may cause degraded performance for building images: running in a user namespace  storage-driver=overlay2
INFO[2022-04-18T14:11:19.791781056+08:00] Docker daemon                                 commit=87a90dc graphdriver(s)=overlay2 version=20.10.14
INFO[2022-04-18T14:11:19.791817509+08:00] Daemon has completed initialization          
INFO[2022-04-18T14:11:19.800899326+08:00] API listen on /run/user/1000/docker.sock

View storage driver information

[kry@localhost ~]$ docker info |grep "Storage"
 Storage Driver: overlay2
WARNING: Running in rootless-mode without cgroups. To enable cgroups in rootless-mode, you need to boot the system in cgroup v2 mode.

Pull image to create container

[kry@localhost ~]$ docker  pull  nginx  
Using default tag: latest
latest: Pulling from library/nginx
c229119241af: Pull complete 
2215908dc0a2: Pull complete 
08c3cb2073f1: Pull complete 
18f38162c0ce: Pull complete 
10e2168f148a: Pull complete 
c4ffe9532b5f: Pull complete 
Digest: sha256:2275af0f20d71b293916f1958f8497f987b8d8fd8113df54635f2a5915002bf1
Status: Downloaded newer image for nginx:latest
docker.io/library/nginx:latest

[kry@localhost ~]$ docker  run -d -p8080:80 nginx  
eb6f16bd2f81eca4adec71938c5f83be234a2f2e2a8c9f1f85d5c8da7f6c624f

[kry@localhost docker]$ netstat  -antulp   |grep "8080"
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      7945/rootlesskit    
tcp6       0      0 :::8080                 :::*                    LISTEN      7945/rootlesskit 

[kry@localhost ~]$ curl  127.0.0.1:8080  -I 
HTTP/1.1 200 OK
Server: nginx/1.21.6
Date: Mon, 18 Apr 2022 07:37:28 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 25 Jan 2022 15:03:52 GMT
Connection: keep-alive
ETag: "61f01158-267"
Accept-Ranges: bytes

There are no problems with pulling, running, and restarting the test container.
Only if the container is stopped, an error will be generated.

Error response from daemon: runc did not terminate successfully: exit status 1: container_linux.go:187: getting all container pids from cgroups caused: lstat : no such file or directory
: unknown

Personally suspect that it is caused by not running docker based on systemd.
Based on “Without systemd” mode, tested with different kernel versions,accumulate rich experience.
Thank you very much for your help and reply!

1 Like
10

Thank you for your detailed post about the steps you have done.

Possible, but the docs of runc says the following about Cgroups V2:

On cgroup v2 hosts, it is highly recommended to run runc with the systemd cgroup driver ( runc --systemd-cgroup ), though not mandatory.

So it should work without Systemd. You had some surprising error messages so I can only guess without repeating your steps in my environment. I would like to do that, but I don’t have the time for it. I appreciate that you shared what you tried in details. If you find out more, please share that too. It looks like Rootless Docker could be improved on CentOS 7, but it might not be the main focus of Docker. Red Hat has its own software, Podman supports rootless containers from the beginning, so I am not sure supporting Docker’s rootless containers is in their interest and I didn’t follow what the newer CentOS versions can do.

11

Good ，thank you for taking the time out of your busy schedule to help me solve the problem.
Thanks a lot for your help！ :smiley: