Docker problem in unpriviledged lxc on Debian 11.2 (Bullseye)

Trying to run docker inside an unprivileged Linux container gives an error.
Can someone help me with this?

root@TEST-LXC:~# docker --version
Docker version 20.10.5+dfsg1, build 55c4c88
root@TEST-LXC:~# uname -a
Linux TEST-LXC 5.10.0-11-amd64 #1 SMP Debian 5.10.92-1 (2022-01-18) x86_64 GNU/Linux
root@TEST-LXC:~# cat /etc/debian_version 
11.2
root@TEST-LXC:~# docker run -d debian
2260b9518cdd21f3cefd37d2ca4a3e3eaf3c15384cb0ee8fcf458dc0eb12a86f
docker: Error response from daemon: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: process_linux.go:458: setting cgroup config for procHooks process caused: can't load program: operation not permitted: unknown.
root@TEST-LXC:~#

Lxc config:

# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template:
# Template script checksum (SHA-1): xxxxx
# For additional config options, please look at lxc.container.conf(5)

# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)
features:  keyctl=1,nesting=1
#lxc.cgroup.devices.allow: a
#lxc.cap.drop:


# Distribution configuration
lxc.include = /usr/share/lxc/config/debian.common.conf
lxc.include = /usr/share/lxc/config/debian.userns.conf
lxc.arch = x86_64

# Container specific configuration
#lxc.apparmor.profile = generated
lxc.apparmor.profile = unconfined
lxc.apparmor.allow_nesting = 1
lxc.idmap = u 0 100000 65536
lxc.idmap = g 0 100000 65536
lxc.rootfs.path = dir:/home/xxx/.local/share/lxc/TEST-LXC/rootfs
lxc.uts.name = TEST-LXC

# Network configuration
lxc.net.0.type = veth
lxc.net.0.link = veth0xxx
lxc.net.0.name = veth1xxx
lxc.net.0.flags = up
#lxc.net.0.hwaddr = 00:00:00:00:00:00
lxc.net.0.ipv6.address = fd75::x00x:b00x/128
lxc.net.0.ipv6.gateway = fd75::x00x:a00x
lxc.net.0.ipv4.address = 192.168.x.xy/31
lxc.net.0.ipv4.gateway = 192.168.x.xx

Docker in Docker works in a privileged Docker container. I suppose the same is true in an LXC container. If you have LXD, not just LXC, you can run containers in virtual machines. I also use it sometimes and the performance was better than in a KVM virtual machine. Running virtual machines with default configuration is easy. You just need to use the --vm parameter after the launch subcommand.

Hello Ákos Takács,
thank you very much for your answer.
Here someone describes, it is working for him, to run docker inside an unpriviledged LXC:

The error message in my try I find not very helpfull:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: process_linux.go:458: setting cgroup config for procHooks process caused: can’t load program: operation not permitted: unknown.
`
I wonder: What program can not be loaded? What operation exactly is not permitted?

Thank you again for your reply, and best regards,
Jan Michael Greiner

The author of the article describes a special case and doesn’t really mention why it worked or why it should work. The configuration seems to be for Proxmox and not any LXC container. “nesting” is a valid configuration for lxc, but it is often used with the privileged mode. I can’t find the keyctl option anywhere, only in posts related to proxmox or systemd. I also checked my files in /usr/share/lxc and couldn’t find anything with “keyctl” or “features”. I don’t have Debian machine currently, so it may be a different version of LXC. Mine is 4.21. Do you also try on Proxmox?

But I managed to run Docker containers in an LXC container with this command:

lxc launch ubuntu:20.04 --config 'security.nesting=true'

It shares /proc and /sys with the guest OS.

My host is also Ubuntu 20.04

Hello Ákos Takács,
thank you very much for looking at the thing and trying out.
I did not find a command like “lxc launch” on my debian, and neither in the Ubuntu man pages Ubuntu Manpage: Directory Listing
I suppose, the command belongs to LXD? (Linux Containers - LXD - Getting started)
And I guess (maybe wrong guess), that this command creates a virtual machine using CPU features. Whereas from my understanding, lxc uses Linux kernel features, not CPU features.
I am working on a cheap virtual server from a hosting provider, where CPU features are not enabled.
After carefully reading again Running docker inside an unprivileged LXC container on Proxmox - du.nkel.dev I see the config entry
vm-100-disk-0.raw
what to my eyes looks not like using a subdirectory of the hosts file system, but like it could be a virtual disk. Maybe I should try that… And I am missing in the configuration the user id/group id mapping, what makes the question come to my mind: Is this really an unprivileged container, or is it a virtual machine…? I will ask the author.
Thank you and best regards
Jan Michael Greiner

Yes, it is. But it still runs LXC containers

You can run virtual machines, but the default behavior is running containers. LXD just makes it easier to use LXC as Docker did in the past. I quote from my own tutorial:

Before using Docker containers it’s good to know a little about a similar tool. LXD can run containers and also virtual machines with similar commands. It uses LXC to run containers (as Docker did at the beginning) and Qemu-KVM to run virtual machines. To install LXD 4.0 LTS you need snap.

I did not want to mention it, because It does not necessarily prove that it is a VM and not a container. That raw disk could be mounted but it was strange to me too.

id mapping is a different thing. You can have different user namespace in a container or not. If you don’t ask for a user namespace, you can still privileged or unprivileged containers. When you have an unprivileged container, your access to the kernel features is more restricted.

If the author can’t answer, you can ask your question on the LXC forum: https://discuss.linuxcontainers.org/

Or you could install snap on Debian and then install LXD and use containers only from its feature set. https://snapcraft.io/install/lxd/debian