Rootless docker: i/o timeout with docker pull

Open Source Projects DockerEngine
1

Dear Docker Community,

I have set up rootless docker following this documentation Run the Docker daemon as a non-root user (Rootless mode) | Docker Docs in openSUSE Tumbleweed (rolling release, currently version 20230917) but every docker pull gives a i/o timeout. The same applies, e.g., for docker login.

someuser@somehost:~> docker pull hello-world
Using default tag: latest
Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 10.0.2.3:53: read udp 10.0.2.100:48971->10.0.2.3:53: i/o timeout

On the same host, there is a docker installation for root, where pulling of docker images works fine. I have tried to add additional DNS servers. No change. I also tried to reinstall rootless docker. No change. Furthermore, I have actually read any forum post one may find with google searching for ā€œdocker pull i/o timeoutā€, but nothing helped.

The current workaround: Pull the images with root docker, save the images, change permissions and ownership of the saved images and load them with rootless docker. Works, but this is an annoying workaround.

Do you have further hints on that? Any hint highly appreciated!

Output of docker version:

someuser@somehost:~> docker version
Client:
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:30:51 2023
 OS/Arch:           linux/amd64
 Context:           rootless

Server: Docker Engine - Community
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:32:17 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.3
  GitCommit:        7880925980b188f4c97b462f709d0db8e8962aff
 runc:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfc
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 rootlesskit:
  Version:          1.1.0
  ApiVersion:       1.1.1
  NetworkDriver:    slirp4netns
  PortDriver:       builtin
  StateDir:         /tmp/rootlesskit878318901
 slirp4netns:
  Version:          1.2.1
  GitCommit:        unknown

root@somehost:~> docker version
Client:
 Version:           24.0.6-ce
 API version:       1.43
 Go version:        go1.20.8
 Git commit:        1a7969545d73
 Built:             Thu Sep 14 00:00:00 2023
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          24.0.6-ce
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.8
  Git commit:       1a7969545d73
  Built:            Thu Sep 14 00:00:00 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.6
  GitCommit:        091922f03c2762540fd057fba91260237ff86acb
 runc:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfcbc907
 docker-init:
  Version:          0.1.7_catatonit
  GitCommit:

Output of docker info:

someuser@somehost:~> docker info
Client:
 Version:    24.0.6
 Context:    rootless
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.11.2
    Path:     /usr/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  2.21.0
    Path:     /usr/lib/docker/cli-plugins/docker-compose

Server:
 Containers: 18
  Running: 6
  Paused: 0
  Stopped: 12
 Images: 6
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7880925980b188f4c97b462f709d0db8e8962aff
 runc version: v1.1.9-0-gccaecfc
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  rootless
  cgroupns
 Kernel Version: 6.4.11-1-default
 Operating System: openSUSE Tumbleweed
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 7.747GiB
 Name: somename
 ID: someid
 Docker Root Dir: /somefolder/.local/share/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: No cpuset support
WARNING: No io.weight support
WARNING: No io.weight (per device) support
WARNING: No io.max (rbps) support
WARNING: No io.max (wbps) support
WARNING: No io.max (riops) support
WARNING: No io.max (wiops) support

root@somehost:~> docker version
Client:
 Version:    24.0.6-ce
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.11.2
    Path:     /usr/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  2.21.0
    Path:     /usr/lib/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 6
 Server Version: 24.0.6-ce
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 oci runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 091922f03c2762540fd057fba91260237ff86acb
 runc version: v1.1.9-0-gccaecfcbc907
 init version:
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.4.11-1-default
 Operating System: openSUSE Tumbleweed
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 7.747GiB
 Name: somename
 ID: someid
 Docker Root Dir: /somedir/dockered
 Debug Mode: false
 Username: someuser
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Thanks in advance!

2

Well, Docker is not supported officially on OpenSUSE, only on SLES (SUSE Linux Enterprise) and only on s390x architectures.

As for why only the rootless docker doesnā€™t work, I can only guess that Docker needs something to run which requires root privileges. Oe there is a proxy setting which is applied only for your rootless Docker. According to the error message, it tries to access a DNS server: 10.0.2.3:53. Is this an existing DNS server? Is 10.0.2.100 the IP of your host machine?

3

Thanks for your reply!

I checked the setting in ~/.docker/config.json, but there is nothing on a proxy specified there. The IP 10.0.2.3 is not part of my list fo DNS servers in /etc/resolve.conf . Furthermore, 10.0.2.100 is not the IP of my host.

Any further hints on that? Thank you in advance!

4

I would recommend an OpenSUSE forum, because I have no idea how they support Docker. I installed rootless docker now in an Ubuntu vm and works perfectly.

5

Good afternoon -

Iā€™m one of the staff over on the openSUSE forums, and am trying to help this individual figure out whatā€™s going on here. Unfortuately, like you, Iā€™m also not seeing any issues in my test envrionment running on openSUSE Tumbleweed - same as @tilfischer is using.

Weā€™re comparing configurations and coming up (so far) with nothing that materially changes the behavior on his system. It seems to be in the network layer - slirp4netns - but looking at the related running processes, Iā€™m not seeing a difference there either.

Do you have any advice on debugging tools or troubleshooting steps that could help further isolate the issue? It doesnā€™t really seem to be an openSUSE-specific issue from what Iā€™ve been able to determine, but Iā€™ll be the first to admit that my knowledge of slirp4netns is virtually nonexistent, and Iā€™ve never used rootless docker before (though I have used Docker extensively on openSUSE running as a non-user process, and never had any issues at all).

Thanks

6

Hi. I ran an opensuse tumbleweed virtual machine using LXD and I got timeout as well when I tried to pull an image.

Did you also use it on OpenSUSE Tumbleweed?

Edit:

For debugging you can try to enter the network namespace of the rootlesskit and try tshark as I suggested here for Docker Desktop: Docker network and network namespaces in practice - DEV Community

In case of rootless kit:

zypper install wireshark-cli
nsenter -n -t $(pidof rootlesskit | awk '{print $1}') tshark -i tap0

Then in another terminal:

docker pull hello-world

You will see some error messages. When you try tshark on the host, you will see that the host network doesnā€™t even see the traffic.

If you want to work with rootlesskits filesystem, you can try this:

nsenter --all -t $(pidof rootlesskit | awk '{print $1}') sh

But that way you wonā€™t be able to use a package manager or write the filesystem. It is for debugging.

7

Dear all,

the wireshark-cli package does exist but I did a zypper in wireshark. The command provided gave :

someuser@somehost:~> nsenter --all -t $(pidof rootlesskit | awk '{print $1}') sh
nsenter: reassociate to namespace 'ns/cgroup' failed: Operation not permitted

Hence, I did a sudo on that, did docker pull hello-world in the rootless docker in another terminal which gave:

root:/ # nsenter -n -t $(pidof rootlesskit | awk '{print $1}') tshark -i tap0
Running as user "root" and group "root". This could be dangerous.
Capturing on 'tap0'
 ** (tshark:30785) 11:57:04.367108 [Main MESSAGE] -- Capture started.
 ** (tshark:30785) 11:57:04.367160 [Main MESSAGE] -- File: "/tmp/wireshark_tap01WZVB2.pcapng"
    1 0.000000000   10.0.2.100 ā†’ 10.0.2.3     DNS 80 Standard query 0x2677 AAAA docker.io OPT
    2 0.000035190   10.0.2.100 ā†’ 10.0.2.3     DNS 80 Standard query 0x379d A docker.io OPT
    3 0.000088373     10.0.2.2 ā†’ 10.0.2.100   ICMP 108 Destination unreachable (Network unreachable)
    4 0.000105361     10.0.2.2 ā†’ 10.0.2.100   ICMP 108 Destination unreachable (Network unreachable)
    5 5.000204366   10.0.2.100 ā†’ 10.0.2.3     DNS 80 Standard query 0xff16 AAAA docker.io OPT
    6 5.000249245   10.0.2.100 ā†’ 10.0.2.3     DNS 80 Standard query 0xdbec A docker.io OPT
    7 5.000313829     10.0.2.2 ā†’ 10.0.2.100   ICMP 108 Destination unreachable (Network unreachable)
    8 5.000332154     10.0.2.2 ā†’ 10.0.2.100   ICMP 108 Destination unreachable (Network unreachable)
    9 5.189607761 06:fe:54:70:9c:47 ā†’ 52:55:0a:00:02:03 ARP 42 Who has 10.0.2.3? Tell 10.0.2.100
   10 5.189687759 52:55:0a:00:02:03 ā†’ 06:fe:54:70:9c:47 ARP 64 10.0.2.3 is at 52:55:0a:00:02:03
   11 10.003753439   10.0.2.100 ā†’ 10.0.2.3     DNS 91 Standard query 0x3fb7 AAAA registry-1.docker.io OPT
   12 10.003803592   10.0.2.100 ā†’ 10.0.2.3     DNS 91 Standard query 0x8b03 A registry-1.docker.io OPT
   13 10.003806775     10.0.2.2 ā†’ 10.0.2.100   ICMP 119 Destination unreachable (Network unreachable)
   14 10.003825573     10.0.2.2 ā†’ 10.0.2.100   ICMP 119 Destination unreachable (Network unreachable)
   15 15.003904272   10.0.2.100 ā†’ 10.0.2.3     DNS 91 Standard query 0xc66d A registry-1.docker.io OPT
   16 15.003911381   10.0.2.100 ā†’ 10.0.2.3     DNS 91 Standard query 0xce68 AAAA registry-1.docker.io OPT
   17 15.003958934     10.0.2.2 ā†’ 10.0.2.100   ICMP 119 Destination unreachable (Network unreachable)
   18 15.003978369     10.0.2.2 ā†’ 10.0.2.100   ICMP 119 Destination unreachable (Network unreachable)

Hope that helps.

Best,

8

My root install has run on Tumbleweed since I installed it over a year ago, and on Leap versions before that for years.

I tested the rootless installation in a fresh Tumbleweed installation in VMware Workstation and had no issues - Iā€™ve not been able to replicate this problem in that installation.

Running the commands you suggested in your edit results in my seeing the download go successfully. Like @tilfischer, I had to run the nsenter command as root (zypper also requires that):

jhenderson@localhost:~/.config/systemd/user> sudo nsenter -n -t $(pidof rootlesskit | awk '{print $1}') tshark -i tap0
Running as user "root" and group "root". This could be dangerous.
Capturing on 'tap0'
 ** (tshark:10077) 08:55:13.614401 [Main MESSAGE] -- Capture started.
 ** (tshark:10077) 08:55:13.614479 [Main MESSAGE] -- File: "/tmp/wireshark_tap0TURDC2.pcapng"
    1 0.000000000   10.0.2.100 ā†’ 10.0.2.3     DNS 80 Standard query 0xf64a A docker.io OPT
    2 0.000015357   10.0.2.100 ā†’ 10.0.2.3     DNS 80 Standard query 0x7bea AAAA docker.io OPT
    3 0.004197941     10.0.2.3 ā†’ 10.0.2.100   DNS 128 Standard query response 0xf64a A docker.io A 52.3.144.121 A 44.196.175.70 A 54.165.156.197 OPT
    4 0.010527897     10.0.2.3 ā†’ 10.0.2.100   DNS 164 Standard query response 0x7bea AAAA docker.io AAAA 2600:1f18:2148:bc00:a66d:e75:e647:fefc AAAA 2600:1f18:2148:bc02:34e0:261c:e99e:cf7d AAAA 2600:1f18:2148:bc01:6fba:f146:1387:f27d OPT
    5 0.011392525   10.0.2.100 ā†’ 10.0.2.3     DNS 91 Standard query 0xa585 AAAA registry-1.docker.io OPT
    6 0.011462041   10.0.2.100 ā†’ 10.0.2.3     DNS 91 Standard query 0x348e A registry-1.docker.io OPT
    7 0.014566621     10.0.2.3 ā†’ 10.0.2.100   DNS 175 Standard query response 0xa585 AAAA registry-1.docker.io AAAA 2600:1f18:2148:bc02:cfd8:db68:ea1f:277c AAAA 2600:1f18:2148:bc00:8334:ca86:c3d6:a507 AAAA 2600:1f18:2148:bc01:a3b0:6734:c617:7c5c OPT
    8 0.022464329     10.0.2.3 ā†’ 10.0.2.100   DNS 139 Standard query response 0x348e A registry-1.docker.io A 18.215.138.58 A 34.194.164.123 A 52.1.184.176 OPT
    9 0.022746893   10.0.2.100 ā†’ 18.215.138.58 TCP 74 51762 ā†’ 443 [SYN] Seq=0 Win=65480 Len=0 MSS=65480 SACK_PERM TSval=379494998 TSecr=0 WS=128
   10 0.111003318 18.215.138.58 ā†’ 10.0.2.100   TCP 58 443 ā†’ 51762 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=65480
   11 0.111046564   10.0.2.100 ā†’ 18.215.138.58 TCP 54 51762 ā†’ 443 [ACK] Seq=1 Ack=1 Win=65480 Len=0
   12 0.387856562   10.0.2.100 ā†’ 18.215.138.58 TLSv1 306 Client Hello
   13 0.388026730 18.215.138.58 ā†’ 10.0.2.100   TCP 54 443 ā†’ 51762 [ACK] Seq=1 Ack=253 Win=65535 Len=0
   14 0.475677479 18.215.138.58 ā†’ 10.0.2.100   TLSv1.3 5294 Server Hello, Change Cipher Spec, Application Data, Application Data
   15 0.475712996   10.0.2.100 ā†’ 18.215.138.58 TCP 54 51762 ā†’ 443 [ACK] Seq=253 Ack=5241 Win=60240 Len=0
[...]

This does seem to confirm that the issue is related to the rootless networking configuration, but Iā€™m really puzzled about why it works for me but not for you or him, when none of us did any configuration for the network at all.

ETA: I also ran nsenter with bash and checked a few things from a network perspective:

jhenderson@localhost:~/.config/systemd/user> sudo nsenter -n -t $(pidof rootlesskit | awk '{print $1}') bash
localhost:/home/jhenderson/.config/systemd/user # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host proto kernel_lo 
       valid_lft forever preferred_lft forever
2: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 0a:0f:44:a5:7e:7b brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.100/24 scope global tap0
       valid_lft forever preferred_lft forever
    inet6 fe80::80f:44ff:fea5:7e7b/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:1d:68:75:ab brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:1dff:fe68:75ab/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
localhost:/home/jhenderson/.config/systemd/user # route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.0.2.2        0.0.0.0         UG    0      0        0 tap0
10.0.2.0        0.0.0.0         255.255.255.0   U     0      0        0 tap0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
localhost:/home/jhenderson/.config/systemd/user #

Additionally, traceroute to 8.8.8.8 shows a proper route.

9

I used an LXD virtual machine. LXD supports multiple distributions and multiple variants of a distribution. For example there is a default, a cloud and a desktop variant usually. I guess the default installation is different or Iā€™m not installing Docker the same way. Since the documentation in Dockerā€™s documentation is about SLES (I still tried and didnā€™t work) and I didnā€™t find an official opensuse guide yet, I just ran zipper install docker-compose which installed Docker too and then installed docker-rootless-extras and ran dockerd-rootless-setuptool.sh install as a non-root user.

10

For my installation, I just installed docker, but the rootless-extras package was installed as part of that. I donā€™t have docker-compose installed, but that shouldnā€™t affect anything.

Like you, I ran dockerd-rootless-setuptool.sh install as a non-root user after making sure that the docker daemon was not enabled in the root context.

The instructions that we both followed were the ones at Run the Docker daemon as a non-root user (Rootless mode) | Docker Docs - Iā€™d never used rootless before (didnā€™t know it was even an option).

Iā€™ve never had to do anything special to get docker running on openSUSE. Iā€™ve never used LXD before, so I donā€™t know how that differs from running on bare metal or in VMwareā€™s hypervisor.

It seems the issue is one of connectivity within the network namespace - the packet captures seem to confirm that. Any thoughts on what might cause that? It seems really strange to me that from an address of 10.0.2.100 that 10.0.2.3 is unreachable. That points to some sort of network config issue, but as none of us have done explicit configuration of the network (it seems), Iā€™m not sure why weā€™d be seeing different results or where to go from there. If it is an issue in the package (which doesnā€™t make sense to me given that it works on my setup with no difference in configuration), I can open a bug in the openSUSE bugzilla and see if our maintainer can identify the issue, but reproducability is going to be important to a fix, if one is needed.

Iā€™ll pull a fresh Tumbleweed ISO and reinstall to see if I can duplicate that way. As a rolling release, TW updates keep it current with the most recent ISO, but Iā€™ll eliminate that as a possible issue.

11

I tried nsloookup in the rootlesskit network namespace, because my guess was that the name resolution is wrong, but it worked. Then I realized that the Network could work but the dns server is on the filesystem (mount namespace) of rootlesskit.

This works

nsenter -n -t $(pidof rootlesskit | awk '{print $1}') -- curl https://google.com

This doesnā€™t:

nsenter --all -t $(pidof rootlesskit | awk '{print $1}') -- curl https://google.com

So the issue is not the network only, but most likely the DNS server which is defined in /etc/resolv.conf

For example this works too:

nsenter --all -t $(pidof rootlesskit | awk '{print $1}') -- curl http://93.184.216.34

The IP is the IP of example.com

You could also report it in

because rootless Docker uses libnetwork.

12

Interesting. So on my VM, all three commands work.

Does your wireshark trace show the same thing as @tilfischerā€™s? With the responses in packets 3 and 4 showing ā€œdestination unreachableā€ for the DNS server?

If so, that would be a difference from what Iā€™m seeing. If the DNS server is in the filesystem and isnā€™t working for the two of you, but is for me, could it maybe be a filesystem permissions issue of some sort (grasping a little bit here as I donā€™t know how that all works).

Can you ping 10.0.2.3 (the DNS server) while in the network namespace and get a response?

13

Out of curiosity, in your Tumbleweed VM, are you using Wicked or NetworkManager to manage the VMā€™s network interface?

14

Yes, I got the same result too, but I could not share (copy-paste) the output then.

I donā€™t see how. I suspected firewall, but I couldnā€™t find any like firewalld on centos or ufw on Ubuntu. I also tried to find a security modul like apparmor or selinux, but I really donā€™t know much about OpenSUSE. SUSE was probably the first Linux I ever tried, but at that time I only cared about GUI.

/etc/resolv.conf is readable and I found 10.0.2.3 in it, but I donā€™t remember if I pinged that. I know I wanted to and I believe I did and worked, but Iā€™m not sure.

No idea. I never heard about Wicked. I will try the ping and checking the network later. Iā€™m not on that machine at the moment.

15

Sounds good. I also had suspected a firewall setting (though firewalld is ingress-only, so I couldnā€™t see how). Apparmor is installed by default, but doesnā€™t seem to be in effect on my setup.

I know the /etc/resolv.conf is set up within the network space automatically (so doesnā€™t match the hostā€™s settings), and it seems that that built-in DNS resolver forwards to the hostā€™s resolver. But in the userā€™s setup, the hostā€™s resolver is working fine, and it seems that the resolver on 10.0.2.3 is nonresponsive (causing the timeout).

My installation is a GNOME installation, but most use KDE (which I believe uses Wicked rather than NetworkManager), so Iā€™m doing a fresh KDE installation to give that a try. I did try switching to Wicked in the GNOME desktop VM, but that didnā€™t change my setupā€™s behavior, so Iā€™m not expecting anything different to happen, but weā€™ll see.

16

I just did a completely fresh Tumbleweed installation (KDE desktop this time - which does use NetworkManager as well - I was mistaken about that). Post-installation, I ran:

sudo zypper in docker
sudo systemctl stop docker; sudo systemctl disable docker
/usr/bin/dockerd-rootless-setuptool.sh install
docker ps
docker pull hello-world
docker run -it --rm hello-world

And everything still worked. Installation settings were to just select the KDE Desktop installation - nothing special configured.

Hereā€™s the full output:

jhenderson@localhost:~> sudo zypper in docker
Loading repository data...
Reading installed packages...
Resolving package dependencies...

The following 4 recommended packages were automatically selected:
  criu docker-buildx docker-rootless-extras git-core

The following 17 NEW packages are going to be installed:
  catatonit containerd criu docker docker-bash-completion docker-buildx docker-rootless-extras fuse-overlayfs
  git-core libnet9 libsha1detectcoll1 libslirp0 python311-ipaddr python311-protobuf rootlesskit runc
  slirp4netns

17 new packages to install.
Overall download size: 76.4 MiB. Already cached: 0 B. After the operation, additional 294.3 MiB will be used.
Continue? [y/n/v/...? shows all options] (y): 
Retrieving: catatonit-0.1.7-1.7.x86_64 (Main Repository (OSS))                           (1/17), 301.8 KiB    
Retrieving: catatonit-0.1.7-1.7.x86_64.rpm ................................................[done (69.0 KiB/s)]
Retrieving: fuse-overlayfs-1.12-1.2.x86_64 (Main Repository (OSS))                       (2/17),  62.5 KiB    
Retrieving: fuse-overlayfs-1.12-1.2.x86_64.rpm .............................................[done (1.2 MiB/s)]
Retrieving: libnet9-1.2-3.9.x86_64 (Main Repository (OSS))                               (3/17),  47.7 KiB    
Retrieving: libnet9-1.2-3.9.x86_64.rpm .....................................................[done (2.5 KiB/s)]
Retrieving: libsha1detectcoll1-1.0.3-4.20.x86_64 (Main Repository (OSS))                 (4/17),  25.3 KiB    
Retrieving: libsha1detectcoll1-1.0.3-4.20.x86_64.rpm ...................................................[done]
Retrieving: libslirp0-4.7.0+44-3.3.x86_64 (Main Repository (OSS))                        (5/17),  77.7 KiB    
Retrieving: libslirp0-4.7.0+44-3.3.x86_64.rpm ............................................[done (226.0 KiB/s)]
Retrieving: python311-ipaddr-2.2.0-1.20.noarch (Main Repository (OSS))                   (6/17),  44.9 KiB    
Retrieving: python311-ipaddr-2.2.0-1.20.noarch.rpm .....................................................[done]
Retrieving: python311-protobuf-4.23.4-6.3.x86_64 (Main Repository (OSS))                 (7/17), 335.1 KiB    
Retrieving: python311-protobuf-4.23.4-6.3.x86_64.rpm .....................................[done (916.7 KiB/s)]
Retrieving: rootlesskit-1.1.1-1.2.x86_64 (Main Repository (OSS))                         (8/17),   5.4 MiB    
Retrieving: rootlesskit-1.1.1-1.2.x86_64.rpm ...............................................[done (3.5 MiB/s)]
Retrieving: git-core-2.42.0-2.1.x86_64 (Main Repository (OSS))                           (9/17),   5.3 MiB    
Retrieving: git-core-2.42.0-2.1.x86_64.rpm .................................................[done (3.5 MiB/s)]
Retrieving: slirp4netns-1.2.1-1.1.x86_64 (Main Repository (OSS))                        (10/17),  47.2 KiB    
Retrieving: slirp4netns-1.2.1-1.1.x86_64.rpm ...........................................................[done]
Retrieving: criu-3.18-1.3.x86_64 (Main Repository (OSS))                                (11/17), 695.5 KiB    
Retrieving: criu-3.18-1.3.x86_64.rpm .....................................................[done (575.9 KiB/s)]
Retrieving: runc-1.1.9-1.1.x86_64 (Main Repository (OSS))                               (12/17),   3.0 MiB    
Retrieving: runc-1.1.9-1.1.x86_64.rpm ......................................................[done (3.0 MiB/s)]
Retrieving: containerd-1.7.6-1.1.x86_64 (Main Repository (OSS))                         (13/17),  20.5 MiB    
Retrieving: containerd-1.7.6-1.1.x86_64.rpm ...............................................[done (12.7 MiB/s)]
Retrieving: docker-24.0.6_ce-1.1.x86_64 (Main Repository (OSS))                         (14/17),  27.7 MiB    
Retrieving: docker-24.0.6_ce-1.1.x86_64.rpm ...............................................[done (12.8 MiB/s)]
Retrieving: docker-buildx-0.11.2-1.2.x86_64 (Main Repository (OSS))                     (15/17),  12.8 MiB    
Retrieving: docker-buildx-0.11.2-1.2.x86_64.rpm ...........................................[done (11.4 MiB/s)]
Retrieving: docker-rootless-extras-24.0.6_ce-1.1.noarch (Main Repository (OSS))         (16/17),  30.4 KiB    
Retrieving: docker-rootless-extras-24.0.6_ce-1.1.noarch.rpm ................................[done (2.8 KiB/s)]
Retrieving: docker-bash-completion-24.0.6_ce-1.1.noarch (Main Repository (OSS))         (17/17),  42.3 KiB    
Retrieving: docker-bash-completion-24.0.6_ce-1.1.noarch.rpm ............................................[done]

Checking for file conflicts: ...........................................................................[done]
( 1/17) Installing: catatonit-0.1.7-1.7.x86_64 .........................................................[done]
( 2/17) Installing: fuse-overlayfs-1.12-1.2.x86_64 .....................................................[done]
( 3/17) Installing: libnet9-1.2-3.9.x86_64 .............................................................[done]
( 4/17) Installing: libsha1detectcoll1-1.0.3-4.20.x86_64 ...............................................[done]
( 5/17) Installing: libslirp0-4.7.0+44-3.3.x86_64 ......................................................[done]
( 6/17) Installing: python311-ipaddr-2.2.0-1.20.noarch .................................................[done]
( 7/17) Installing: python311-protobuf-4.23.4-6.3.x86_64 ...............................................[done]
( 8/17) Installing: rootlesskit-1.1.1-1.2.x86_64 .......................................................[done]
( 9/17) Installing: git-core-2.42.0-2.1.x86_64 .........................................................[done]
(10/17) Installing: slirp4netns-1.2.1-1.1.x86_64 .......................................................[done]
(11/17) Installing: criu-3.18-1.3.x86_64 ...............................................................[done]
(12/17) Installing: runc-1.1.9-1.1.x86_64 ..............................................................[done]
(13/17) Installing: containerd-1.7.6-1.1.x86_64 ........................................................[done]
/usr/bin/systemd-sysusers --replace=/usr/lib/sysusers.d/docker.conf -
Creating group 'docker' with GID 460.
Creating group 'dockremap' with GID 459.
Creating user 'dockremap' (docker --userns-remap=default) with UID 459 and GID 459.
Updating /etc/sysconfig/docker ...
(14/17) Installing: docker-24.0.6_ce-1.1.x86_64 ........................................................[done]
(15/17) Installing: docker-buildx-0.11.2-1.2.x86_64 ....................................................[done]
(16/17) Installing: docker-rootless-extras-24.0.6_ce-1.1.noarch ........................................[done]
(17/17) Installing: docker-bash-completion-24.0.6_ce-1.1.noarch ........................................[done]
jhenderson@localhost:~> sudo systemctl stop docker; sudo systemctl disable docker
jhenderson@localhost:~> /usr/bin/dockerd-rootless-setuptool.sh install
[INFO] Creating /home/jhenderson/.config/systemd/user/docker.service
[INFO] starting systemd service docker.service
+ systemctl --user start docker.service
+ sleep 3
+ systemctl --user --no-pager --full status docker.service
ā— docker.service - Docker Application Container Engine (Rootless)
     Loaded: loaded (/home/jhenderson/.config/systemd/user/docker.service; disabled; preset: disabled)
     Active: active (running) since Fri 2023-09-29 10:59:53 PDT; 3s ago
       Docs: https://docs.docker.com/go/rootless/
   Main PID: 3441 (rootlesskit)
      Tasks: 31
     Memory: 59.6M
        CPU: 275ms
     CGroup: /user.slice/user-1000.slice/user@1000.service/app.slice/docker.service
             ā”œā”€3441 rootlesskit --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --propagation=rslave /usr/bin/dockerd-rootless.sh
             ā”œā”€3451 /proc/self/exe --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --propagation=rslave /usr/bin/dockerd-rootless.sh
             ā”œā”€3472 slirp4netns --mtu 65520 -r 3 --disable-host-loopback --enable-sandbox --enable-seccomp 3451 tap0
             ā”œā”€3479 dockerd
             ā””ā”€3494 containerd --config /run/user/1000/docker/containerd/containerd.toml

Sep 29 10:59:53 localhost.localdomain dockerd-rootless.sh[3479]: time="2023-09-29T10:59:53.910535052-07:00" level=warning msg="WARNING: No io.max (rbps) support"
Sep 29 10:59:53 localhost.localdomain dockerd-rootless.sh[3479]: time="2023-09-29T10:59:53.910566410-07:00" level=warning msg="WARNING: No io.max (wbps) support"
Sep 29 10:59:53 localhost.localdomain dockerd-rootless.sh[3479]: time="2023-09-29T10:59:53.910596516-07:00" level=warning msg="WARNING: No io.max (riops) support"
Sep 29 10:59:53 localhost.localdomain dockerd-rootless.sh[3479]: time="2023-09-29T10:59:53.910634070-07:00" level=warning msg="WARNING: No io.max (wiops) support"
Sep 29 10:59:53 localhost.localdomain dockerd-rootless.sh[3479]: time="2023-09-29T10:59:53.910663853-07:00" level=warning msg="WARNING: bridge-nf-call-iptables is disabled"
Sep 29 10:59:53 localhost.localdomain dockerd-rootless.sh[3479]: time="2023-09-29T10:59:53.910692912-07:00" level=warning msg="WARNING: bridge-nf-call-ip6tables is disabled"
Sep 29 10:59:53 localhost.localdomain dockerd-rootless.sh[3479]: time="2023-09-29T10:59:53.910741606-07:00" level=info msg="Docker daemon" commit=1a7969545d73 graphdriver=overlay2 version=24.0.6-ce
Sep 29 10:59:53 localhost.localdomain dockerd-rootless.sh[3479]: time="2023-09-29T10:59:53.910844024-07:00" level=info msg="Daemon has completed initialization"
Sep 29 10:59:53 localhost.localdomain dockerd-rootless.sh[3479]: time="2023-09-29T10:59:53.930389269-07:00" level=info msg="API listen on /run/user/1000/docker.sock"
Sep 29 10:59:53 localhost.localdomain systemd[1387]: Started Docker Application Container Engine (Rootless).
+ DOCKER_HOST=unix:///run/user/1000/docker.sock
+ /usr/bin/docker version
Client:
 Version:           24.0.6-ce
 API version:       1.43
 Go version:        go1.20.8
 Git commit:        1a7969545d73
 Built:             Thu Sep 14 00:00:00 2023
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          24.0.6-ce
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.8
  Git commit:       1a7969545d73
  Built:            Thu Sep 14 00:00:00 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.6
  GitCommit:        091922f03c2762540fd057fba91260237ff86acb
 runc:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfcbc907
 docker-init:
  Version:          0.1.7_catatonit
  GitCommit:        
 rootlesskit:
  Version:          1.1.1
  ApiVersion:       1.1.1
  NetworkDriver:    slirp4netns
  PortDriver:       builtin
  StateDir:         /tmp/rootlesskit1745771322
 slirp4netns:
  Version:          1.2.1
  GitCommit:        unknown
+ systemctl --user enable docker.service
Created symlink /home/jhenderson/.config/systemd/user/default.target.wants/docker.service ā†’ /home/jhenderson/.config/systemd/user/docker.service.
[INFO] Installed docker.service successfully.
[INFO] To control docker.service, run: `systemctl --user (start|stop|restart) docker.service`
[INFO] To run docker.service on system startup, run: `sudo loginctl enable-linger jhenderson`

[INFO] Creating CLI context "rootless"
Successfully created context "rootless"
[INFO] Using CLI context "rootless"
Current context is now "rootless"

[INFO] Make sure the following environment variable(s) are set (or add them to ~/.bashrc):
export PATH=/usr/bin:$PATH

[INFO] Some applications may require the following environment variable too:
export DOCKER_HOST=unix:///run/user/1000/docker.sock

jhenderson@localhost:~> docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
jhenderson@localhost:~> docker pull hello-world
Using default tag: latest
latest: Pulling from library/hello-world
719385e32844: Pull complete 
Digest: sha256:4f53e2564790c8e7856ec08e384732aa38dc43c52f02952483e3f003afbf23db
Status: Downloaded newer image for hello-world:latest
docker.io/library/hello-world:latest
jhenderson@localhost:~> docker run -it --rm hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

jhenderson@localhost:~>

Iā€™m stumped as to why itā€™s not working for either of you. This is a fresh install of Tumbleweed 20230926. Maybe the full output above will provide some clues as a point of comparison.

17 
cat /etc/os-release

NAME="openSUSE Tumbleweed"
# VERSION="20230926"
ID="opensuse-tumbleweed"
ID_LIKE="opensuse suse"
VERSION_ID="20230926"
PRETTY_NAME="openSUSE Tumbleweed"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:tumbleweed:20230926"
BUG_REPORT_URL="https://bugzilla.opensuse.org"
SUPPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org"
DOCUMENTATION_URL="https://en.opensuse.org/Portal:Tumbleweed"
LOGO="distributor-logo-Tumbleweed"

It uses wicked for network according to the process list

root         1  0.6  0.3  21984 13464 ?        Ss   18:30   0:02 /usr/lib/systemd/systemd --switched-root --system --deserialize=31
root       508  0.0  0.3  50580 15416 ?        Ss   18:30   0:00 /usr/lib/systemd/systemd-journald
root       553  0.2  0.6 723924 25112 ?        Ssl  18:30   0:00 /run/incus_agent/incus-agent
root      1036  0.0  0.1   5876  4608 pts/0    Ss   18:31   0:00  \_ bash
root      1350  5.0  0.0   8016  3584 pts/0    R+   18:35   0:00  |   \_ ps auxf
root      1062  0.0  0.1   5876  4608 pts/1    Ss   18:34   0:00  \_ bash
root      1086  0.0  0.1   7192  4224 pts/1    S    18:34   0:00      \_ su - ta
ta        1098  0.0  0.1   5564  4352 pts/1    S+   18:34   0:00          \_ -bash
root       564  0.0  0.2  33764  9728 ?        Ss   18:30   0:00 /usr/lib/systemd/systemd-udevd
message+   629  0.0  0.1   8780  4736 ?        Ss   18:30   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root       639  0.0  0.2  18248  8832 ?        Ss   18:30   0:00 /usr/lib/systemd/systemd-logind
root       710  0.0  0.1  10112  6144 ?        SLs  18:30   0:00 /usr/libexec/wicked/bin/wickedd-auto4 --systemd --foreground
root       711  0.0  0.1  10116  6272 ?        SLs  18:30   0:00 /usr/libexec/wicked/bin/wickedd-dhcp4 --systemd --foreground
root       712  0.0  0.1  10116  6272 ?        SLs  18:30   0:00 /usr/libexec/wicked/bin/wickedd-dhcp6 --systemd --foreground
root       714  0.0  0.1  10232  6528 ?        SLs  18:30   0:00 /usr/sbin/wickedd --systemd --foreground
root       718  0.0  0.1  10140  6656 ?        SLs  18:30   0:00 /usr/sbin/wickedd-nanny --systemd --foreground
root      1032  0.0  0.0   3464  2048 tty1     Ss+  18:30   0:00 /sbin/agetty -o -p -- \u --noclear - linux
root      1033  0.0  0.0   3508  2176 ttyS0    Ss+  18:30   0:00 /sbin/agetty -o -p -- \u --keep-baud 115200,57600,38400,9600 - vt220
ta        1089  0.1  0.2  20752 11648 ?        Ss   18:34   0:00 /usr/lib/systemd/systemd --user
ta        1090  0.0  0.1  23216  5196 ?        S    18:34   0:00  \_ (sd-pam)
ta        1097  0.1  0.4 1752940 16092 ?       Ssl  18:34   0:00  \_ rootlesskit --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-
ta        1134  0.0  0.3 1679164 12604 ?       Sl   18:34   0:00      \_ /proc/self/exe --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback 
ta        1166  1.2  2.0 1463472 82508 ?       Sl   18:34   0:00      |   \_ dockerd
ta        1183  0.5  1.2 1800908 50264 ?       Ssl  18:34   0:00      |       \_ containerd --config /run/user/1000/docker/containerd/containerd.toml
ta        1158  0.0  0.0   5404  2944 ?        S    18:34   0:00      \_ slirp4netns --mtu 65520 -r 3 --disable-host-loopback --enable-sandbox --enable-seccomp 1134 tap0

Yes, worked.

nsenter --all -t $(pidof rootlesskit | awk '{print $1}') -- ping 10.0.2.3

PING 10.0.2.3 (10.0.2.3) 56(84) bytes of data.
64 bytes from 10.0.2.3: icmp_seq=1 ttl=255 time=0.175 ms
64 bytes from 10.0.2.3: icmp_seq=2 ttl=255 time=0.374 ms
18

Was the installation a pre-installed image in the lxd repos, or a fresh install from DVD? If the former, which image did you use?

Very strange that you can ping the address, but it is giving a timeout on name resolution.

19

pre-installed

lxc launch images:opensuse/tumbleweed opensuse  --vm -c limits.cpu=2 -c liits.memory=4GiB

The behaviour of ping is not really strange, but I was confused as well. We tried to ping the DNS server but thatā€™s not the problem. If I now interpret it correctly, the ICMP packages from 10.0.2.2 to 10.0.2.100 are failing. So we should actually ping the host of the rootlesskit namespace from the gateway, which is 10.0.2.2 and I donā€™t know how I could do that.

20

I had seen it as a failed response from the DNS server, but youā€™re right, itā€™s not that (not directly, in any case), but seems to be a ping response from the gateway back to the local tap0 interfaceā€™s address.

Are you able to ping the gateway from inside the namespace? It would be particularly strange if you could ping it, but the gateway couldnā€™t ping back.