Rootless docker: i/o timeout with docker pull

[hendersj (Jim Henderson)]

I wonder what either of you sees within the rootlesskit namespace when you run iptables -L. Here’s what I see:

target     prot opt source               destination         

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (1 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere     

That ping response kinda feels like ICMP is just being dropped, and while in the host there’s probably nothing defined, the setup inside the namespace is different. In the host, I see:

jhenderson@localhost:~> iptables -L
Absolute path to 'iptables' is '/usr/sbin/iptables', so running it may require superuser privileges (eg. root).
jhenderson@localhost:~> sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

[tilfischer (tfischer)]

Dear all,

thank you all so much for help in troubleshooting this issue! I will to through (I have KDE) all of your posts and post what I got:

someuser@somehost:~/.config/systemd/user> sudo nsenter -n -t $(pidof rootlesskit | awk '{print $1}') bash
homehost:/home/someuser/.config/systemd/user # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host proto kernel_lo 
       valid_lft forever preferred_lft forever
2: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 06:fe:54:70:9c:47 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.100/24 scope global tap0
       valid_lft forever preferred_lft forever
    inet6 fe80::4fe:54ff:fe70:9c47/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:76:d7:95:4a brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
4: br-94190f408626: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:84:29:49:5b brd ff:ff:ff:ff:ff:ff
    inet 172.25.0.1/16 brd 172.25.255.255 scope global br-94190f408626
       valid_lft forever preferred_lft forever
    inet6 fe80::42:84ff:fe29:495b/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
74: vethc5a41ad@if73: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-94190f408626 state UP group default 
    link/ether ca:75:53:a1:fe:c9 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::c875:53ff:fea1:fec9/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
76: veth8a03f8c@if75: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-94190f408626 state UP group default 
    link/ether c6:de:e4:50:c4:3b brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::c4de:e4ff:fe50:c43b/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
78: vethc17f928@if77: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-94190f408626 state UP group default 
    link/ether 72:9a:43:22:83:e9 brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::709a:43ff:fe22:83e9/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
someuser@somehost:~/.config/systemd/user> route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         n9***** 0.0.0.0         UG    0      0        0 ens192
1**.**.**.**    *               255.255.254.0   U     0      0        0 ens192

Thats interesting, as it does not show the tap0 or docker0 interface.

I could not run nsenter as user (permission denied) but had to sudo it:

someuser@someuser:~/.config/systemd/user> sudo nsenter -n -t $(pidof rootlesskit | awk '{print $1}') -- curl https://google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>
somehost@someuser:~/.config/systemd/user> sudo nsenter --all -t $(pidof rootlesskit | awk '{print $1}') -- curl https://google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>
someuser@somehost:~/.config/systemd/user> sudo nsenter --all -t $(pidof rootlesskit | awk '{print $1}') -- curl http://93.184.216.34
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
	<head>
		<title>404 - Not Found</title>
	</head>
	<body>
		<h1>404 - Not Found</h1>
	</body>
</html>

Ping 10.0.2.3:

someuser@somehost:~/.config/systemd/user> ping 10.0.2.3
PING 10.0.2.3 (10.0.2.3) 56(84) bytes of data.
^C
--- 10.0.2.3 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4059ms

someuser@somehost:~/.config/systemd/user> nsenter --all -t $(pidof rootlesskit | awk '{print $1}') -- ping 10.0.2.3
nsenter: reassociate to namespace 'ns/cgroup' failed: Operation not permitted
someuser@somehost:~/.config/systemd/user> sudo nsenter --all -t $(pidof rootlesskit | awk '{print $1}') -- ping 10.0.2.3
PING 10.0.2.3 (10.0.2.3) 56(84) bytes of data.
64 bytes from 10.0.2.3: icmp_seq=1 ttl=255 time=0.070 ms
64 bytes from 10.0.2.3: icmp_seq=2 ttl=255 time=0.051 ms
64 bytes from 10.0.2.3: icmp_seq=3 ttl=255 time=0.058 ms
^C
--- 10.0.2.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2014ms
rtt min/avg/max/mdev = 0.051/0.059/0.070/0.007 ms

someuser@somehost:~/.config/systemd/user> iptables -L
Fatal: can't open lock file /run/xtables.lock: Permission denied
someuser@somehost:~/.config/systemd/user> sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination  

Hope that helps.

Best

[hendersj (Jim Henderson)]

Did you also have a chance to run iptables -L inside the nsenter command environment, as well as the route command? Those look to be external to that environment.

[hendersj (Jim Henderson)]

@rimelek

The behaviour of ping is not really strange, but I was confused as well. We tried to ping the DNS server but that’s not the problem. If I now interpret it correctly, the ICMP packages from 10.0.2.2 to 10.0.2.100 are failing. So we should actually ping the host of the rootlesskit namespace from the gateway, which is 10.0.2.2 and I don’t know how I could do that.

I was thinking this was maybe an ICMP response packet from a ping from .100 to .2, but looking at it with fresh eyes this morning, it seems to just be a normal ICMP. So while the “no route to host” message makes no sense for hosts on the same network. I would just expect to see the ping request go out and there to be no response, not a device saying “I don’t know the route to this destination” since there’s no need for the traffic to be routed.

[tilfischer (tfischer)]

Yes, here we are:

someuser@somehost:~/bin> sudo nsenter -n -t $(pidof rootlesskit | awk '{print $1}') bash
somehost:/ # iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (2 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             172.25.0.5           tcp dpt:terabase

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere 

[hendersj (Jim Henderson)]

Looks like a few minor differences to what’s in mine, but nothing that sticks out as potentially problematic.

How about the route command in that context as well?

[hendersj (Jim Henderson)]

I’ve managed to reproduce the issue, finally, by using the lxd image.

Bizarrely, I’ve still not managed to reproduce it in my VMware image, but in an lxd image inside the very same VM, I’ve duplicated it.

I tried something that it occurred to me we hadn’t tried - I traced traffic with tshark both inside the namespace network using nsenter, and on the lxd host (ie, my vmware VM; nested VMs are a challenge to refer to).

What I expected to see was the DNS request for registry-1.docker.io to be sent and a response to come back, and to see a failure inside the lxd VM. Instead, what I saw was no traffic at all outside the namespace’s network.

So it seems traffic is not leaving that network at all.

Within slirp4netns, how is traffic supposed to flow from DNS lookups at 10.0.2.3?

[tilfischer (tfischer)]

someuser@somehost:~> route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         1xx.xx.247.254  0.0.0.0         UG    0      0        0 ens192
1xx.xx.246.0    0.0.0.0         255.255.254.0   U     0      0        0 ens192

someuser@somehost:~> sudo nsenter -n -t $(pidof rootlesskit | awk '{print $1}') bash
[sudo] password for root: 
somehost:/ # route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.0.2.2        0.0.0.0         UG    0      0        0 tap0
10.0.2.0        0.0.0.0         255.255.255.0   U     0      0        0 tap0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.25.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-94190f408626

Compared to:

localhost:/home/jhenderson/.config/systemd/user # route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.0.2.2        0.0.0.0         UG    0      0        0 tap0
10.0.2.0        0.0.0.0         255.255.255.0   U     0      0        0 tap0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0

[tilfischer (tfischer)]

I have found a solution to my issue by reading the slirp4netns documentation, which says:

Requires /etc/resolv.conf not to be a symlink to a file outside /etc and /run.

I did the following:

someuser@somehost:/etc> ls -l resolv.conf 
lrwxrwxrwx 1 root root 30 11. Apr 2019  resolv.conf -> /var/run/netconfig/resolv.conf
someuser@somehost:~> cp /etc/resolv.conf /somefolder/
someuser@somehost:~> sudo rm /etc/resolv.conf 
someuser@somehost:~> sudo cp /somefolder/resolv.conf /etc/resolv.conf 
someuser@somehost:~> ls -l /etc/resolv.conf 
-rw-r--r-- 1 root root 665 Oct  1 18:21 /etc/resolv.conf
someuser@somehost:~> systemctl --user daemon-reload
someuser@somehost:~> systemctl --user restart docker

Tried to pull an image:

someuser@somehost:~> docker pull hello-world
Using default tag: latest
latest: Pulling from library/hello-world
719385e32844: Pull complete 
Digest: sha256:4f53e2564790c8e7856ec08e384732aa38dc43c52f02952483e3f003afbf23db
Status: Downloaded newer image for hello-world:latest
docker.io/library/hello-world:latest

Works. However, in openSUSE DNS servers are specified in /etc/sysconfig/network/config , which generated the resolve.conf which is then symlinked to /etc/resolve.conf .

Is this a bug report for the openSUSE maintainers or for the maintainers of slirp4netns? I started with an issue to slirp4netns.

Best,

[hendersj (Jim Henderson)]

That is excellent - given that it works for rimelek on a non-openSUSE distribution, I’d suggest submitting a bug to bugzilla.opensuse.org - and see if they suggest pushing the report upstream.

Glad to hear you found a solution! I’d noticed that in the docs as well, but it didn’t register that that might be the issue. I was playing last night with it a bit and trying to run dig @8.8.8.8 registry-1.docker.io inside the userspace, and it was throwing an error I couldn’t figure out. When I got up this morning, I was going to dig into that (pun really not intended! ) into that a bit more.

[tilfischer (tfischer)]

Dear Jim,

thank you for your support on that! I will submit this to bugzilla.opensuse.org .

Best

[rimelek (Ákos Takács)]

Could you share the link to the submitted ticket? There might be a related issue. I’m not sure, I didn’t have time to investigate, but the link could help @lovestha in the following topic.

DNS settings ignored?

I’m having trouble getting dockers to have valid DNS settings. gareth@cube-server ~/d/qbittorrent> docker --version Docker version 24.0.6-ce, build 1a7969545d73 gareth@cube-server ~/d/qbittorrent> docker-compose --version Docker Compose version 2.22.0 gareth@cube-server ~/d/qbittorrent> cat docker-compose.yml --- version: "2.1" services: qbittorrent: image: lscr.io/linuxserver/qbittorrent:latest container_name: qbittorrent environment: - PUID=1000 - PGID=100 - T…

EDIT:

Turned out to be a different issue, but the link could still be useful to anyone

DNS settings ignored?

^ is the solution, sorry I didn’t mention the first DNS server being on the same host, that didn’t occur to me as relevant.

[tilfischer (tfischer)]

OpenSUSE: 1215853 – Cannot pull images with rootless docker on openSUSE Tumbleweed due to symlinked /etc/resolve.conf

The issue with slirp4netns was closed, as this seemed to be a OpenSUSE issue rather than a slirp4netns issue.