DOCKER-USER chain for ip6tables

127156 · January 8, 2023, 4:06am

Our system has some iptables firewall rules on DOCKER-USER chain. Now I want to see if we can add similar rules with ip6tables. But I do not see both DOCKER and DOCKER-USER chains for ip6tables. Is there a way to achieve this with ip6tables?

The docker version is 19.03.8-ce.

Thanks.

127156 · January 10, 2023, 10:46pm

Maybe I did not make my question clearer. Below are the command outputs:

vm12:~# ip6tables -S | grep DOCKER-USER

vm12:~# iptables -S | grep DOCKER-USER
-N DOCKER-USER
-A FORWARD -j DOCKER-USER
-A DOCKER-USER -j RETURN

Can some docker expert answer my question?

Thanks.

rimelek · January 10, 2023, 11:59pm

I have read your question 3 days ago and it was clear to me, but don’t use IPv6 for containers so I was waiting for someone who uses IPv6. Now I checked the documentation and also searched for “ipv6 iptables docker” and I found that ipv6 iptables rules are not enabled by default and that is still an experimental feature.

--iptables          Enable addition of iptables rules (default true)
--ip6tables         Enable addition of ip6tables rules (default false)

When you enable ip6tables you also need

--experimental      Enable experimental features

Or in the daemon json

{
    "experimental": true,
    "ipv6": true,
    "ip6tables": true,
    "fixed-cidr-v6": "fd9e:63ac:6dcd::/48"
}

My first source where I found this json before checking the docs:

github.com/moby/moby

Docker uses 'iptables' instead of 'ip6tables' for IPv6 NAT rule, crashes

opened 09:28PM - 06 Jan 21 UTC

closed 04:43PM - 29 Mar 21 UTC

fnkr

kind/bug area/networking version/20.10

**Description** I'm trying to enable IPv6 NAT for the default bridge network. If the `ip6tables` daemon option is enabled, Docker tries to add an IPv6 NAT rule using `iptables`, which fails because `ip6tables` should be used instead. Related to https://github.com/moby/moby/pull/41622. **Steps to reproduce the issue:** 1. Update `/etc/docker/daemon.json`: ```json { "experimental": true, "ipv6": true, "ip6tables": true, "fixed-cidr-v6": "fd9e:63ac:6dcd::/48" } ``` 2. Restart Docker: `sudo systemctl stop docker; sudo ip link del docker0; sudo systemctl start docker` 3. Docker daemon crashed during start. ``` # sudo journalctl -u docker dockerd[11587]: failed to start daemon: Error initializing network controller: Error creating default "bridge" network: Failed to Setup IP tables: Unable to enable NAT rule: (COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -I POSTROUTING -s fd9e:63ac:6dcd::/48 ! -o docker0 -j MASQUERADE' failed: iptables v1.8.5 (legacy): invalid mask `48' specified ``` **Describe the results you received:** Docker tries to run the following command, which fails and Docker crashes: ``` /usr/sbin/iptables -w10 -t nat -I POSTROUTING -s fd9e:63ac:6dcd::/48 ! -o docker0 -j MASQUERADE ``` **Describe the results you expected:** Docker should run the command using `ip6tables` instead: ``` /usr/sbin/ip6tables -w10 -t nat -I POSTROUTING -s fd9e:63ac:6dcd::/48 ! -o docker0 -j MASQUERADE ``` **Additional information you deem important (e.g. issue happens only occasionally):** If I remove the `ip6tables` option from `daemon.json`, restart Docker and run the fixed command from above (using `ip6tables`), IPv6 NAT works: ``` ➜ docker run --rm -it alpine ping -c 4 2606:4700:4700::1111 PING 2606:4700:4700::1111 (2606:4700:4700::1111): 56 data bytes 64 bytes from 2606:4700:4700::1111: seq=0 ttl=63 time=5.24 ms 64 bytes from 2606:4700:4700::1111: seq=1 ttl=63 time=5.40 ms 64 bytes from 2606:4700:4700::1111: seq=2 ttl=63 time=5.44 ms 64 bytes from 2606:4700:4700::1111: seq=3 ttl=63 time=5.46 ms --- fde0:8899:8672:1::1 ping statistics --- 4 packets transmitted, 4 packets received, 0% packet loss round-trip min/avg/max = 5.236/5.381/5.456 ms ``` **Output of `docker version`:** ``` Client: Docker Engine - Community Version: 20.10.2 API version: 1.41 Go version: go1.13.15 Git commit: 2291f61 Built: Mon Dec 28 16:18:35 2020 OS/Arch: linux/amd64 Context: default Experimental: true Server: Docker Engine - Community Engine: Version: 20.10.2 API version: 1.41 (minimum version 1.12) Go version: go1.13.15 Git commit: 8891c58 Built: Mon Dec 28 16:15:44 2020 OS/Arch: linux/amd64 Experimental: true containerd: Version: 1.4.3 GitCommit: 269548fa27e0089a8b8278fc4fc781d7f65a939b runc: Version: 1.0.0-rc92 GitCommit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff docker-init: Version: 0.19.0 GitCommit: de40ad0 ``` **Output of `docker info`:** ``` Client: Context: default Debug Mode: false Plugins: app: Docker App (Docker Inc., v0.9.1-beta3) buildx: Build with BuildKit (Docker Inc., v0.5.1-docker) Server: Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 13 Server Version: 20.10.2 Storage Driver: btrfs Build Version: Btrfs v5.9 Library Version: 102 Logging Driver: json-file Cgroup Driver: systemd Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc Default Runtime: runc Init Binary: docker-init containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff init version: de40ad0 Security Options: seccomp Profile: default cgroupns Kernel Version: 5.9.16-200.fc33.x86_64 Operating System: Fedora 33 (Workstation Edition) OSType: linux Architecture: x86_64 CPUs: 8 Total Memory: 39.07GiB Name: redacted ID: S5VU:MHDD:7D2V:YPGW:IOU2:AUQG:DMYS:IJ3Y:IRZD:5MWB:DBV4:PRR2 Docker Root Dir: /var/lib/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: true Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false Default Address Pools: Base: 100.96.0.0/16, Size: 24 WARNING: No kernel memory TCP limit support WARNING: No oom kill disable support WARNING: Support for cgroup v2 is experimental ``` **Additional environment details (AWS, VirtualBox, physical, etc.):** Physical (Notebook)

127156 · January 11, 2023, 12:07am

I saw this article before I posted my question:

Does this mean we need to upgrade our docker engine to 20.10, right?

rimelek · January 11, 2023, 12:08am

Then it looks like it does.

127156 · January 11, 2023, 12:10am

Thanks. Will try to upgrade and see.

cf3005 · January 22, 2023, 1:49pm

I am on docker 20.10 and the different docker chain appear in ip6tables but I doesn’t have the DOCKER-USER

docker version :

Client: Docker Engine - Community
 Version:           20.10.23
 API version:       1.41
 Go version:        go1.18.10
 Git commit:        7155243
 Built:             Thu Jan 19 17:34:13 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.23
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.10
  Git commit:       6051f14
  Built:            Thu Jan 19 17:32:04 2023
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.6.15
  GitCommit:        5b842e528e99d4d4c1686467debf2bd4b88ecd86
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

ip6tables -L -v -n

Chain INPUT (policy ACCEPT 3475 packets, 598K bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     udp      *      *       ::/0                 ::/0                 udp dpt:443
15534 2622K ACCEPT     tcp      *      *       ::/0                 ::/0                 tcp dpt:443
   18  1755 ACCEPT     tcp      *      *       ::/0                 ::/0                 tcp dpt:80
16000 1582K ACCEPT     icmpv6    *      *       ::/0                 ::/0                
   15  2116 ACCEPT     tcp      *      *       ::/0                 ::/0                 tcp dpt:22
97069   75M ACCEPT     all      *      *       ::/0                 ::/0                 ctstate ESTABLISHED
  227 22160 DROP       all      eth0   *       ::/0                 ::/0                

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   62 10256 DOCKER-ISOLATION-STAGE-1  all      *      *       ::/0                 ::/0                
    3   312 ACCEPT     all      *      docker0  ::/0                 ::/0                 ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all      *      docker0  ::/0                 ::/0                
   59  9944 ACCEPT     all      docker0 !docker0  ::/0                 ::/0                
    0     0 ACCEPT     all      docker0 docker0  ::/0                 ::/0                

Chain OUTPUT (policy ACCEPT 144K packets, 95M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   59  9944 DOCKER-ISOLATION-STAGE-2  all      docker0 !docker0  ::/0                 ::/0                
   62 10256 RETURN     all      *      *       ::/0                 ::/0                

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all      *      docker0  ::/0                 ::/0                
   59  9944 RETURN     all      *      *       ::/0                 ::/0

127156 · January 22, 2023, 4:20pm

Do you have the following in daemon.json?

{
“experimental”: true,
“ipv6”: true,
“ip6tables”: true,
“fixed-cidr-v6”: “fd9e:63ac:6dcd::/48”
}

Here is the default chain for iptables:

host:~# iptables -S | grep DOCKER-USER
-N DOCKER-USER
-A FORWARD -j DOCKER-USER
-A DOCKER-USER -j RETURN

What is the output of ip6tables -S?

cf3005 · January 23, 2023, 8:46am

cat /etc/docker/daemon.json

{
    "ipv6": true,
    "experimental": true,
    "ip6tables": true,
    "fixed-cidr-v6": "fd00:dead:beef::/48",
    "runtimes": {
        "sysbox-runc": {
            "path": "/usr/bin/sysbox-runc"
        }
    },
    "bip": "172.20.0.1/16",
    "default-address-pools": [
        {
            "base": "172.25.0.0/16",
            "size": 24
        }
    ]
}

iptables -S | grep DOCKER-USER

-N DOCKER-USER
-A FORWARD -j DOCKER-USER
-A DOCKER-USER -m conntrack --ctstate ESTABLISHED -j ACCEPT
-A DOCKER-USER -i eth0 -j DROP
-A DOCKER-USER -j RETURN

ip6tables -S | grep DOCKER-USER
no ouput

127156 · January 23, 2023, 10:46am

Thanks for sharing this info. Per Docker and iptables | Docker Documentation, user should use DOCKER-USER chain for iptables. But it has not mentioned the same for ip6tables. Can some docker expert please provide an answer to this?

rimelek · January 24, 2023, 7:58pm

I checked the sourcecode of Docker and I found that “DOCKER-USER” is mentioned in the “libnetwork” library which doesn’t support this chain for IPv6. You can find the TODO line which was added two and a half years ago and it was the last commit on that file

github.com

moby/libnetwork/blob/05b93e0d3a95952f70c113b0bc5bdb538d7afdd7/firewall_linux.go#L29


      
          // This chain allow users to configure firewall policies in a way that persists
          // docker operations/restarts. Docker will not delete or modify any pre-existing
          // rules from the DOCKER-USER filter chain.
          // Note once DOCKER-USER chain is created, docker engine does not remove it when
          // IPTableForwarding is disabled, because it contains rules configured by user that
          // are beyond docker engine's control.
          func arrangeUserFilterRule() {
          	if ctrl == nil || !ctrl.iptablesEnabled() {
          		return
          	}
          	// TODO IPv6 support
          	iptable := iptables.GetIptable(iptables.IPv4)
          	_, err := iptable.NewChain(userChain, iptables.Filter, false)
          	if err != nil {
          		logrus.Warnf("Failed to create %s chain: %v", userChain, err)
          		return
          	}
          
          
	if err = iptable.AddReturnRule(userChain); err != nil {
          		logrus.Warnf("Failed to add the RETURN rule for %s: %v", userChain, err)
          		return

127156 · January 24, 2023, 8:12pm

@rimelek, Thank you so much to confirm this. Any idea when this will be supported?

rimelek · January 24, 2023, 8:19pm

No idea But there is a request on GitHub. You could add your comments to the issue and hopefully someone eventually replies

github.com/moby/libnetwork

No DOCKER-USER chain with `ip6tables: true`

opened 09:16AM - 08 Jun 22 UTC

RaphMad

I'm using the setting `ip6tables: true` option with no problems so far. The o…nly conceptual difference I found to IPv4 is that no `DOCKER-USER` chain gets created, which makes it hard to insert custom ip6tables rules at a defined location. `dockerd --version`: `Docker version 20.10.17, build a89b842` ``` iptables -nvL | grep DOCKER-USER 417K 335M DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0 Chain DOCKER-USER (1 references) ``` ``` ip6tables -nvL | grep DOCKER-USER ``` Is this chain still planned / slipped through / not planned for IPv6? I could not really find any discussion about it browsing the old merged/closed tickets regarding ip6tables.

127156 · January 24, 2023, 8:51pm

Thanks for the info.

cf3005 · February 2, 2023, 6:40pm

Added with docker v23

docker version

Client: Docker Engine - Community
 Version:           23.0.0
 API version:       1.42
 Go version:        go1.19.5
 Git commit:        e92dd87
 Built:             Wed Feb  1 17:43:17 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          23.0.0
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.5
  Git commit:       d7573ab
  Built:            Wed Feb  1 17:43:17 2023
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.6.16
  GitCommit:        31aa4358a36870b21a992d3ad2bef29e1d693bec
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

iptables -S | grep DOCKER-USER

-N DOCKER-USER
-A FORWARD -j DOCKER-USER
-A DOCKER-USER -m conntrack --ctstate ESTABLISHED -j ACCEPT
-A DOCKER-USER -i eth0 -j DROP
-A DOCKER-USER -j RETURN

ip6tables -S | grep DOCKER-USER

-N DOCKER-USER
-A FORWARD -j DOCKER-USER
-A DOCKER-USER -j RETURN