Docker tries to create a chain on table nat with ip6tables (but there's no nat table on ipv6) [SOLVED]

General
, ,
1

Hi,

I have configure my docker daemon with the following daemon.json:

{
  "ipv6": false,
  "fixed-cidr-v6": "2001:db8:1::/64"
}

This used to work in older docker versions. So here’s the current setup:

Client:
 Version:           27.4.1
 API version:       1.47
 Go version:        go1.23.4
 Git commit:        b9d17eaebb55b7652ce37ae5c7c52fcb34194956
 Built:             Fri Jan 17 08:02:51 2025
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          27.4.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.23.4
  Git commit:       c710b88579fcb5e0d53f96dcae976d79323b9166
  Built:            Tue Jan 14 09:47:44 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.0.1
  GitCommit:        88aa2f531d6c2922003cc7929e51daf1c14caa0a
 runc:
  Version:          1.2.4
  GitCommit:        6c52b3fc541fb26fe8c374d5f58112a0a5dbda66
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad007797e0dcd8b7126f27bb87401d224240

since yesterdays system update when I start docker it errors out with the following error:

time="2025-01-28T18:26:39.643255160+01:00" level=info msg="Loading containers: start."
time="2025-01-28T18:26:40.751789695+01:00" level=warning msg="ip6tables is enabled, but cannot set up ip6tables chains" error="failed to create NAT chain DOCKER: iptables failed: ip6tables --wait -t nat -N DOCKER: ip6tables v1.8.11 (legacy): can't initialize ip6tables table `nat': Table does not exist (do you need to insmod?)\nPerhaps ip6tables or your kernel needs to be upgraded.\n (exit status 3)"
time="2025-01-28T18:26:41.763463183+01:00" level=warning msg="could not create bridge network for id b752425d73efd37c2789477a796435f217738c00ea012cf4285a542c100f1fc1 bridge name docker0 while booting up from persistent state: Failed to Setup IP tables: Unable to enable NAT rule:  (iptables failed: ip6tables --wait -t nat -I POSTROUTING -s 2001:db8:1::/64 ! -o docker0 -j MASQUERADE: ip6tables v1.8.11 (legacy): can't initialize ip6tables table `nat': Table does not exist (do you need to insmod?)\nPerhaps ip6tables or your kernel needs to be upgraded.\n (exit status 3))"
time="2025-01-28T18:26:43.501596376+01:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
time="2025-01-28T18:26:44.255610268+01:00" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
failed to start daemon: Error initializing network controller: error creating default "bridge" network: Failed to Setup IP tables: Unable to enable NAT rule:  (iptables failed: ip6tables --wait -t nat -I POSTROUTING -s 2001:db8:1::/64 ! -o docker0 -j MASQUERADE: ip6tables v1.8.11 (legacy): can't initialize ip6tables table `nat': Table does not exist (do you need to insmod?)
Perhaps ip6tables or your kernel needs to be upgraded.
 (exit status 3))

I already tried to circumvent this problem by setting “ipv6_enable:false” in the docker-compose files I have ìn their networks section, but the problem persists.

The only way I can make docker start is to disable ipv6 in daemon.json to “ipv6: disabled”. But I’m am planning to run a container that needs an ipv6 network so I need to re-enable it.

I don’t know if this is some kind of bug that docker tries to use ‘nat’ table with ip6tables, this of course fails or if I have configured something wrong.

As I said, it used to work with ipv6 enabled until the latest system update. So something changed that it now tries to setup a chain DOCKER on table nat on ip6tables which ofc doesnot work and I have no clue if this needs a bug report or how to fix it without completely disabling ipv6 on docker as a whole.

2

According to the release notes, Docker 27.0.1 introduced changes to ipv6:

Seems like this is your solution:

  • To restore the behavior of earlier releases, no ip6tables at all, set "ip6tables": false in daemon.json, or use the CLI option --ip6tables=false. Alternatively, leave ip6tables enabled, publish ports, and enable direct routing.
1 Like
3

ok, this would restore the old behaviour but why does it break at all? Is my ip6tables broken or docker?

4

You could try if you miss kernel modules that should be loaded:

 lsmod | grep table
iptable_mangle         16384  10
ip6table_nat           16384  1
ip6table_filter        16384  1
ip6_tables             32768  2 ip6table_filter,ip6table_nat
iptable_nat            16384  21
nf_nat                 45056  5 ip6table_nat,xt_nat,iptable_nat,xt_MASQUERADE,xt_REDIRECT
iptable_filter         16384  8
nf_tables_set          32768  1
nf_tables             151552  10 nft_dup_ipv4,nf_tables_set
nfnetlink              16384  4 nf_conntrack_netlink,nf_tables
ip_tables              32768  3 iptable_filter,iptable_nat,iptable_mangle
x_tables               40960  15 ip6table_filter,xt_conntrack,iptable_filter,xt_tcpudp,xt_bpf,xt_addrtype,xt_nat,xt_ipvs,xt_policy,ip6_tables,ip_tables,xt_MASQUERADE,iptable_mangle,xt_REDIRECT,xt_mark

You can manually load modules with sudo modprobe ip6table_nat (replace ip6table_nat with whatever kernel module you want to load). If loading one of the kernel modules fixes the problem for you, you can persist it by editing and adding the module in/etc/modules.

5

Indeed, I wasn’t aware that even an ip6table_nat module exists. I was always told there’s no NAT on ipv6. But when I tried it was failing telling me that there’s no such module. So I checked the kernel and found that NAT support for ipv6 was disabled. After recompiling the kernel with NAT support it started to work. Although it created another problem. It was telling me that it cannot create default Bridge because an identical already existed. I had to remove network dB file and delete the docker0 link to made it start and then recreate all containers. Somehow the existing container networks were interfering with docker0 bridge and you can’t manage networks without docker daemon running. So before enabling NAT for ipv6 I should have removed the existing containers.

6

Sounds like an odyssey.

Ipv6 NAT is around for a long time, but is frowned upon in the ipv6 community. I can see that even within a ULA range, NAT can be helpful, as it allows using the predictable ipv6 address of the host and published container ports to access containers.

I hope you just used this range as a placeholder, and don’t actually use it in your setup. Ranges with the range 2001:DB8::/32 are reserved for documentation purposes. You should prefer a ULA range within the fd00::/8 range.

Interesting. So it wasn’t enough to stop the docker demon, apply your changes to the daemon.json, remove the docker0 interface, and then restart the docker demon again?

Note: only the default bridge is associated with the docker0 interface and it’s subnet. User defined networks are not.

7

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.