Docker Community Forums

Share and learn in the Docker community.

Failing to start dockerd: failed to create NAT chain DOCKER

OS Version: Debian buster
Docker version 18.09.8, build 0dd43dd

Kernel info:

info: reading kernel config from /boot/config-4.19.57-custom ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_NF_NAT_IPV4: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_MEMCG_SWAP_ENABLED: enabled
    (cgroup swap accounting is currently enabled)
- CONFIG_LEGACY_VSYSCALL_EMULATE: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_IOSCHED_CFQ: enabled (as module)
- CONFIG_CFQ_GROUP_IOSCHED: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled (as module)
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: enabled
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
    - CONFIG_BRIDGE_VLAN_FILTERING: enabled
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled (as module)
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: enabled (as module)
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled (as module)
      - CONFIG_XFRM_ALGO: enabled (as module)
      - CONFIG_INET_ESP: enabled (as module)
      - CONFIG_INET_XFRM_MODE_TRANSPORT: enabled (as module)
  - "ipvlan":
    - CONFIG_IPVLAN: enabled (as module)
  - "macvlan":
    - CONFIG_MACVLAN: enabled (as module)
    - CONFIG_DUMMY: enabled (as module)
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_FTP: enabled (as module)
    - CONFIG_NF_NAT_TFTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled (as module)
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled (as module)
    - CONFIG_DM_THIN_PROVISIONING: missing
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

When trying to run dockerd, I get the following error (see line ‘Error starting daemon’):

INFO[2019-07-18T18:33:10.197537317+02:00] parsed scheme: "unix"                         module=grpc
INFO[2019-07-18T18:33:10.197576572+02:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2019-07-18T18:33:10.197612240+02:00] parsed scheme: "unix"                         module=grpc
INFO[2019-07-18T18:33:10.197623638+02:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2019-07-18T18:33:10.197667784+02:00] ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0  <nil>}]  module=grpc
INFO[2019-07-18T18:33:10.197674102+02:00] ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0  <nil>}]  module=grpc
INFO[2019-07-18T18:33:10.197699688+02:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2019-07-18T18:33:10.197704373+02:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2019-07-18T18:33:10.197734321+02:00] pickfirstBalancer: HandleSubConnStateChange: 0xc4201b7570, CONNECTING  module=grpc
INFO[2019-07-18T18:33:10.197748570+02:00] pickfirstBalancer: HandleSubConnStateChange: 0xc4209aa120, CONNECTING  module=grpc
INFO[2019-07-18T18:33:10.197854584+02:00] pickfirstBalancer: HandleSubConnStateChange: 0xc4201b7570, READY  module=grpc
INFO[2019-07-18T18:33:10.197866220+02:00] pickfirstBalancer: HandleSubConnStateChange: 0xc4209aa120, READY  module=grpc
INFO[2019-07-18T18:33:10.199208498+02:00] [graphdriver] using prior storage driver: overlay2 
INFO[2019-07-18T18:33:10.215168664+02:00] Graph migration to content-addressability took 0.00 seconds 
WARN[2019-07-18T18:33:10.215725104+02:00] Your kernel does not support cgroup blkio weight 
WARN[2019-07-18T18:33:10.215760611+02:00] Your kernel does not support cgroup blkio weight_device 
INFO[2019-07-18T18:33:10.216626678+02:00] Loading containers: start.                   
INFO[2019-07-18T18:33:10.451838313+02:00] stopping event stream following graceful shutdown  error="<nil>" module=libcontainerd namespace=moby
Error starting daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.8.2 (nf_tables):  CHAIN_ADD failed (No such file or directory): chain PREROUTING
 (exit status 4)

When trying to execute sudo iptables -t nat -N DOCKER manually, I get this:

iptables v1.8.2 (nf_tables):  CHAIN_ADD failed (No such file or directory): chain PREROUTING

Have I installed everything correctly or is something missing, maybe in the kernel?

The docker installer uses iptables for nat. Unfortunately Debian uses nftables. You can convert the entries over to nftables or just setup Debian to use the legacy iptables.

sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

dockerd, should start fine after switching to iptables-legacy.

sudo service docker start

-blemis

3 Likes

Thank you, this solved the issue.

Hi,

i have tried above solution but in my case i am getting
"update-alternatives: error: no alternatives for iptables"
any suggestion would be appriciated.

Regards
Fais

$ sudo docker daemon on ] 9:21 AM
ERRO[0000] ‘overlay’ not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded.
INFO[0000] Graph migration to content-addressability took 0.00 seconds
WARN[0000] Running modprobe bridge br_netfilter failed with message: modprobe: ERROR: …/libkmod/libkmod.c:578 kmod_search_moddep() could not open moddep file ‘/lib/modul
es/4.2.0-18-generic/modules.dep.bin’
modprobe: ERROR: …/libkmod/libkmod.c:578 kmod_search_moddep() could not open moddep file ‘/lib/modules/4.2.0-18-generic/modules.dep.bin’
, error: exit status 1
WARN[0000] Running modprobe nf_nat failed with message: modprobe: ERROR: ../libkmod/libkmod.c:578 kmod_search_moddep() could not open moddep file '/lib/modules/4.2.0-18- generic/modules.dep.bin', error: exit status 1
FATA[0000] Error starting daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain: iptables failed: iptables -t nat
-N DOCKER: iptables v1.4.21: can’t initialize iptables table `nat’: Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded.
(exit status 3)

Which errors are you getting?

Do you have a custom kernel? The kernel must have some extra modules. You can check what they are with: https://github.com/moby/moby/blob/master/contrib/check-config.sh

Thanks for your response i am running docker container inside docker container.
actually i want to run another container on top of teamcity agent contianer…
thats where i am getting error “docker deamon not running”

Try running dockerd service without iptables rules, e.g.

!dockerd --iptables=false

Related: https://stackoverflow.com/a/64261697/55075