Failing to start dockerd: failed to create NAT chain DOCKER

OS Version: Debian buster
Docker version 18.09.8, build 0dd43dd

Kernel info:

info: reading kernel config from /boot/config-4.19.57-custom ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_NF_NAT_IPV4: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_MEMCG_SWAP_ENABLED: enabled
    (cgroup swap accounting is currently enabled)
- CONFIG_LEGACY_VSYSCALL_EMULATE: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_IOSCHED_CFQ: enabled (as module)
- CONFIG_CFQ_GROUP_IOSCHED: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled (as module)
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: enabled
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
    - CONFIG_BRIDGE_VLAN_FILTERING: enabled
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled (as module)
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: enabled (as module)
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled (as module)
      - CONFIG_XFRM_ALGO: enabled (as module)
      - CONFIG_INET_ESP: enabled (as module)
      - CONFIG_INET_XFRM_MODE_TRANSPORT: enabled (as module)
  - "ipvlan":
    - CONFIG_IPVLAN: enabled (as module)
  - "macvlan":
    - CONFIG_MACVLAN: enabled (as module)
    - CONFIG_DUMMY: enabled (as module)
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_FTP: enabled (as module)
    - CONFIG_NF_NAT_TFTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled (as module)
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled (as module)
    - CONFIG_DM_THIN_PROVISIONING: missing
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

When trying to run dockerd, I get the following error (see line ‘Error starting daemon’):

INFO[2019-07-18T18:33:10.197537317+02:00] parsed scheme: "unix"                         module=grpc
INFO[2019-07-18T18:33:10.197576572+02:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2019-07-18T18:33:10.197612240+02:00] parsed scheme: "unix"                         module=grpc
INFO[2019-07-18T18:33:10.197623638+02:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2019-07-18T18:33:10.197667784+02:00] ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0  <nil>}]  module=grpc
INFO[2019-07-18T18:33:10.197674102+02:00] ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0  <nil>}]  module=grpc
INFO[2019-07-18T18:33:10.197699688+02:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2019-07-18T18:33:10.197704373+02:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2019-07-18T18:33:10.197734321+02:00] pickfirstBalancer: HandleSubConnStateChange: 0xc4201b7570, CONNECTING  module=grpc
INFO[2019-07-18T18:33:10.197748570+02:00] pickfirstBalancer: HandleSubConnStateChange: 0xc4209aa120, CONNECTING  module=grpc
INFO[2019-07-18T18:33:10.197854584+02:00] pickfirstBalancer: HandleSubConnStateChange: 0xc4201b7570, READY  module=grpc
INFO[2019-07-18T18:33:10.197866220+02:00] pickfirstBalancer: HandleSubConnStateChange: 0xc4209aa120, READY  module=grpc
INFO[2019-07-18T18:33:10.199208498+02:00] [graphdriver] using prior storage driver: overlay2 
INFO[2019-07-18T18:33:10.215168664+02:00] Graph migration to content-addressability took 0.00 seconds 
WARN[2019-07-18T18:33:10.215725104+02:00] Your kernel does not support cgroup blkio weight 
WARN[2019-07-18T18:33:10.215760611+02:00] Your kernel does not support cgroup blkio weight_device 
INFO[2019-07-18T18:33:10.216626678+02:00] Loading containers: start.                   
INFO[2019-07-18T18:33:10.451838313+02:00] stopping event stream following graceful shutdown  error="<nil>" module=libcontainerd namespace=moby
Error starting daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.8.2 (nf_tables):  CHAIN_ADD failed (No such file or directory): chain PREROUTING
 (exit status 4)

When trying to execute sudo iptables -t nat -N DOCKER manually, I get this:

iptables v1.8.2 (nf_tables):  CHAIN_ADD failed (No such file or directory): chain PREROUTING

Have I installed everything correctly or is something missing, maybe in the kernel?

The docker installer uses iptables for nat. Unfortunately Debian uses nftables. You can convert the entries over to nftables or just setup Debian to use the legacy iptables.

sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

dockerd, should start fine after switching to iptables-legacy.

sudo service docker start

-blemis

4 Likes

Thank you, this solved the issue.

Hi,

i have tried above solution but in my case i am getting
"update-alternatives: error: no alternatives for iptables"
any suggestion would be appriciated.

Regards
Fais

Which errors are you getting?

Thanks for your response i am running docker container inside docker container.
actually i want to run another container on top of teamcity agent contianer…
thats where i am getting error “docker deamon not running”

Try running dockerd service without iptables rules, e.g.

!dockerd --iptables=false

Related: https://stackoverflow.com/a/64261697/55075

@fais786 did you find the solution or what did you do?

thanks in advance

It’s stuck on API listen on /var/run/docker.sock

INFO[2022-09-20T09:26:53.346228872Z] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2022-09-20T09:26:53.347457464Z] parsed scheme: "unix"                         module=grpc
INFO[2022-09-20T09:26:53.347491995Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2022-09-20T09:26:53.347532393Z] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-09-20T09:26:53.347619741Z] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2022-09-20T09:26:53.651040767Z] [graphdriver] using prior storage driver: overlay2 
WARN[2022-09-20T09:26:53.863726637Z] Unable to find cpu cgroup in mounts          
WARN[2022-09-20T09:26:53.865065168Z] Your kernel does not support cgroup blkio weight 
WARN[2022-09-20T09:26:53.865387770Z] Your kernel does not support cgroup blkio weight_device 
INFO[2022-09-20T09:26:53.865882964Z] Loading containers: start.                   
INFO[2022-09-20T09:26:54.121014619Z] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address 
WARN[2022-09-20T09:26:54.121646758Z] unable to disable IPv6 router advertisement   error="open /proc/sys/net/ipv6/conf/docker0/accept_ra: read-only file system"
WARN[2022-09-20T09:26:54.121686961Z] unable to disable IPv6 router advertisement   error="open /proc/sys/net/ipv6/conf/docker0/accept_ra: read-only file system"
INFO[2022-09-20T09:26:54.254928442Z] Loading containers: done.                    
WARN[2022-09-20T09:26:54.550146690Z] Not using native diff for overlay2, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled  storage-driver=overlay2
INFO[2022-09-20T09:26:54.550397589Z] Docker daemon                                 commit=e42327a graphdriver(s)=overlay2 version=20.10.18
INFO[2022-09-20T09:26:54.550557935Z] Daemon has completed initialization          
INFO[2022-09-20T09:26:54.967549879Z] API listen on /var/run/docker.sock