Failing to start dockerd: failed to create NAT chain DOCKER

General
1

OS Version: Debian buster
Docker version 18.09.8, build 0dd43dd

Kernel info:

info: reading kernel config from /boot/config-4.19.57-custom ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_NF_NAT_IPV4: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_MEMCG_SWAP_ENABLED: enabled
    (cgroup swap accounting is currently enabled)
- CONFIG_LEGACY_VSYSCALL_EMULATE: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_IOSCHED_CFQ: enabled (as module)
- CONFIG_CFQ_GROUP_IOSCHED: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled (as module)
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: enabled
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
    - CONFIG_BRIDGE_VLAN_FILTERING: enabled
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled (as module)
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: enabled (as module)
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled (as module)
      - CONFIG_XFRM_ALGO: enabled (as module)
      - CONFIG_INET_ESP: enabled (as module)
      - CONFIG_INET_XFRM_MODE_TRANSPORT: enabled (as module)
  - "ipvlan":
    - CONFIG_IPVLAN: enabled (as module)
  - "macvlan":
    - CONFIG_MACVLAN: enabled (as module)
    - CONFIG_DUMMY: enabled (as module)
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_FTP: enabled (as module)
    - CONFIG_NF_NAT_TFTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled (as module)
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled (as module)
    - CONFIG_DM_THIN_PROVISIONING: missing
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

When trying to run dockerd, I get the following error (see line ‘Error starting daemon’):

INFO[2019-07-18T18:33:10.197537317+02:00] parsed scheme: "unix"                         module=grpc
INFO[2019-07-18T18:33:10.197576572+02:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2019-07-18T18:33:10.197612240+02:00] parsed scheme: "unix"                         module=grpc
INFO[2019-07-18T18:33:10.197623638+02:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2019-07-18T18:33:10.197667784+02:00] ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0  <nil>}]  module=grpc
INFO[2019-07-18T18:33:10.197674102+02:00] ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0  <nil>}]  module=grpc
INFO[2019-07-18T18:33:10.197699688+02:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2019-07-18T18:33:10.197704373+02:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2019-07-18T18:33:10.197734321+02:00] pickfirstBalancer: HandleSubConnStateChange: 0xc4201b7570, CONNECTING  module=grpc
INFO[2019-07-18T18:33:10.197748570+02:00] pickfirstBalancer: HandleSubConnStateChange: 0xc4209aa120, CONNECTING  module=grpc
INFO[2019-07-18T18:33:10.197854584+02:00] pickfirstBalancer: HandleSubConnStateChange: 0xc4201b7570, READY  module=grpc
INFO[2019-07-18T18:33:10.197866220+02:00] pickfirstBalancer: HandleSubConnStateChange: 0xc4209aa120, READY  module=grpc
INFO[2019-07-18T18:33:10.199208498+02:00] [graphdriver] using prior storage driver: overlay2 
INFO[2019-07-18T18:33:10.215168664+02:00] Graph migration to content-addressability took 0.00 seconds 
WARN[2019-07-18T18:33:10.215725104+02:00] Your kernel does not support cgroup blkio weight 
WARN[2019-07-18T18:33:10.215760611+02:00] Your kernel does not support cgroup blkio weight_device 
INFO[2019-07-18T18:33:10.216626678+02:00] Loading containers: start.                   
INFO[2019-07-18T18:33:10.451838313+02:00] stopping event stream following graceful shutdown  error="<nil>" module=libcontainerd namespace=moby
Error starting daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.8.2 (nf_tables):  CHAIN_ADD failed (No such file or directory): chain PREROUTING
 (exit status 4)

When trying to execute sudo iptables -t nat -N DOCKER manually, I get this:

iptables v1.8.2 (nf_tables):  CHAIN_ADD failed (No such file or directory): chain PREROUTING

Have I installed everything correctly or is something missing, maybe in the kernel?

2

The docker installer uses iptables for nat. Unfortunately Debian uses nftables. You can convert the entries over to nftables or just setup Debian to use the legacy iptables.

sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

dockerd, should start fine after switching to iptables-legacy.

sudo service docker start

-blemis

4 Likes
3

Thank you, this solved the issue.

4

Hi,

i have tried above solution but in my case i am getting
"update-alternatives: error: no alternatives for iptables"
any suggestion would be appriciated.

Regards
Fais

6

Which errors are you getting?

8

Thanks for your response i am running docker container inside docker container.
actually i want to run another container on top of teamcity agent contianer…
thats where i am getting error “docker deamon not running”

9

Try running dockerd service without iptables rules, e.g.

!dockerd --iptables=false

Related: https://stackoverflow.com/a/64261697/55075

10

@fais786 did you find the solution or what did you do?

thanks in advance

11

It’s stuck on API listen on /var/run/docker.sock

INFO[2022-09-20T09:26:53.346228872Z] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2022-09-20T09:26:53.347457464Z] parsed scheme: "unix"                         module=grpc
INFO[2022-09-20T09:26:53.347491995Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2022-09-20T09:26:53.347532393Z] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-09-20T09:26:53.347619741Z] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2022-09-20T09:26:53.651040767Z] [graphdriver] using prior storage driver: overlay2 
WARN[2022-09-20T09:26:53.863726637Z] Unable to find cpu cgroup in mounts          
WARN[2022-09-20T09:26:53.865065168Z] Your kernel does not support cgroup blkio weight 
WARN[2022-09-20T09:26:53.865387770Z] Your kernel does not support cgroup blkio weight_device 
INFO[2022-09-20T09:26:53.865882964Z] Loading containers: start.                   
INFO[2022-09-20T09:26:54.121014619Z] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address 
WARN[2022-09-20T09:26:54.121646758Z] unable to disable IPv6 router advertisement   error="open /proc/sys/net/ipv6/conf/docker0/accept_ra: read-only file system"
WARN[2022-09-20T09:26:54.121686961Z] unable to disable IPv6 router advertisement   error="open /proc/sys/net/ipv6/conf/docker0/accept_ra: read-only file system"
INFO[2022-09-20T09:26:54.254928442Z] Loading containers: done.                    
WARN[2022-09-20T09:26:54.550146690Z] Not using native diff for overlay2, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled  storage-driver=overlay2
INFO[2022-09-20T09:26:54.550397589Z] Docker daemon                                 commit=e42327a graphdriver(s)=overlay2 version=20.10.18
INFO[2022-09-20T09:26:54.550557935Z] Daemon has completed initialization          
INFO[2022-09-20T09:26:54.967549879Z] API listen on /var/run/docker.sock