Container cannot ping local network, but can ping gateway and host

General
,
1

From the container, I can ping the Docker host (10.0.0.100/24), my gateway (10.0.0.1/24), and things on other subnets (10.0.2.1/24), but not anything else on the same subnet as my host.
The funny thing is that I got it working on an LXC container on this same computer, but there were other problems so I elected to install Docker straight to the host.

Network inspect output for each Docker network involved:

bridge:

[
    {
        "Name": "bridge",
        "Id": "8a5491f38105ee653495f60582d79cba5feeb52e03a38b9df5aa05738fd37211",
        "Created": "2021-05-21T13:56:28.153866116-04:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "192.168.200.0/24",
                    "Gateway": "192.168.200.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

docker_gwbridge

[
    {
        "Name": "docker_gwbridge",
        "Id": "26356f2a70b03bf5479d114aa9905f8311de27c8f6978e8f1c2261e022adf5ba",
        "Created": "2021-05-21T13:56:54.192644203-04:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "24cb68d0065a0d8dbe81a7f1bbc6d7c4a8d5efe030dc0e269db623b4c6330b9e": {
                "Name": "gateway_9527e9d95a0e",
                "EndpointID": "fc3c0d89a17fc3288f746df9a37a4cd7ea1cc64ae04179eb27d09c43e2863e50",
                "MacAddress": "02:42:ac:11:00:04",
                "IPv4Address": "172.17.0.4/16",
                "IPv6Address": ""
            },
            "859d8beef9b97cf715ef866630f0733d45027f8759f3714ffb557cb5d9029cc5": {
                "Name": "gateway_ec4f062be987",
                "EndpointID": "b5ffd158d0e51a62d794b0b71abc66ba6c5e05a31ac70c61c80070770e4bc2d5",
                "MacAddress": "02:42:ac:11:00:06",
                "IPv4Address": "172.17.0.6/16",
                "IPv6Address": ""
            },
            "aa71e016d05d3cd6bffdfd6ba2489d33af2764a36ef1dd1c6aef105bc4fea32b": {
                "Name": "gateway_399f3ef73ab8",
                "EndpointID": "f5fb19c944fc2fd0e859bcaf0c06b51245df3d694d29a5b0d61b607e3bfb8c7c",
                "MacAddress": "02:42:ac:11:00:03",
                "IPv4Address": "172.17.0.3/16",
                "IPv6Address": ""
            },
            "ca29d89aa4180eca9120a48d3527aadf8bf1d0a5cb537838e2ccc19ef7f8127e": {
                "Name": "gateway_3444a7855c3a",
                "EndpointID": "0db3b1a64ed2917d310695887713486cc2102422732fe52ff7704b6f5a70c261",
                "MacAddress": "02:42:ac:11:00:05",
                "IPv4Address": "172.17.0.5/16",
                "IPv6Address": ""
            },
            "ingress-sbox": {
                "Name": "gateway_ingress-sbox",
                "EndpointID": "1c7ebf0cce1311bbe760796731d00d68eb39129340c050e59220ed454eb721eb",
                "MacAddress": "02:42:ac:11:00:02",
                "IPv4Address": "172.17.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.enable_icc": "false",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.name": "docker_gwbridge"
        },
        "Labels": {}
    }
]

ingress:

[
    {
        "Name": "ingress",
        "Id": "nn6ke664gi6nb9n3sfskv919t",
        "Created": "2021-05-21T13:56:51.984084879-04:00",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "192.168.0.0/24",
                    "Gateway": "192.168.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": true,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "859d8beef9b97cf715ef866630f0733d45027f8759f3714ffb557cb5d9029cc5": {
                "Name": "media_stack_sonarr.1.zrq12rcp27y3wgagwepv0vm47",
                "EndpointID": "849343db73747831e46339715988e984235455826a94889a00f6806f79f1c71c",
                "MacAddress": "02:42:c0:a8:00:2f",
                "IPv4Address": "192.168.0.47/24",
                "IPv6Address": ""
            },
            "aa71e016d05d3cd6bffdfd6ba2489d33af2764a36ef1dd1c6aef105bc4fea32b": {
                "Name": "portainer_portainer.1.tvtuqx7w86xg2f3ka27mhs8rp",
                "EndpointID": "3ae4c395be8977b65ab311d2169dad1b370f411dbb5c1ed50cf3c08f79fca599",
                "MacAddress": "02:42:c0:a8:00:05",
                "IPv4Address": "192.168.0.5/24",
                "IPv6Address": ""
            },
            "ca29d89aa4180eca9120a48d3527aadf8bf1d0a5cb537838e2ccc19ef7f8127e": {
                "Name": "media_stack_radarr.1.z6aow6ws2vm3qme6v34mphsgy",
                "EndpointID": "0044c6e368aafea8a1c7110e793ddae36cfe94737f756652a25ef7c5f80f899b",
                "MacAddress": "02:42:c0:a8:00:30",
                "IPv4Address": "192.168.0.48/24",
                "IPv6Address": ""
            },
            "ingress-sbox": {
                "Name": "ingress-endpoint",
                "EndpointID": "dbeb3885039ea0af0cd9d8819e96d8c85d99a84d974f6e812db5a30a5e0e011b",
                "MacAddress": "02:42:c0:a8:00:02",
                "IPv4Address": "192.168.0.2/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4096"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "33d050b7f3e8",
                "IP": "10.0.0.100"
            }
        ]
    }
]

media_stack_default (where my containers live):

[
    {
        "Name": "media_stack_default",
        "Id": "hovcq4lpnd4w1blslgppeyk9c",
        "Created": "2021-05-21T14:01:23.706193994-04:00",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "192.168.2.0/24",
                    "Gateway": "192.168.2.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "859d8beef9b97cf715ef866630f0733d45027f8759f3714ffb557cb5d9029cc5": {
                "Name": "media_stack_sonarr.1.zrq12rcp27y3wgagwepv0vm47",
                "EndpointID": "4fcdff233de4a893005cb0b342be48d7c4cd31f93504a7682efbd7aad92ccd3e",
                "MacAddress": "02:42:c0:a8:02:3f",
                "IPv4Address": "192.168.2.63/24",
                "IPv6Address": ""
            },
            "ca29d89aa4180eca9120a48d3527aadf8bf1d0a5cb537838e2ccc19ef7f8127e": {
                "Name": "media_stack_radarr.1.z6aow6ws2vm3qme6v34mphsgy",
                "EndpointID": "e7029ffa2494986477c6fd70623023aac712aa54531a1bffbef1c27fdf34efe0",
                "MacAddress": "02:42:c0:a8:02:40",
                "IPv4Address": "192.168.2.64/24",
                "IPv6Address": ""
            },
            "lb-media_stack_default": {
                "Name": "media_stack_default-endpoint",
                "EndpointID": "9bec96bc583ec71738acae1692fb686b5dc7658f3fab15e4a67e20df80fab9d3",
                "MacAddress": "02:42:c0:a8:02:3d",
                "IPv4Address": "192.168.2.61/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4098"
        },
        "Labels": {
            "com.docker.stack.namespace": "media_stack"
        },
        "Peers": [
            {
                "Name": "33d050b7f3e8",
                "IP": "10.0.0.100"
            }
        ]
    }
]

output of ip a:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:1b:21:29:cc:c9 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.100/24 brd 10.0.0.255 scope global enp2s0
       valid_lft forever preferred_lft forever
    inet6 fe80::21b:21ff:fe29:ccc9/64 scope link 
       valid_lft forever preferred_lft forever
3: eno1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 0c:c4:7a:a8:86:0e brd ff:ff:ff:ff:ff:ff
4: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 0c:c4:7a:a8:86:0f brd ff:ff:ff:ff:ff:ff
23: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:03:1b:70 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
24: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:03:1b:70 brd ff:ff:ff:ff:ff:ff
79: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:bc:de:47:61 brd ff:ff:ff:ff:ff:ff
    inet 192.168.200.1/24 brd 192.168.200.255 scope global docker0
       valid_lft forever preferred_lft forever
80: docker_gwbridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:7d:cf:30:57 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker_gwbridge
       valid_lft forever preferred_lft forever
    inet6 fe80::42:7dff:fecf:3057/64 scope link 
       valid_lft forever preferred_lft forever
96: vetha697992@veth26f59ba: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff
97: veth26f59ba@vetha697992: <NO-CARRIER,BROADCAST,MULTICAST,UP,M-DOWN> mtu 1500 qdisc noqueue master docker_gwbridge state LOWERLAYERDOWN group default 
    link/ether be:cf:55:fb:2e:f7 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::bccf:55ff:fefb:2ef7/64 scope link 
       valid_lft forever preferred_lft forever
148: veth311f3ed@if147: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker_gwbridge state UP group default 
    link/ether de:98:34:a6:ea:0b brd ff:ff:ff:ff:ff:ff link-netnsid 21
    inet6 fe80::dc98:34ff:fea6:ea0b/64 scope link 
       valid_lft forever preferred_lft forever
161: veth8149a09@if160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker_gwbridge state UP group default 
    link/ether 06:65:fb:30:f3:21 brd ff:ff:ff:ff:ff:ff link-netnsid 25
    inet6 fe80::465:fbff:fe30:f321/64 scope link 
       valid_lft forever preferred_lft forever
165: veth03feab7@if164: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker_gwbridge state UP group default 
    link/ether 9a:e0:9f:f5:47:03 brd ff:ff:ff:ff:ff:ff link-netnsid 24
    inet6 fe80::98e0:9fff:fef5:4703/64 scope link 
       valid_lft forever preferred_lft forever
229: veth44ad186@if228: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker_gwbridge state UP group default 
    link/ether de:fa:07:c7:6b:a6 brd ff:ff:ff:ff:ff:ff link-netnsid 28
    inet6 fe80::dcfa:7ff:fec7:6ba6/64 scope link 
       valid_lft forever preferred_lft forever
231: veth18ae407@if230: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker_gwbridge state UP group default 
    link/ether 42:fd:52:c9:31:3d brd ff:ff:ff:ff:ff:ff link-netnsid 29
    inet6 fe80::40fd:52ff:fec9:313d/64 scope link 
       valid_lft forever preferred_lft forever

iptables:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:domain
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootps
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:bootps

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
DOCKER-INGRESS  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootpc

Chain DOCKER (2 references)
target     prot opt source               destination         

Chain DOCKER-INGRESS (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

daemon.json:

{
  "storage-driver": "zfs",
  "data-root": "/Tank/Docker",
  "dns": ["10.0.0.1"],
  "bip": "192.168.200.1/24",
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "1",
    "labels": "production_status",
    "env": "os,customer"
  }
}

ping results from a container in media_stack_default:

root@ac5b36e9365c:/# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=116 time=8.39 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=116 time=8.20 ms
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 8.196/8.292/8.389/0.096 ms
root@ac5b36e9365c:/# ping 10.0.0.1
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
64 bytes from 10.0.0.1: icmp_seq=1 ttl=63 time=0.201 ms
64 bytes from 10.0.0.1: icmp_seq=2 ttl=63 time=0.235 ms
--- 10.0.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1023ms
rtt min/avg/max/mdev = 0.201/0.218/0.235/0.017 ms
root@ac5b36e9365c:/# ping 10.0.0.100
PING 10.0.0.100 (10.0.0.100) 56(84) bytes of data.
64 bytes from 10.0.0.100: icmp_seq=1 ttl=64 time=0.077 ms
64 bytes from 10.0.0.100: icmp_seq=2 ttl=64 time=0.064 ms
--- 10.0.0.100 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1010ms
rtt min/avg/max/mdev = 0.064/0.070/0.077/0.006 ms
root@ac5b36e9365c:/# ping 10.0.0.101
PING 10.0.0.101 (10.0.0.101) 56(84) bytes of data.
From 10.0.0.100 icmp_seq=1 Destination Host Unreachable
From 10.0.0.100 icmp_seq=2 Destination Host Unreachable
From 10.0.0.100 icmp_seq=3 Destination Host Unreachable
--- 10.0.0.101 ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3067ms

and a traceroute for good measure:

root@ac5b36e9365c:/# traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  172.17.0.1 (172.17.0.1)  0.084 ms  0.033 ms  0.021 ms
 2  usg (10.0.0.1)  0.159 ms  0.149 ms  0.187 ms
 3  * * *
 4  [LOCATION CENSORED].verizon-gni.net (100.41.XXX.XXX)  5.391 ms  5.356 ms  5.327 ms
 5  * * *
 6  * * *
 7  0.et-9-1-5.GW15.NYC1.ALTER.NET (140.222.230.215)  7.230 ms 0.et-10-0-5.GW15.NYC1.ALTER.NET (140.222.1.83)  6.651 ms 0.et-9-1-2.GW15.NYC1.ALTER.NET (140.222.227.25)  10.064 ms
 8  204.148.20.6 (204.148.20.6)  7.929 ms 72.14.208.130 (72.14.208.130)  10.079 ms 204.148.20.6 (204.148.20.6)  7.869 ms
 9  108.170.248.1 (108.170.248.1)  10.298 ms 108.170.248.97 (108.170.248.97)  10.014 ms 108.170.248.33 (108.170.248.33)  12.803 ms
10  dns.google (8.8.8.8)  7.430 ms 142.250.224.247 (142.250.224.247)  10.507 ms dns.google (8.8.8.8)  9.416 ms

I’ve been banging my head against a wall on and off for a few days now, either I’m stupid and it’s an incredibly simple fix, or something really weird is happening.

2

docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  scan: Docker Scan (Docker Inc., v0.7.0)

Server:
 Containers: 3
  Running: 2
  Paused: 0
  Stopped: 1
 Images: 4
 Server Version: 20.10.6
 Storage Driver: zfs
  Zpool: Tank
  Zpool Health: ONLINE
  Parent Dataset: Tank/Docker
  Space Used By Parent: 5454034560
  Space Available: 45437256705360
  Parent Quota: no
  Compression: lz4
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: gilzm3mebiqpwkiem9e6sqil9
  Is Manager: true
  ClusterID: ky5h6ohh59yrd6wun0ymxl8kf
  Managers: 1
  Nodes: 1
  Default Address Pool: 192.168.0.0/17  
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 10.0.0.100
  Manager Addresses:
   10.0.0.100:2377
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.15.0-143-generic
 Operating System: Ubuntu 18.04.5 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 31.31GiB
 Name: artemis
 ID: ZE3T:CFW7:5JCT:5ZON:YPND:7SIB:UTI5:HHYT:SAHP:VQUJ:EP46:GBQX
 Docker Root Dir: /Tank/Docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support
4

I can ping the internet just fine and I can ping my host fine, it’s pinging my local network that doesn’t work. I also don’t see how running commands to look at my network config (which I have posted) will solve my issue.

5

Turns out I can ping local IP addresses, it’s just that the ones I was pinging weren’t responding to the Docker container for some reason. Managed to get a kludge in place until I can migrate those services to Docker.