SOLVED: External network addresses starting 172 being blocked from server running docker

Open Source Projects DockerEngine
1

A server that we have docker installed on has suddenly become unavailable from one of our internal networks.

The network team has identified that the issue only affects users on a network 172.31.0.0/18 other users on networks starting 10.x.x.x are not affected.

Our docker bridge network is 172.17.0.0/16

The symptoms are that the server is totally inaccessible, we don’t even get a response from a ping to the server.

The only drop messages in iptables are as follows

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all – anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all – anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all – anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all – anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all – anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all – anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all – anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all – anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all – anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all – anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all – anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all – anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all – anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all – anywhere anywhere
RETURN all – anywhere anywhere

Chain DOCKER-ISOLATION-STAGE-2 (14 references)
target prot opt source destination
DROP all – anywhere anywhere
DROP all – anywhere anywhere
DROP all – anywhere anywhere
DROP all – anywhere anywhere
DROP all – anywhere anywhere
DROP all – anywhere anywhere
DROP all – anywhere anywhere
DROP all – anywhere anywhere
DROP all – anywhere anywhere
DROP all – anywhere anywhere
DROP all – anywhere anywhere
DROP all – anywhere anywhere
DROP all – anywhere anywhere
DROP all – anywhere anywhere
RETURN all – anywhere anywhere

Does anyone know if these iptables entries could cause these symptoms?

If not, is there anything else I could try to remove docker or the server itself from the equation and perhaps narrow the source of the problem down to a network router issue?

2

Problem sorted, I went through all the networks that had been setup on the server and found a conflicting network 172.31.0.0/16. I deleted that network which fixed the issue.

By the way is there an easy way to list all the docker subnets in use?, rather than issuing docker network inspect xxx for each network?

3

You can try this on Linux, Mac or WSL2. It handles the bridge networks only:

docker network ls -q --filter 'driver=bridge' | xargs -I '{}' docker network inspect '{}' --format '{{ range .IPAM.Config }}{{ .Subnet }}{{ end }}'

Or you can just list the networks without docker on Linux using “ip addr”

6: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:bf:17:1a:28 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
7: br-05d69f0514ac: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:cc:1a:6b:08 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-05d69f0514ac
       valid_lft forever preferred_lft forever

It is useful when there are other tools creating networks.