User defined networks doesn't work, containers can't connect each other

I have two containers and need them talk to each other.

When I set --network=bridge, I can ping the other one by ip, but I can’t do the same with --network=custom-network, no matter it’s created with docker network create or auto-created with docker compose up.

Both containers are listed in docker network inspect, DNS resolution works as the ip is correct when I ping containers’ hostname.

The problem only happens on a server, I can’t reproduce it on my laptop. Please help if you know anything possibly caused the issue, thanks!

docker version:

Client: Docker Engine - Community
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.6
 Git commit:        ced0996
 Built:             Fri Jul 21 20:35:23 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.6
  Git commit:       a61e2b4
  Built:            Fri Jul 21 20:35:23 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker network inspect bridge:

[
    {
        "Name": "bridge",
        "Id": "fa51ad8c47ad702faa1ac67d771cb364073943ea56700d59514e234066152985",
        "Created": "2023-08-09T11:35:16.307967777+02:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "70b07f85e10b1c165023577afd1158f180a959e76e833a6011928eee521bd686": {
                "Name": "objective_burnell",
                "EndpointID": "5179333804d36dbbf8c1c2414d3bc0d429763d0226c43f6170f07d2c7af28218",
                "MacAddress": "02:42:ac:11:00:02",
                "IPv4Address": "172.17.0.2/16",
                "IPv6Address": ""
            },
            "747e6f2404e5f1e7ce3e2816398aa9e2d44e64fcaf5ef796cb570128df88098c": {
                "Name": "optimistic_turing",
                "EndpointID": "d4334b5d2314d87474963028ff5086bfabd9881fe88f13cce79cce1631a95d6d",
                "MacAddress": "02:42:ac:11:00:03",
                "IPv4Address": "172.17.0.3/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

docker network inspect custom-network:

[
    {
        "Name": "custom-network",
        "Id": "928e554fef69ff5f1d21c834d5768d1a7a4b1e8c844ab72738351d192f1e49af",
        "Created": "2023-08-28T17:13:59.427385022+02:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "192.168.48.0/20",
                    "Gateway": "192.168.48.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "2d50c82223d68a1866611f8c475d9851727629b157e8491a8c2e66a4d632a61c": {
                "Name": "hardcore_greider",
                "EndpointID": "f22e924ce3602834d03941dbf5725db6c2334502981b87bb4c7c4e558cb9117f",
                "MacAddress": "02:42:c0:a8:30:03",
                "IPv4Address": "192.168.48.3/20",
                "IPv6Address": ""
            },
            "5955555bd2f5252a9c9e4fe7b21d4443903eb314e5bd7bcc59ebc4ab9e8bc9bc": {
                "Name": "great_snyder",
                "EndpointID": "e159864411573d38d579c0cfa9132d88c57da8b0a240e28084c53a5e1f16710b",
                "MacAddress": "02:42:c0:a8:30:02",
                "IPv4Address": "192.168.48.2/20",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.driver.mtu": "1500",
            "com.docker.network.bridgs.enable_ip_masquerade": "true"
        },
        "Labels": {}
    }
]

What’s your server OS? Everything updated? Any special firewall settings? The ping works on server?

Thanks for your reply!

  1. Ubuntu 20.04
  2. yes, I upgraded everything within a month
  3. ufw allows ssh, 80 and 443. nginx is listening 80 and 443 port
  4. yes, I can ping both containers on host

Can you reproduce the issue when you use a subnet from the default IP range? 172.18.0.0/16 for example.

Yes,

        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "98a872ab60c0f23df7977518e9c7e084db8cc7b397271382b856b4f841abff4b": {
                "Name": "hardcore_greider",
                "EndpointID": "89ef62ddfa785b04aa0b998c90122861d3080186e5b6a68e0632c2e4418cb4ec",
                "MacAddress": "02:42:ac:12:00:02",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": ""
            },
            "98dd956b2e40910179d8ff0a8b0171af083ad2cebf4a4deb0b032b053d954e0c": {
                "Name": "great_snyder",
                "EndpointID": "9fb4595a1aebcab3c60cf855b43e72fb4838860a308fa651cedfb1082fb461c7",
                "MacAddress": "02:42:ac:12:00:03",
                "IPv4Address": "172.18.0.3/16",
                "IPv6Address": ""
            }
        },

ip route:

default via 100.xxx.xxx.xxx dev eth0 onlink 
100.xxx.xxx.xx/26 dev eth0 proto kernel scope link src 100.xxx.xxx.xx
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 
172.18.0.0/16 dev br-ea7da31d6add proto kernel scope link src 172.18.0.1 

ip link show: (veth4b3a730@if188 is a running container under bridge, stopping it has no help)

3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:28:09:b2:db brd ff:ff:ff:ff:ff:ff
189: veth4b3a730@if188: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default 
    link/ether 6e:68:d8:7a:f7:6c brd ff:ff:ff:ff:ff:ff link-netnsid 0
190: br-ea7da31d6add: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:5b:ee:3b:32 brd ff:ff:ff:ff:ff:ff
200: veth48bdcbb@if199: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ea7da31d6add state UP mode DEFAULT group default 
    link/ether 6e:a0:ed:cb:24:3a brd ff:ff:ff:ff:ff:ff link-netnsid 1
202: veth3da0217@if201: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ea7da31d6add state UP mode DEFAULT group default 
    link/ether 3e:90:45:8c:9a:aa brd ff:ff:ff:ff:ff:ff link-netnsid 2

I don’t know what else it could be than still UFW. You could try to trace network packats (wireshark, tshark, tcpdump). Or if it is still UFW that blocks packages, you could try to disable it temporarily.

I turned off ufw but doesn’t help. I’ll try to trace network packets, thank you anyway!