Docker Swarm Service unreachable from manager node

sroumane · January 3, 2023, 12:20pm

Hi to all,

Please be tolerant as it is the first time I post a question here and english is not my native language.

I had setup a docker swarm with 3 manager nodes (no workers node) on 3 debian 11 virtual machines.
This 3 VM are in the subnet:

 Network Address: 10.205.8.0
 Usable Host IP Range: 10.205.8.1 - 10.205.11.254
 Broadcast Address: 10.205.11.255

I had setup my docker swarm for adress-pool on subnet: 192.168.100.0/24 to avoid address overlapping.
No active firewall on the VM.

Docker version 20.10.5+dfsg1, build 55c4c88

When i deploy a simple service (nginx) with one replica only.

docker service create --name mytestweb --replicas 1 --publish published=8083,target=80,mode=ingress nginx

My service is reachable if i target the host node within the container is running:

wget http://apppool-2:8083
Résolution de apppool-2 (apppool-2)… 10.205.11.210
Connexion à apppool-2 (apppool-2)|10.205.11.210|:8083… connecté.
requête HTTP transmise, en attente de la réponse… 200 OK
Taille : 615 [text/html]
Sauvegarde en : « index.html »

index.html                                      100%[=======================================================================================================>]     615  --.-KB/s    ds 0,002s

2023-01-03 12:45:15 (315 KB/s) — « index.html » sauvegardé [615/615]

But if i target another manager node the request is stucked:

 wget http://apppool-1:8083
Résolution de apppool-1 (apppool-1)… 10.205.11.205
Connexion à apppool-1 (apppool-1)|10.205.11.205|:8083… ^C

All the nodes are listening on port 8083 (ipv6 but should listen ipv4 too):

tcp6       0      0 :::8083                 :::*                   LISTEN      757/dockerd

I think my issue is network related but i can’t figure why my service is not reachable from outside or via any manager nodes.

If someone can help me to figure why my configuration is faulty, i will really appreciate ( weeks on this issue).

I found similar problems on forums but none of the discussions allowed me to solve my problem.

I hope i’m not in the wrong place to post this question

meyay · January 3, 2023, 2:06pm

This doesn’t help us to understand what you actually tried so far, so we know what can be ruled out in the further process.

Typically, it’s either a firewall blocking required ports required for the overlay network, a misalignment of the mtu size between host and used mtu size in containers, or a setup where the nodes don’t share low latency network connections (using WAN on multiple locations).

You are using the docker.io package from the Debian repositories. This distribution is maintained and supported by Debian. The distribution may or may not be modified to align with Debian’s philosophy.

If you want to use vanilla Docker, I strongly suggest, to install docker-ce following these instructions: Install Docker Engine on Debian | Docker Docs

sroumane · January 3, 2023, 2:26pm

Hello meyay, thanks for the reply.
I saw on other forum that the problem can be due to network overlap. So i change the address-pool at swarm init with a 192.168.100.0/24 subnet.
Unfortunately this doesn’t work.

My swarm cluster is in private network without any firewall or proxy between nodes. All nodes can be reached from other nodes (ping, wget …).

Actually i can access my service from the node itself but not from another manager.
Can you provide me a process to check if the mtu or the low latency network are involved?

If i have no other choice i will install vanilla docker and try if my configuration is working.

If needed i can post some request result if it can help (iptables, ip addr, …). Just ask me.
Thanks again for the reply

Edit: I have reinstall docker from official docker links, it did’nt resolve my issue.
I use wireshark to check whats going on when i try to contact my service.
I saw this:

"1433","2023-01-03 18:41:23,763050","10.0.0.3","43378","10.0.0.7","8083","TCP","124","43378 → 8083 [SYN] Seq=0 Win=65495 Len=0 MSS=65495 SACK_PERM=1 TSval=1924997737 TSecr=0 WS=128"

"1517","2023-01-03 18:41:24,772482","10.0.0.3","43378","10.0.0.7","8083","TCP","124","[TCP Retransmission] 43378 → 8083 [SYN] Seq=0 Win=65495 Len=0 MSS=65495 SACK_PERM=1 TSval=1924998746 TSecr=0 WS=128"

"1917","2023-01-03 18:41:30,884615","10.0.0.3","43378","10.0.0.7","8083","TCP","124","[TCP Retransmission] 43378 → 8083 [SYN] Seq=0 Win=65495 Len=0 MSS=65495 SACK_PERM=1 TSval=1925004859 TSecr=0 WS=128"

It seem that the packet forwarding from the manager node to the container is done via one of the ingress subnet IP (10.0.0.7, 10.0.0.3). But this IP are not bind to any interface on hosts and i don’t see any packet rewritting in the iptables rules.

Maybe i misunderstood how the swarm ingress network work.
somebody can help me?

meyay · January 3, 2023, 8:27pm

Seems like setting the address-pool didn’t work, as it seems that the default pool 10.0.0.0/8 was used.
Can you share your exact command? It must look smth like this: docker swarm init --default-addr-pool 192.168.100.24/24.

The default address pool mask has 24 bits (per created overlay network), so maybe declaring the default address pool (for all overlay networks) isn’t possible with a 24bit mask.

You could try to resolve your swarm and re-initialize it again using:
docker swarm init --default-addr-pool 192.168.100.24/24 --default-addr-pool-mask-length 26

If it still doesn’t work than this is not the problem responsible that the docker_gwbridges

The overlay networks are spun across the nodes using the docker_gwbridge interface on the host machines, which will be used to transport the docker networks encapulated in vxlan networks.

When your nodes are in the same local network, I am sure they are low latency networks. The same holds true for availability zones in the same region of a cloud hyperscaler.

You can identify the mtu using this command: ip addr | grep mtu

sroumane · January 4, 2023, 10:13am

I had setup a new cluster with the comand :

docker swarm init --default-addr-pool 192.168.100.24/24 --default-addr-pool-mask-length 26

and i join two node as manager (so a cluster of 3 nodes: apppool-1, apppool-2, apppool-3, all as managers)

The 3 nodes ip are:

apppool-1 10.205.11.205/22
apppool-2 10.205.11.210/22
apppool-3 10.205.11.211/22

and deploy a simple nginx service:

docker service create --name mytestweb --replicas 1 --publish published=8083,target=80,mode=ingress nginx

docker service ls
ID             NAME        MODE         REPLICAS   IMAGE          PORTS
jc8h5ztsy5gk   mytestweb   replicated   1/1        nginx:latest   *:8083->80/tcp

The service is running on the apppool-1

root@apppool-1:~# docker ps
CONTAINER ID   IMAGE          COMMAND                  CREATED          STATUS          PORTS     NAMES
74c7d6ee4d70   nginx:latest   "/docker-entrypoint.…"   18 minutes ago   Up 18 minutes   80/tcp    mytestweb.1.uw5ie6z4hveu1ahu7806l88hq

Some “inspect” comand:

docker network ls
NETWORK ID     NAME              DRIVER    SCOPE
8f4480d8b21d   bridge            bridge    local
612c3c3897de   docker_gwbridge   bridge    local
7028cf00471d   host              host      local
nqqth5f1l98o   ingress           overlay   swarm
98fb2472ad2f   none              null      local

Node 1 (apppool-1)

root@apppool-1:~# docker network inspect docker_gwbridge
[
    {
        "Name": "docker_gwbridge",
        "Id": "8facc7ac6d404ef5de5abc41c066ab16b9f655129353329b2205d719b8474555",
        "Created": "2022-07-11T15:58:45.571785528+02:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "74c7d6ee4d70ef7cbaaedb5ba221b41743dd49d0a45c7ad78312bf8bd47c6774": {
                "Name": "gateway_2f2dae3bca89",
                "EndpointID": "871864f70c087c3245cbd079793d1c7ed97ac4b218e3eff61b1a47109e94a946",
                "MacAddress": "02:42:ac:12:00:03",
                "IPv4Address": "172.18.0.3/16",
                "IPv6Address": ""
            },
            "ingress-sbox": {
                "Name": "gateway_ingress-sbox",
                "EndpointID": "e6f12dc85f6cbc45f72f8b63fe7ecc24e6dd1416a8e14b02d8f2702e92d4c1b6",
                "MacAddress": "02:42:ac:12:00:02",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.enable_icc": "false",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.name": "docker_gwbridge"
        },
        "Labels": {}
    }
]

root@apppool-1:~# docker network inspect ingress
[
    {
        "Name": "ingress",
        "Id": "nqqth5f1l98ou10ompfshd3ik",
        "Created": "2023-01-04T10:28:14.77374724+01:00",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "192.168.100.0/26",
                    "Gateway": "192.168.100.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": true,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "74c7d6ee4d70ef7cbaaedb5ba221b41743dd49d0a45c7ad78312bf8bd47c6774": {
                "Name": "mytestweb.1.uw5ie6z4hveu1ahu7806l88hq",
                "EndpointID": "8c772610c1a5aaf015bf29022d069fa8ab644e198cfb6a344cc725685dea3ea9",
                "MacAddress": "02:42:c0:a8:64:06",
                "IPv4Address": "192.168.100.6/26",
                "IPv6Address": ""
            },
            "ingress-sbox": {
                "Name": "ingress-endpoint",
                "EndpointID": "55f53eeb1bc4be5fbe96940e99090c35719f4d805b7168d9d26d09ea72991f2e",
                "MacAddress": "02:42:c0:a8:64:02",
                "IPv4Address": "192.168.100.2/26",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4096"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "2abf3f8dbe87",
                "IP": "10.205.11.205"
            },
            {
                "Name": "0240e192ad2d",
                "IP": "10.205.11.210"
            },
            {
                "Name": "a553104340c5",
                "IP": "10.205.11.211"
            }
        ]
    }
]

root@apppool-1:~# docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "f375fa547dae93b34b0fae10267f5e7a61f26f5227bf1f591ce216907d917c88",
        "Created": "2023-01-03T16:15:50.969878515+01:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

Node 2 (apppool-2)

root@apppool-2:~# docker network inspect docker_gwbridge
[
    {
        "Name": "docker_gwbridge",
        "Id": "63ed9fd2c0bf1c0e065b79ff526cf6db67316995b83f1810ea4e9483e52e4def",
        "Created": "2022-08-12T16:30:53.671887535+02:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "ingress-sbox": {
                "Name": "gateway_ingress-sbox",
                "EndpointID": "241d4638bebe8bc1d7ed9dd79052411d1d5af00cb85cde961ea49bc9460ea38e",
                "MacAddress": "02:42:ac:12:00:02",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.enable_icc": "false",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.name": "docker_gwbridge"
        },
        "Labels": {}
    }
]

root@apppool-2:~# docker network inspect ingress
[
    {
        "Name": "ingress",
        "Id": "nqqth5f1l98ou10ompfshd3ik",
        "Created": "2023-01-04T10:28:34.146028458+01:00",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "192.168.100.0/26",
                    "Gateway": "192.168.100.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": true,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "ingress-sbox": {
                "Name": "ingress-endpoint",
                "EndpointID": "a90cfc98376691b2f6f7ea663918a50b3dbc08515ecce1d505cbea60007d09a0",
                "MacAddress": "02:42:c0:a8:64:03",
                "IPv4Address": "192.168.100.3/26",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4096"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "2abf3f8dbe87",
                "IP": "10.205.11.205"
            },
            {
                "Name": "0240e192ad2d",
                "IP": "10.205.11.210"
            },
            {
                "Name": "a553104340c5",
                "IP": "10.205.11.211"
            }
        ]
    }
]

root@apppool-2:~# docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "fe99090bbb9f002fb34b44f845774b245bfa6d9fa26f656639493baf8f6288d5",
        "Created": "2023-01-03T17:03:40.709820594+01:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

Node 3 (apppool-3)

root@apppool-3:~# docker network inspect docker_gwbridge
[
    {
        "Name": "docker_gwbridge",
        "Id": "612c3c3897dea2dacf5b6b37de53d49bf66d7120cff68a88ffc74238691b9e96",
        "Created": "2022-08-17T11:00:54.913689601+02:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "ingress-sbox": {
                "Name": "gateway_ingress-sbox",
                "EndpointID": "ca13bb2c82747d8e39c03881d3042e5ba11153289c03541622e24431ec1c6cd9",
                "MacAddress": "02:42:ac:12:00:02",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.enable_icc": "false",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.name": "docker_gwbridge"
        },
        "Labels": {}
    }
]

root@apppool-3:~# docker network inspect ingress
[
    {
        "Name": "ingress",
        "Id": "nqqth5f1l98ou10ompfshd3ik",
        "Created": "2023-01-04T10:29:17.422388709+01:00",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "192.168.100.0/26",
                    "Gateway": "192.168.100.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": true,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "ingress-sbox": {
                "Name": "ingress-endpoint",
                "EndpointID": "4787231486a276dc71de6fc02d25bf8c9d7e6c573a9cfd49fd846a68e55cf866",
                "MacAddress": "02:42:c0:a8:64:04",
                "IPv4Address": "192.168.100.4/26",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4096"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "2abf3f8dbe87",
                "IP": "10.205.11.205"
            },
            {
                "Name": "0240e192ad2d",
                "IP": "10.205.11.210"
            },
            {
                "Name": "a553104340c5",
                "IP": "10.205.11.211"
            }
        ]
    }
]

root@apppool-3:~# docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "8f4480d8b21d9787e31895f9369c01bae9df640b82274313e68a5239bb7c6708",
        "Created": "2023-01-03T18:17:14.324188094+01:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

The MTU:

Node 1 (apppool-1)

 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:b4:f7:71 brd ff:ff:ff:ff:ff:ff
    altname enp11s0
    inet 10.205.11.205/22 brd 10.205.11.255 scope global ens192
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:feb4:f771/64 scope link
       valid_lft forever preferred_lft forever
3: docker_gwbridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:f8:5f:f7:bd brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global docker_gwbridge
       valid_lft forever preferred_lft forever
    inet6 fe80::42:f8ff:fe5f:f7bd/64 scope link
       valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:3b:3c:bb:5e brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
37: vetha7422a1@if36: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker_gwbridge state UP group default
    link/ether ca:da:25:92:ad:1d brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::c8da:25ff:fe92:ad1d/64 scope link
       valid_lft forever preferred_lft forever
41: vethe8146b1@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker_gwbridge state UP group default
    link/ether ce:c6:ac:5b:64:f3 brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::ccc6:acff:fe5b:64f3/64 scope link
       valid_lft forever preferred_lft forever

Node 2 (apppool-2)

root@apppool-2:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:b4:87:0b brd ff:ff:ff:ff:ff:ff
    altname enp11s0
    inet 10.205.11.210/22 brd 10.205.11.255 scope global ens192
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:feb4:870b/64 scope link
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:00:62:32:91 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:ff:fe62:3291/64 scope link
       valid_lft forever preferred_lft forever
4: docker_gwbridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:57:a0:f9:6d brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global docker_gwbridge
       valid_lft forever preferred_lft forever
    inet6 fe80::42:57ff:fea0:f96d/64 scope link
       valid_lft forever preferred_lft forever
48: vethd7408f9@if47: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker_gwbridge state UP group default
    link/ether be:ec:c5:4b:d8:35 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::bcec:c5ff:fe4b:d835/64 scope link
       valid_lft forever preferred_lft forever

Node 3 (apppool-3)

root@apppool-3:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:b4:47:9f brd ff:ff:ff:ff:ff:ff
    altname enp11s0
    inet 10.205.11.211/22 brd 10.205.11.255 scope global ens192
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:feb4:479f/64 scope link
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:13:21:fb:b3 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
4: docker_gwbridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:0a:40:fc:00 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global docker_gwbridge
       valid_lft forever preferred_lft forever
    inet6 fe80::42:aff:fe40:fc00/64 scope link
       valid_lft forever preferred_lft forever
10: veth9e72e17@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker_gwbridge state UP group default
    link/ether 0e:11:b7:97:dd:92 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::c11:b7ff:fe97:dd92/64 scope link
       valid_lft forever preferred_lft forever

Unfortunately the service is not reachable via the nodes where it is not running.

root@apppool-3:~# wget http://apppool-3:8083
--2023-01-04 10:46:40--  http://apppool-3:8083/
Résolution de apppool-3 (apppool-3)… 10.205.11.211
Connexion à apppool-3 (apppool-3)|10.205.11.211|:8083… ^C
root@apppool-3:~# wget http://apppool-2:8083
--2023-01-04 10:46:47--  http://apppool-2:8083/
Résolution de apppool-2 (apppool-2)… 10.205.11.210
Connexion à apppool-2 (apppool-2)|10.205.11.210|:8083… ^C
root@apppool-3:~# wget http://apppool-1:8083
--2023-01-04 10:46:54--  http://apppool-1:8083/
Résolution de apppool-1 (apppool-1)… 10.205.11.205
Connexion à apppool-1 (apppool-1)|10.205.11.205|:8083… connecté.
requête HTTP transmise, en attente de la réponse… 200 OK
Taille : 615 [text/html]
Sauvegarde en : « index.html.1 »

index.html.1                                      100%[==========================================================================================================>]     615  --.-KB/s    ds 0s

2023-01-04 10:46:54 (63,9 MB/s) — « index.html.1 » sauvegardé [615/615]

Edit: To complete the analys i run a tcpdump when trying to reach my service, the ingress IP address seem to be used to connect to the others nodes (192.168.100.X). I don’t know why docker use this ip address wich is not bind to any network interface

"917","2023-01-04 11:51:23,618404","192.168.100.4","37288","192.168.100.6","8083","TCP","124","[TCP Retransmission] 37288 → 8083 [SYN] Seq=0 Win=65495 Len=0 MSS=65495 SACK_PERM=1 TSval=874079223 TSecr=0 WS=128"

"1061","2023-01-04 11:51:25,634397","192.168.100.4","37288","192.168.100.6","8083","TCP","124","[TCP Retransmission] 37288 → 8083 [SYN] Seq=0 Win=65495 Len=0 MSS=65495 SACK_PERM=1 TSval=874081239 TSecr=0 WS=128"

"1393","2023-01-04 11:51:29,826388","192.168.100.4","37288","192.168.100.6","8083","TCP","124","[TCP Retransmission] 37288 → 8083 [SYN] Seq=0 Win=65495 Len=0 MSS=65495 SACK_PERM=1 TSval=874085431 TSecr=0 WS=128"

meyay · January 4, 2023, 5:31pm

Of course this should have been 192.168.100.0/24 instead 192.168.100.24/24, which technically shouldn’t matter as the resulting address pool still is the same. It just looks kind of looks “unclean”.

I am going to take a look at the rest late.

meyay · January 4, 2023, 6:04pm

It is the ip of the service task in the ingress network, check your output of root@apppool-1:~# docker network inspect ingress and you will find it there. I tried to explain why the network is not bound to any of the host’s network interfaces: overlay networks are vxlan’s, they are encapsulated in the traffic of the docker_gwbridge network. You won’t see a host interface with that ip.

You mtu size looks fine.
You say there is no firewall at all. (By any chance are you on cloud platform or a vm platform that uses security groups or a firewall on the outside of the vm?)
Your nodes are in the same lan.

So the only thing left that comes to mind is missing kernel modules?

curl -L https://github.com/moby/moby/raw/master/contrib/check-config.sh | bash

Those are the factors that come to mind, of course there could be environment specific things we have no idea of. I can say I never experienced this problem on AWS, on ESXi or Proxmox

sroumane · January 4, 2023, 6:25pm

Hello Metin, and thanks again for your help.

My nodes are Debian 11 VM hosted by an ESXi. They are plugged to a VSwitch in the same LAN.

I’m sure that no external firewall are involved. I have on the same infrastructure setup a lot of tools with UDP and TCP port binding like NFS, IPSEC, DDS without any firewall issue.

I post the iptables rules created by docker itself:

root@apppool-1:~# iptables -S
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-INGRESS
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-INGRESS
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o docker_gwbridge -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker_gwbridge -j DOCKER
-A FORWARD -i docker_gwbridge ! -o docker_gwbridge -j ACCEPT
-A FORWARD -i docker_gwbridge -o docker_gwbridge -j DROP
-A DOCKER-INGRESS -p tcp -m tcp --dport 8083 -j ACCEPT
-A DOCKER-INGRESS -p tcp -m state --state RELATED,ESTABLISHED -m tcp --sport 8083 -j ACCEPT
-A DOCKER-INGRESS -j RETURN
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i docker_gwbridge ! -o docker_gwbridge -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o docker_gwbridge -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN

iptables -L -v -n
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
   36  4860 DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0
   36  4860 DOCKER-INGRESS  all  --  *      *       0.0.0.0/0            0.0.0.0/0
 9245 9229K DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0
 6207 9067K ACCEPT     all  --  *      docker_gwbridge  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      docker_gwbridge  0.0.0.0/0            0.0.0.0/0
 3038  161K ACCEPT     all  --  docker_gwbridge !docker_gwbridge  0.0.0.0/0            0.0.0.0/0
    0     0 DROP       all  --  docker_gwbridge docker_gwbridge  0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER-INGRESS (1 references)
 pkts bytes target     prot opt in     out     source               destination
   21  1497 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8083
   15  3363 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:8083
 9245 9229K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0
 3038  161K DOCKER-ISOLATION-STAGE-2  all  --  docker_gwbridge !docker_gwbridge  0.0.0.0/0            0.0.0.0/0
 9245 9229K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       all  --  *      docker0  0.0.0.0/0            0.0.0.0/0
    0     0 DROP       all  --  *      docker_gwbridge  0.0.0.0/0            0.0.0.0/0
 3038  161K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination
 9491 9246K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

And the result of the script

check-config.sh

root@apppool-1:~# sh check-config.sh
warning: /proc/config.gz does not exist, searching other paths for kernel config ...
info: reading kernel config from /boot/config-5.10.0-14-amd64 ...

Generally Necessary:
- cgroup hierarchy: cgroupv2
  Controllers:
  - cpu: available
  - cpuset: available
  - io: available
  - memory: available
  - pids: available
- apparmor: enabled and tools installed
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_NETFILTER_XT_MARK: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_POSIX_MQUEUE: enabled
- CONFIG_CGROUP_BPF: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_SECCOMP_FILTER: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: enabled
    (cgroup swap accounting is currently enabled)
- CONFIG_LEGACY_VSYSCALL_NONE: enabled
    (containers using eglibc <= 2.13 will not work. Switch to
     "CONFIG_VSYSCALL_[NATIVE|EMULATE]" or use "vsyscall=[native|emulate]"
     on kernel command line. Note that this will disable ASLR for the,
     VDSO which may assist in exploiting security vulnerabilities.)
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled (as module)
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: missing
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_SECURITY_SELINUX: enabled
- CONFIG_SECURITY_APPARMOR: enabled
- CONFIG_EXT4_FS: enabled (as module)
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
    - CONFIG_BRIDGE_VLAN_FILTERING: enabled
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled (as module)
      - CONFIG_CRYPTO_GCM: enabled (as module)
      - CONFIG_CRYPTO_SEQIV: enabled (as module)
      - CONFIG_CRYPTO_GHASH: enabled (as module)
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled (as module)
      - CONFIG_XFRM_ALGO: enabled (as module)
      - CONFIG_INET_ESP: enabled (as module)
  - "ipvlan":
    - CONFIG_IPVLAN: enabled (as module)
  - "macvlan":
    - CONFIG_MACVLAN: enabled (as module)
    - CONFIG_DUMMY: enabled (as module)
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_FTP: enabled (as module)
    - CONFIG_NF_NAT_TFTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled (as module)
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled (as module)
    - CONFIG_DM_THIN_PROVISIONING: enabled (as module)
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

meyay · January 4, 2023, 6:39pm

I never had to touch any iptables rules to make swarm mode work.
Still, the iptables rules look like on my homelab swarm cluster.
Your kernel looks fine as well: no overlay releated modules are missing.The missing storage Driver aufs isn’t related, and become irrelevant since overlay is available. CONFIG_RT_GROUP_SCHED isn’t relevant for that issue either.

It’s been a minute since I used Swarm on ESXi, I kind of remember my vSwitch was running in Promiscuous mode. I am not sure if it will help in this situation, but at least it’s worth trying.

Your whole setup on the nodes looks good.

sroumane · January 4, 2023, 7:01pm

I will try to enable the Promiscuous mode on my vswitch (if not already enabled).
I will post the result tomorrow.

sroumane · January 5, 2023, 10:08am

I set the Promiscous mode on my VSwitch, unfortunately this not solved my problem

sroumane · January 5, 2023, 10:47am

I put the tcpdump filtrered on the port 8083. (as image, but i can export to csv if needed)

Someone can help me to explain all the TCP retransmission packet?

meyay · January 5, 2023, 5:37pm

It seems the tcp packages never received an ACK, and therefor are retransmitted by the sender. Are you sure there is no ip or ip-range collision? There must be a reason why transfer is flacky. Can you add the mac-addresses as to the output as well?

sroumane · January 5, 2023, 5:48pm

The same tcpdump with Hardware address

meyay · January 5, 2023, 6:31pm

It’s odd that only traffic from 10.205.11.211 to 10.205.11.210 seem to be affected (and of course traffic piggybacking on that connection).

I hope someone comes around who will be able to help you.

sroumane · January 5, 2023, 7:10pm

The 10.205.11.211 is the apppool-3. It is the node where i had launch the test (wget).
The whireshark sample from above is dump from 10.205.11.205 apppool-1 (the node where the container is running).
Below the dump of:

apppool-3 (10.205.11.211)

sroumane · January 5, 2023, 7:11pm

apppool-2 (10.205.11.210)

image1920×642 94.3 KB

The dumps seems to be the same.

Do you know a forum or a channel (discord or others) where i can find more help?

sroumane · January 5, 2023, 7:29pm

Ok ok ok i got it.

I’m a little ashamed, but I’m very disappointed with the lack of documentation on instantiating a swarm cluster on a vmware architecture.

Port 4789 used by docker swarm for data exchange is reserved and restricted by vmware.

So I changed the “data-path-port” to initializing the swarm cluster and everything works now.

docker swarm init --default-addr-pool 192.168.100.0/24 --default-addr-pool-mask-length 26 --data-path-port=7789

The Vmware link: Network port requirements for VMware NSX for vSphere 6.x (2079386)

I hope this thread will help future users.

Thank @meyay for your help (So you are rigth, it look like a collision cause vmware keep the data on the data-port)

meyay · January 5, 2023, 7:34pm

Back in the day when I was running swarm on ESXi, of course I didn’t use NXS and therefor didn’t have the problem.

I am glad this mysterious issue is finally resolved. Thank you for sharing the final solution!

Topic		Replies	Views
In SWARM not able to access service from worker node Swarm swarm	4	3204	February 1, 2023
All deployed services within a swarm are unreachable, while containers deployed normally work fine General swarm	4	7041	December 20, 2020
Docker swarm overlay network not working General docker	8	4718	April 25, 2020
Unable to reach docker swarm services externally Swarm	1	32	August 9, 2024
Docker Swarm Ingress Network Load Balancing Issue Swarm	4	1497	September 22, 2020

Docker Swarm Service unreachable from manager node

Related topics