Docker Swarm network overlay issue

General
1

I created 3 ubuntu 20.04 VM in Proxmox VE 7 for the docker swarm. I tried to follow the site Docker Swarm - Documentation to setup the Portainer on my Swarm. However I can’t browse any IP address of the ubuntu VMs to access Portainer site to setup the docker container.

Something is go wrong on the overlay network on my swarm. It looks like the ingress not enable. Please see the below network inspect for portainer_agent_network.

And I found that all swarm machines not listen the port 4789. When I run the command sudo lsof -i:4789 , it shows nothing.

Does anyone help me to troubleshoot it? What is going wrong on my docker swarm?

ubuntu@swarm01:~$ docker network inspect portainer_agent_network 
[
    {
        "Name": "portainer_agent_network",
        "Id": "tzm9sx2zifgaxhpmrd8xk7gti",
        "Created": "2021-08-07T14:24:33.835202371Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.11.0/24",
                    "Gateway": "10.0.11.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "54a9491638f699fc6441961b04b91c8ca923bd8e4980dbe36651fa2618cdbe2c": {
                "Name": "portainer_portainer.1.fd5m3wvccnxrl43iwst2imwti",
                "EndpointID": "4537774ec3c146843b48ab89707df7b04a6a76880af85dbe025fcc4d7422262c",
                "MacAddress": "02:42:0a:00:0b:0c",
                "IPv4Address": "10.0.11.12/24",
                "IPv6Address": ""
            },
            "83044215d796b649ee8fc78be2d1364c80646448db3a933ee9a48ff0b0b7fe24": {
                "Name": "portainer_agent.idso1hec0iqiyvm1jhu1iaoq1.qidcsempp75po4znf1c7pj09r",
                "EndpointID": "dfdd91e83969150ea70674b9ea998690b47a6abf113c9a644315d641c6b68e1c",
                "MacAddress": "02:42:0a:00:0b:05",
                "IPv4Address": "10.0.11.5/24",
                "IPv6Address": ""
            },
            "lb-portainer_agent_network": {
                "Name": "portainer_agent_network-endpoint",
                "EndpointID": "be0b5a8bdda9ccae975314fad1424d96e3c57763b1c145f4a67e286f54300195",
                "MacAddress": "02:42:0a:00:0b:08",
                "IPv4Address": "10.0.11.8/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4107"
        },
        "Labels": {
            "com.docker.stack.namespace": "portainer"
        },
        "Peers": [
            {
                "Name": "0589007b93f4",
                "IP": "10.0.0.241"
            },
            {
                "Name": "be83a3dd8fbd",
                "IP": "10.0.0.242"
            },
            {
                "Name": "f937ea4c2dbf",
                "IP": "10.0.0.243"
            }
        ]
    }
]

ubuntu@swarm01:~$ sudo lsof -i:7946
COMMAND PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
dockerd 451 root   30u  IPv6  14558      0t0  TCP *:7946 (LISTEN)
dockerd 451 root   32u  IPv6  14559      0t0  UDP *:7946 
ubuntu@swarm01:~$ sudo lsof -i:4789
ubuntu@swarm01:~$

Thanks with the best regards

Patrick Lee