I cannot seem to connect two containers using an overlay network and I run out of the ideas

Docker Hub
1

Hi,
I got a docker swarm on a red hat linux, quite simple setup: one master, and two workers.

Now I created an overlay network:
docker network create --driver overlay demonet --attachable

now I create a two containers which I start with something like this:
docker run --rm -it --network=demonet alpine sh

Now I check the ippaddresses of interfaces within those containers:

Container on worker node:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
14: eth0@if15: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP
    link/ether 02:42:0a:00:01:08 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.8/24 brd 10.0.1.255 scope global eth0
       valid_lft forever preferred_lft forever
16: eth1@if17: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
    link/ether 02:42:ac:13:00:03 brd ff:ff:ff:ff:ff:ff
    inet 172.19.0.3/16 brd 172.19.255.255 scope global eth1
       valid_lft forever preferred_lft forever

and the manager:

/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
67: eth0@if68: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP
    link/ether 02:42:0a:00:01:07 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.7/24 brd 10.0.1.255 scope global eth0
       valid_lft forever preferred_lft forever
69: eth1@if70: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
    link/ether 02:42:ac:13:00:03 brd ff:ff:ff:ff:ff:ff
    inet 172.19.0.3/16 brd 172.19.255.255 scope global eth1
       valid_lft forever preferred_lft forever

now I would like to test some traffic within the overlay network, so I start netcat:

/ #  ip a; nc -v -l -p 8888
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
67: eth0@if68: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP
    link/ether 02:42:0a:00:01:07 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.7/24 brd 10.0.1.255 scope global eth0
       valid_lft forever preferred_lft forever
69: eth1@if70: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
    link/ether 02:42:ac:13:00:03 brd ff:ff:ff:ff:ff:ff
    inet 172.19.0.3/16 brd 172.19.255.255 scope global eth1
       valid_lft forever preferred_lft forever
listening on [::]:8888 ...

and on the other container I will try to connect to that netcat:

/ # nc -w 5 -vvv 10.0.1.7 8888
nc: 10.0.1.7 (10.0.1.7:8888): Operation timed out
sent 0, rcvd 0

but the ping works, though:

/ # ping 10.0.1.7
PING 10.0.1.7 (10.0.1.7): 56 data bytes
64 bytes from 10.0.1.7: seq=0 ttl=64 time=0.583 ms
64 bytes from 10.0.1.7: seq=1 ttl=64 time=0.427 ms

When I inspect the network, it not show anything that would catch my eye.

[
    {
        "Name": "demonet",
        "Id": "qon4ywcktoqt1ma51qswvbwmq",
        "Created": "2023-03-09T15:43:23.60734495+01:00",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.1.0/24",
                    "Gateway": "10.0.1.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "a42d9ab154ce4f50c40e5bd6ab8fb420c2f189c1b6171a35a1ae4094a3597f2a": {
                "Name": "stoic_sanderson",
                "EndpointID": "44e3349c52ef13436775d3f6d94d7bfe148a06941bc7330f0e08a5ab6a56b3f5",
                "MacAddress": "02:42:0a:00:01:07",
                "IPv4Address": "10.0.1.7/24",
                "IPv6Address": ""
            },
            "lb-demonet": {
                "Name": "demonet-endpoint",
                "EndpointID": "6e96eafb200f6b851b7107fce524b8fc414102b5a126d017f06d4d0ea31c1337",
                "MacAddress": "02:42:0a:00:01:05",
                "IPv4Address": "10.0.1.5/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4097"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "d86bb12134be",
                "IP": "10.139.9.11"
            },
            {
                "Name": "bb79ece247ae",
                "IP": "10.139.9.12"
            }
        ]
    }
]

Now the docker container inspect for the containers look like this:

[
    {
        "Id": "a42d9ab154ce4f50c40e5bd6ab8fb420c2f189c1b6171a35a1ae4094a3597f2a",
        "Created": "2023-03-09T14:53:03.645625607Z",
        "Path": "sh",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 32997,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2023-03-09T14:53:04.512149872Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:b2aa39c304c27b96c1fef0c06bee651ac9241d49c4fe34381cab8453f9a89c7d",
        "ResolvConfPath": "/net/si0vm08509/fs0/docker/containers/a42d9ab154ce4f50c40e5bd6ab8fb420c2f189c1b6171a35a1ae4094a3597f2a/resolv.conf",
        "HostnamePath": "/net/si0vm08509/fs0/docker/containers/a42d9ab154ce4f50c40e5bd6ab8fb420c2f189c1b6171a35a1ae4094a3597f2a/hostname",
        "HostsPath": "/net/si0vm08509/fs0/docker/containers/a42d9ab154ce4f50c40e5bd6ab8fb420c2f189c1b6171a35a1ae4094a3597f2a/hosts",
        "LogPath": "/net/si0vm08509/fs0/docker/containers/a42d9ab154ce4f50c40e5bd6ab8fb420c2f189c1b6171a35a1ae4094a3597f2a/a42d9ab154ce4f50c40e5bd6ab8fb420c2f189c1b6171a35a1ae4094a3597f2a-json.log",
        "Name": "/stoic_sanderson",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "demonet",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": true,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "ConsoleSize": [
                46,
                95
            ],
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "private",
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": [],
            "BlkioDeviceWriteBps": [],
            "BlkioDeviceReadIOps": [],
            "BlkioDeviceWriteIOps": [],
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": null,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/net/si0vm08509/fs0/docker/overlay2/48c0bb33b1f58c273ea2f2df0e43b839ce1a01b56a76de801a9b1fe625bc3760-init/diff:/net/si0vm08509/fs0/docker/overlay2/b4183127e2657aaee9163ec6b5bbafec5093ef9712b79c9eb95e997b4df6c9c5/diff",
                "MergedDir": "/net/si0vm08509/fs0/docker/overlay2/48c0bb33b1f58c273ea2f2df0e43b839ce1a01b56a76de801a9b1fe625bc3760/merged",
                "UpperDir": "/net/si0vm08509/fs0/docker/overlay2/48c0bb33b1f58c273ea2f2df0e43b839ce1a01b56a76de801a9b1fe625bc3760/diff",
                "WorkDir": "/net/si0vm08509/fs0/docker/overlay2/48c0bb33b1f58c273ea2f2df0e43b839ce1a01b56a76de801a9b1fe625bc3760/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [],
        "Config": {
            "Hostname": "a42d9ab154ce",
            "Domainname": "",
            "User": "",
            "AttachStdin": true,
            "AttachStdout": true,
            "AttachStderr": true,
            "Tty": true,
            "OpenStdin": true,
            "StdinOnce": true,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "sh"
            ],
            "Image": "alpine",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {}
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "07d297c38d4509bf34d0230f1cfdedce8a14a16fcf8973f8ba87c60c458bf0ff",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/07d297c38d45",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "demonet": {
                    "IPAMConfig": {
                        "IPv4Address": "10.0.1.7"
                    },
                    "Links": null,
                    "Aliases": [
                        "a42d9ab154ce"
                    ],
                    "NetworkID": "qon4ywcktoqt1ma51qswvbwmq",
                    "EndpointID": "44e3349c52ef13436775d3f6d94d7bfe148a06941bc7330f0e08a5ab6a56b3f5",
                    "Gateway": "",
                    "IPAddress": "10.0.1.7",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:0a:00:01:07",
                    "DriverOpts": null
                }
            }
        }
    }
]

and


    {
        "Id": "5d381f4a38ed1d38dd280f3b23dd663ff4abd0bdaa113852ff4eeba707a045df",
        "Created": "2023-03-09T15:08:56.746593503Z",
        "Path": "sh",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 37823,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2023-03-09T15:08:57.836877035Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:b2aa39c304c27b96c1fef0c06bee651ac9241d49c4fe34381cab8453f9a89c7d",
        "ResolvConfPath": "/net/si0vm08510/fs0/docker/containers/5d381f4a38ed1d38dd280f3b23dd663ff4abd0bdaa113852ff4eeba707a045df/resolv.conf",
        "HostnamePath": "/net/si0vm08510/fs0/docker/containers/5d381f4a38ed1d38dd280f3b23dd663ff4abd0bdaa113852ff4eeba707a045df/hostname",
        "HostsPath": "/net/si0vm08510/fs0/docker/containers/5d381f4a38ed1d38dd280f3b23dd663ff4abd0bdaa113852ff4eeba707a045df/hosts",
        "LogPath": "/net/si0vm08510/fs0/docker/containers/5d381f4a38ed1d38dd280f3b23dd663ff4abd0bdaa113852ff4eeba707a045df/5d381f4a38ed1d38dd280f3b23dd663ff4abd0bdaa113852ff4eeba707a045df-json.log",
        "Name": "/zealous_mccarthy",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "demonet",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": true,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "ConsoleSize": [
                46,
                95
            ],
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "private",
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": [],
            "BlkioDeviceWriteBps": [],
            "BlkioDeviceReadIOps": [],
            "BlkioDeviceWriteIOps": [],
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": null,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/net/si0vm08510/fs0/docker/overlay2/a4fd602071e61847c6bc25e4a62eb79f331a48bf93ed5f0e1dd584301a11a401-init/diff:/net/si0vm08510/fs0/docker/overlay2/dbbe5a17c548cc686c416c8c2e20bd849777c053b68092d87187f94132e73698/diff",
                "MergedDir": "/net/si0vm08510/fs0/docker/overlay2/a4fd602071e61847c6bc25e4a62eb79f331a48bf93ed5f0e1dd584301a11a401/merged",
                "UpperDir": "/net/si0vm08510/fs0/docker/overlay2/a4fd602071e61847c6bc25e4a62eb79f331a48bf93ed5f0e1dd584301a11a401/diff",
                "WorkDir": "/net/si0vm08510/fs0/docker/overlay2/a4fd602071e61847c6bc25e4a62eb79f331a48bf93ed5f0e1dd584301a11a401/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [],
        "Config": {
            "Hostname": "5d381f4a38ed",
            "Domainname": "",
            "User": "",
            "AttachStdin": true,
            "AttachStdout": true,
            "AttachStderr": true,
            "Tty": true,
            "OpenStdin": true,
            "StdinOnce": true,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "sh"
            ],
            "Image": "alpine",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {}
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "1fd2998c0e8623c967b8acc358dd8890fdace41133b1a7463dad32a3430a010a",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/1fd2998c0e86",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "demonet": {
                    "IPAMConfig": {
                        "IPv4Address": "10.0.1.10"
                    },
                    "Links": null,
                    "Aliases": [
                        "5d381f4a38ed"
                    ],
                    "NetworkID": "qon4ywcktoqt1ma51qswvbwmq",
                    "EndpointID": "8ee31b08275585b8c3064431d63db54672bda21307467aac46e03409889e1682",
                    "Gateway": "",
                    "IPAddress": "10.0.1.10",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:0a:00:01:0a",
                    "DriverOpts": null
                }
            }
        }
    }
]

I disabled the firewall (actually I did that before installing docker)
I checked the selinux logs wirh:
setroubleshoot - but it gave me no entries.
I performed setenforce 0
I also manipulated with the network ip addresses, but it also led me nowhere…
And there is no firewall in between the nodes…

Now I’m kinda stuck.
I would appreciate for any hints I could try to get that issue resolved.

2

perhaps someone will find that useful:
It seems like there was an issue with vmware network driver dropping the connections because of wrong checksum.
What fixed the problem in my situation was:
ethtool -K ethname tx off