Can't attach a standalone container to a multi-host overlay network

arkint · November 22, 2021, 7:31pm

Hi,

When I attach a standalone container to an attachable overlay network, I get this error:
docker: Error response from daemon: attaching to network failed, make sure your network options are correct and check manager logs: context deadline exceeded.

Setup:

I have two hosts:

Linux (Mint 19) Swarm manager - docker engine version 20.10.10
macOS (Big Sur) Swarm worker - docker engine version 20.10.10

Steps on Linux host (Swarm manager):

docker swarm init
docker network create -d overlay --attachable mynet

Steps on macOS host (Swarm worker):

Use token to register swarm worker.
docker run -it --rm --name worker1 --hostname worker1 --net mynet -P ubuntu:18.04 bash

when I run the docker container, I get the error I mentioned above.

Please note: If I don’t use “–attachable” parameter when creating the overlay network, I’m able to create service on both machines and works. But nothing works with attachable overlay network.

Any ideas?

Thank you for your help in advance!

meyay · November 22, 2021, 8:54pm

I just tested in on two existin swarm nodes, with exactly the commands you provided.

on node1:

docker network create -d overlay --attachable mynet
docker run -it --rm --name worker1 --hostname worker1 --net mynet -P ubuntu:18.04 bash

on node2

docker run -it --rm --name worker2 --hostname worker2 --net mynet -P ubuntu:18.04 bash

inside container worker1:

root@worker1:/# apt update && apt upgrade
root@worker1:/# apt install iputils-ping
root@worker1:/# ping worker2 -c 5

Result:

inside container worker2:

root@worker2:/# apt update && apt upgrade
root@worker2:/# apt install iputils-ping
root@worker2:/# ping worker1 -c 5

Result:

It works with Docker 20.10.10 on Linux nodes.

I am not sure what prevents it to not work for you.

arkint · November 22, 2021, 9:02pm

Your validation has different setup then mine. It works if the two hosts are linux machines but it does not work if you add a macOS host.

meyay · November 22, 2021, 9:09pm

Yep, I just wanted to point out that it’s not a general problem, as it works amongst linux nodes.

Are you sure all relevant ports are unblocked in the firewalls and traffic can flow freely amongst the nodes? See https://www.bretfisher.com/docker-swarm-firewall-ports/

arkint · November 22, 2021, 9:14pm

No firewalls are running on hosts. The issue happens only when you create an attachable network. As I mentioned before, if I remove “–attachable” parameter when you create a network, I can run services on both hosts without any issue.

rimelek · November 22, 2021, 10:10pm

Are those really Docker 12.10.10? I assumed they are 20.10.10 because I don’t think 12.10.10 ever existed.

Assuming the Docker version is 20.10.10 I tried to do the same as close as it was possible. Used MacBook with Docker desktop as a manager node and an Ubuntu 18.04 Linux laptop for worker.

There were only two differences. My hostname is different so I didn’t use “worker1” but “ta-lxlt” and I used the same for the name of the container. There was no error message. My container started. I copied your command to create the attachable network so that is the same.

I have MacOS Monterey not Big Sur so this is the second differnce.

meyay · November 22, 2021, 10:21pm

Is it maybe related to the 2nd node beeing joined after the network was created?
I haven’t tested that scenario.

arkint · November 22, 2021, 10:33pm

Thank you for pointing out the version issue. it was a typo. It is 20.10.10. Fixed in the post.

I’m still trying to figure out why it can’t work when you have a macOS as worker.

arkint · November 23, 2021, 6:24am

I just upgraded to macOS Monterey. If possible, can you please try making your macOS worker and see what is happening.

rimelek · November 23, 2021, 7:16am

Sorry, I am confused. I had the same MacOS earlier. Do you expect my MacOS to work differently or would like me to test the scenario that @meyay mentioned?

arkint · November 23, 2021, 8:38am

You mentioned that your MacBook was a manager. Can you try to make your MacBook worker and make Linux host manager.

I also tried meyay’s suggestion but didn’t work.

rimelek · November 23, 2021, 11:02am

Oh… sorry. My mistake. The manager was the linux machine. I wrote it wrong before.

arkint · November 23, 2021, 7:54pm

No worries. That’s actually great since we have similar setup now.

When you create the network and while two containers are running on these hosts, can you please send

docker network inspect mynet results from these machines. Let’s see what differences we have there first.

rimelek · November 23, 2021, 8:08pm

On MacBook (worker)

[
    {
        "Name": "mynet",
        "Id": "4e5e2k4gedqxxji32idjb5rnk",
        "Created": "2021-11-23T20:02:51.857877881Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.2.0/24",
                    "Gateway": "10.0.2.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "6b18490be51f9915d9380af7bc3826a173af8841b7ab218b83e55a99f789f59d": {
                "Name": "ta-mb",
                "EndpointID": "9c98198db61cc9ca9852d0d2cf46466b408dbd39a7d7bbc66c3471b5dd4548d2",
                "MacAddress": "02:42:0a:00:02:02",
                "IPv4Address": "10.0.2.2/24",
                "IPv6Address": ""
            },
            "lb-mynet": {
                "Name": "mynet-endpoint",
                "EndpointID": "d0374d8e70c8092343cf8a3afe894cd312aa4879468ec79fa0d51ae651aa3ac2",
                "MacAddress": "02:42:0a:00:02:03",
                "IPv4Address": "10.0.2.3/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4098"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "7045cc388115",
                "IP": "192.168.65.3"
            },
            {
                "Name": "459da1b5cdcc",
                "IP": "192.168.4.102"
            }
        ]
    }
]

On the Linux laptop (manager)

[
    {
        "Name": "mynet",
        "Id": "4e5e2k4gedqxxji32idjb5rnk",
        "Created": "2021-11-23T21:03:32.854005823+01:00",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.2.0/24",
                    "Gateway": "10.0.2.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "de21eceee9842637702f2aac566f95469a9b501b863828f7db9e7deac0fa996f": {
                "Name": "ta-lxlt",
                "EndpointID": "93852af1315a5a5537ed79d4bb72b58dccd3e20bdb806b478185a3fa6fe3b21f",
                "MacAddress": "02:42:0a:00:02:04",
                "IPv4Address": "10.0.2.4/24",
                "IPv6Address": ""
            },
            "lb-mynet": {
                "Name": "mynet-endpoint",
                "EndpointID": "197de03cfa1c599e44adac6561a2f3bd1f34570fc4ead7500026d0fbd3eb50f3",
                "MacAddress": "02:42:0a:00:02:05",
                "IPv4Address": "10.0.2.5/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4098"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "459da1b5cdcc",
                "IP": "192.168.4.102"
            },
            {
                "Name": "7045cc388115",
                "IP": "192.168.65.3"
            }
        ]
    }
]

arkint · November 23, 2021, 9:07pm

Awesome! One more thing: Let’s check ports.

Can you please run the following command from your terminal.

On your MacBook host:
$ nc -vz <linux host IP> 2377
$ nc -vz <linux host IP> 7946
$ nc -vzu <linux host IP> 7946
$ nc -vzu <linux host IP> 4789

On your Linux host:
$ nc -vz <macbook host IP> 2377
$ nc -vz <macbook host IP> 7946
$ nc -vzu <macbook host IP> 7946
$ nc -vzu <macbook host IP> 4789

Also can you please let me know your MacBook and Linux IP addresses. I can read your docker network inspect results better.

rimelek · November 23, 2021, 9:36pm

My network configuration is a little more complicated than the usual so I collected information on multiple networks. I also shared the result of the commands inside the Desktop virtual machine. A gift for you if you didn’t know how to enter the virtual machine using Docker In the VM the output is a little different since it is an alpine Linux. It looks like it does not show anything when the port is not open.

You can see the addresses in the inserted commands below

Linux (manager)

root@ta-lxlt:/home/ta# nc -vz 192.168.4.102 2377
Connection to 192.168.4.102 2377 port [tcp/*] succeeded!

root@ta-lxlt:/home/ta# nc -vz 192.168.4.102 7946
Connection to 192.168.4.102 7946 port [tcp/*] succeeded!

root@ta-lxlt:/home/ta# nc -vzu 192.168.4.102 7946
Connection to 192.168.4.102 7946 port [udp/*] succeeded!

root@ta-lxlt:/home/ta# nc -vzu 192.168.4.102 4789
Connection to 192.168.4.102 4789 port [udp/*] succeeded!

MacBook (worker, WIFI network)

sh-3.2# nc -vz 192.168.4.141 2377
nc: connectx to 192.168.4.141 port 2377 (tcp) failed: Connection refused

sh-3.2# nc -vz 192.168.4.141 7946
nc: connectx to 192.168.4.141 port 7946 (tcp) failed: Connection refused

sh-3.2# nc -vzu 192.168.4.141 7946
Connection to 192.168.4.141 port 7946 [udp/*] succeeded!

sh-3.2# nc -vzu 192.168.4.141 4789
Connection to 192.168.4.141 port 4789 [udp/*] succeeded!

MacBook (worker, LAN network)

sh-3.2# nc -vz 192.168.2.6 2377
nc: connectx to 192.168.2.6 port 2377 (tcp) failed: Connection refused

sh-3.2# nc -vz 192.168.2.6 7946
nc: connectx to 192.168.2.6 port 7946 (tcp) failed: Connection refused

sh-3.2# nc -vzu 192.168.2.6 7946
Connection to 192.168.2.6 port 7946 [udp/*] succeeded!

sh-3.2# nc -vzu 192.168.2.6 4789
Connection to 192.168.2.6 port 4789 [udp/*] succeeded!

MacBook (worker, Desktop VM network)

docker run -it --privileged --pid=host ubuntu:20.04 nsenter -t 1 -m -u -n -i sh

# source: https://gist.github.com/BretFisher/5e1a0c7bcca4c735e716abf62afad389#option-2-easier-use-nsenter-in-priviledged-container

/ # nc -vz 192.168.65.3 2377

/ # nc -vz 192.168.65.3 7946
192.168.65.3 (192.168.65.3:7946) open

/ # nc -vzu 192.168.65.3 7946
192.168.65.3 (192.168.65.3:7946) open

/ # nc -vzu 192.168.65.3 4789
192.168.65.3 (192.168.65.3:4789) open

arkint · November 23, 2021, 10:07pm

That’s great. Our manager and worker open port accesses are the same. I ran out of ideas.

@meyay Any other ideas?

rimelek · November 23, 2021, 10:22pm

What are your nodes statuses? I drained my worker node and got the same error as you. Although I could not run the container on mynet when I created the network without --attachable which is not as surprising as why you could.

Try to run:

docker node ls

on the manager

arkint · November 23, 2021, 10:26pm

It seems normal to me.

ID                            HOSTNAME         STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
lv7t5xl4jtj6ggzxjy68biwkg     docker-desktop   Ready     Active                          20.10.10
x8tukhx05yvpjt4gblz90s26r *   mint             Ready     Active         Leader           20.10.10

arkint · November 23, 2021, 10:46pm

Still no luck. I thought the issue was resolved but it was my bad.

@rimelek When you create a new attachable network on the manager, can you see the new network on the worker machine when you run docker network ls ?

Topic		Replies	Views
How a local container uses a swarm network? General	1	11199	September 13, 2017
Tasks can't communicate via overlay network when running on different swarm nodes Swarm swarm	7	2587	April 4, 2023
Docker Swarm overlay network - containers can't communicate General docker , swarm	7	4978	March 13, 2020
Overlay network with standalone containers, container restart when managers are unavailable General docker , swarm	3	882	March 23, 2021
Need help connecting containers in swarm mode General swarm	6	2646	July 13, 2019

Can't attach a standalone container to a multi-host overlay network

Related Topics