Docker Community Forums

Share and learn in the Docker community.

Can't attach a standalone container to a multi-host overlay network

Hi,

When I attach a standalone container to an attachable overlay network, I get this error:
docker: Error response from daemon: attaching to network failed, make sure your network options are correct and check manager logs: context deadline exceeded.

Setup:

I have two hosts:

  1. Linux (Mint 19) Swarm manager - docker engine version 20.10.10
  2. macOS (Big Sur) Swarm worker - docker engine version 20.10.10

Steps on Linux host (Swarm manager):

  1. docker swarm init
  2. docker network create -d overlay --attachable mynet

Steps on macOS host (Swarm worker):

  1. Use token to register swarm worker.
  2. docker run -it --rm --name worker1 --hostname worker1 --net mynet -P ubuntu:18.04 bash

when I run the docker container, I get the error I mentioned above.

Please note: If I don’t use “–attachable” parameter when creating the overlay network, I’m able to create service on both machines and works. But nothing works with attachable overlay network.

Any ideas?

Thank you for your help in advance!

I just tested in on two existin swarm nodes, with exactly the commands you provided.

on node1:

docker network create -d overlay --attachable mynet
docker run -it --rm --name worker1 --hostname worker1 --net mynet -P ubuntu:18.04 bash

on node2

docker run -it --rm --name worker2 --hostname worker2 --net mynet -P ubuntu:18.04 bash

inside container worker1:

root@worker1:/# apt update && apt upgrade
root@worker1:/# apt install iputils-ping
root@worker1:/# ping worker2 -c 5

Result:

inside container worker2:

root@worker2:/# apt update && apt upgrade
root@worker2:/# apt install iputils-ping
root@worker2:/# ping worker1 -c 5

Result:

It works with Docker 20.10.10 on Linux nodes.

I am not sure what prevents it to not work for you.

Your validation has different setup then mine. It works if the two hosts are linux machines but it does not work if you add a macOS host.

Yep, I just wanted to point out that it’s not a general problem, as it works amongst linux nodes.

Are you sure all relevant ports are unblocked in the firewalls and traffic can flow freely amongst the nodes? See https://www.bretfisher.com/docker-swarm-firewall-ports/

1 Like

No firewalls are running on hosts. The issue happens only when you create an attachable network. As I mentioned before, if I remove “–attachable” parameter when you create a network, I can run services on both hosts without any issue.

Are those really Docker 12.10.10? I assumed they are 20.10.10 because I don’t think 12.10.10 ever existed.

Assuming the Docker version is 20.10.10 I tried to do the same as close as it was possible. Used MacBook with Docker desktop as a manager node and an Ubuntu 18.04 Linux laptop for worker.

There were only two differences. My hostname is different so I didn’t use “worker1” but “ta-lxlt” and I used the same for the name of the container. There was no error message. My container started. I copied your command to create the attachable network so that is the same.

I have MacOS Monterey not Big Sur so this is the second differnce.

Is it maybe related to the 2nd node beeing joined after the network was created?
I haven’t tested that scenario.

Thank you for pointing out the version issue. it was a typo. It is 20.10.10. Fixed in the post.

I’m still trying to figure out why it can’t work when you have a macOS as worker.

I just upgraded to macOS Monterey. If possible, can you please try making your macOS worker and see what is happening.

Sorry, I am confused. I had the same MacOS earlier. Do you expect my MacOS to work differently or would like me to test the scenario that @meyay mentioned?

You mentioned that your MacBook was a manager. Can you try to make your MacBook worker and make Linux host manager.

I also tried meyay’s suggestion but didn’t work.

Oh… sorry. My mistake. The manager was the linux machine. I wrote it wrong before.

1 Like

No worries. That’s actually great since we have similar setup now.

When you create the network and while two containers are running on these hosts, can you please send

docker network inspect mynet results from these machines. Let’s see what differences we have there first.

On MacBook (worker)

[
    {
        "Name": "mynet",
        "Id": "4e5e2k4gedqxxji32idjb5rnk",
        "Created": "2021-11-23T20:02:51.857877881Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.2.0/24",
                    "Gateway": "10.0.2.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "6b18490be51f9915d9380af7bc3826a173af8841b7ab218b83e55a99f789f59d": {
                "Name": "ta-mb",
                "EndpointID": "9c98198db61cc9ca9852d0d2cf46466b408dbd39a7d7bbc66c3471b5dd4548d2",
                "MacAddress": "02:42:0a:00:02:02",
                "IPv4Address": "10.0.2.2/24",
                "IPv6Address": ""
            },
            "lb-mynet": {
                "Name": "mynet-endpoint",
                "EndpointID": "d0374d8e70c8092343cf8a3afe894cd312aa4879468ec79fa0d51ae651aa3ac2",
                "MacAddress": "02:42:0a:00:02:03",
                "IPv4Address": "10.0.2.3/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4098"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "7045cc388115",
                "IP": "192.168.65.3"
            },
            {
                "Name": "459da1b5cdcc",
                "IP": "192.168.4.102"
            }
        ]
    }
]

On the Linux laptop (manager)

[
    {
        "Name": "mynet",
        "Id": "4e5e2k4gedqxxji32idjb5rnk",
        "Created": "2021-11-23T21:03:32.854005823+01:00",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.2.0/24",
                    "Gateway": "10.0.2.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "de21eceee9842637702f2aac566f95469a9b501b863828f7db9e7deac0fa996f": {
                "Name": "ta-lxlt",
                "EndpointID": "93852af1315a5a5537ed79d4bb72b58dccd3e20bdb806b478185a3fa6fe3b21f",
                "MacAddress": "02:42:0a:00:02:04",
                "IPv4Address": "10.0.2.4/24",
                "IPv6Address": ""
            },
            "lb-mynet": {
                "Name": "mynet-endpoint",
                "EndpointID": "197de03cfa1c599e44adac6561a2f3bd1f34570fc4ead7500026d0fbd3eb50f3",
                "MacAddress": "02:42:0a:00:02:05",
                "IPv4Address": "10.0.2.5/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4098"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "459da1b5cdcc",
                "IP": "192.168.4.102"
            },
            {
                "Name": "7045cc388115",
                "IP": "192.168.65.3"
            }
        ]
    }
]
1 Like

Awesome! One more thing: Let’s check ports.

Can you please run the following command from your terminal.

On your MacBook host:
$ nc -vz <linux host IP> 2377
$ nc -vz <linux host IP> 7946
$ nc -vzu <linux host IP> 7946
$ nc -vzu <linux host IP> 4789

On your Linux host:
$ nc -vz <macbook host IP> 2377
$ nc -vz <macbook host IP> 7946
$ nc -vzu <macbook host IP> 7946
$ nc -vzu <macbook host IP> 4789

Also can you please let me know your MacBook and Linux IP addresses. I can read your docker network inspect results better.

My network configuration is a little more complicated than the usual so I collected information on multiple networks. I also shared the result of the commands inside the Desktop virtual machine. A gift for you if you didn’t know how to enter the virtual machine using Docker :slight_smile: In the VM the output is a little different since it is an alpine Linux. It looks like it does not show anything when the port is not open.

You can see the addresses in the inserted commands below

Linux (manager)

root@ta-lxlt:/home/ta# nc -vz 192.168.4.102 2377
Connection to 192.168.4.102 2377 port [tcp/*] succeeded!

root@ta-lxlt:/home/ta# nc -vz 192.168.4.102 7946
Connection to 192.168.4.102 7946 port [tcp/*] succeeded!

root@ta-lxlt:/home/ta# nc -vzu 192.168.4.102 7946
Connection to 192.168.4.102 7946 port [udp/*] succeeded!

root@ta-lxlt:/home/ta# nc -vzu 192.168.4.102 4789
Connection to 192.168.4.102 4789 port [udp/*] succeeded!

MacBook (worker, WIFI network)

sh-3.2# nc -vz 192.168.4.141 2377
nc: connectx to 192.168.4.141 port 2377 (tcp) failed: Connection refused

sh-3.2# nc -vz 192.168.4.141 7946
nc: connectx to 192.168.4.141 port 7946 (tcp) failed: Connection refused

sh-3.2# nc -vzu 192.168.4.141 7946
Connection to 192.168.4.141 port 7946 [udp/*] succeeded!

sh-3.2# nc -vzu 192.168.4.141 4789
Connection to 192.168.4.141 port 4789 [udp/*] succeeded!

MacBook (worker, LAN network)

sh-3.2# nc -vz 192.168.2.6 2377
nc: connectx to 192.168.2.6 port 2377 (tcp) failed: Connection refused

sh-3.2# nc -vz 192.168.2.6 7946
nc: connectx to 192.168.2.6 port 7946 (tcp) failed: Connection refused

sh-3.2# nc -vzu 192.168.2.6 7946
Connection to 192.168.2.6 port 7946 [udp/*] succeeded!

sh-3.2# nc -vzu 192.168.2.6 4789
Connection to 192.168.2.6 port 4789 [udp/*] succeeded!

MacBook (worker, Desktop VM network)

docker run -it --privileged --pid=host ubuntu:20.04 nsenter -t 1 -m -u -n -i sh

# source: https://gist.github.com/BretFisher/5e1a0c7bcca4c735e716abf62afad389#option-2-easier-use-nsenter-in-priviledged-container
/ # nc -vz 192.168.65.3 2377

/ # nc -vz 192.168.65.3 7946
192.168.65.3 (192.168.65.3:7946) open

/ # nc -vzu 192.168.65.3 7946
192.168.65.3 (192.168.65.3:7946) open

/ # nc -vzu 192.168.65.3 4789
192.168.65.3 (192.168.65.3:4789) open
1 Like

That’s great. Our manager and worker open port accesses are the same. I ran out of ideas.

@meyay Any other ideas?

What are your nodes statuses? I drained my worker node and got the same error as you. Although I could not run the container on mynet when I created the network without --attachable which is not as surprising as why you could.

Try to run:

docker node ls

on the manager

It seems normal to me.

ID                            HOSTNAME         STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
lv7t5xl4jtj6ggzxjy68biwkg     docker-desktop   Ready     Active                          20.10.10
x8tukhx05yvpjt4gblz90s26r *   mint             Ready     Active         Leader           20.10.10

Still no luck. I thought the issue was resolved but it was my bad.

@rimelek When you create a new attachable network on the manager, can you see the new network on the worker machine when you run docker network ls ?