Can not join swarm, manager does not listen on port 2377

I am trying to run Docker Swarm on Raspberry Pis, but nodes can not connect master.

Master

pi@rpi4b-node1:~ $ docker swarm init --advertise-addr=192.168.49.101
Swarm initialized: current node (tdfutddkxmjyxz3bsudtdwh4z) is now a manager.

To add a worker to this swarm, run the following command:

docker swarm join --token SWMTKN-1-5at4916cfchiomuwra18jqaw8gs211x6ky48twnsf0b5k1dnuo-6copxcw7vq47hvhkiaixv844a 192.168.49.101:2377

To add a manager to this swarm, run ‘docker swarm join-token manager’ and follow the instructions.

pi@rpi4b-node1:~ $ nc -zv localhost 2377
nc: connect to localhost (::1) port 2377 (tcp) failed: Connection refused
nc: connect to localhost (127.0.0.1) port 2377 (tcp) failed: Connection refused
pi@rpi4b-node1:~ $ netstat -tuplen
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 0 12682 -
tcp6 0 0 :::22 :::* LISTEN 0 12684 -
udp 0 0 0.0.0.0:68 0.0.0.0:* 0 10210 -
udp 0 0 0.0.0.0:5353 0.0.0.0:* 108 12581 -
udp 0 0 0.0.0.0:52269 0.0.0.0:* 108 12583 -
udp6 0 0 :::5353 :::* 108 12582 -
udp6 0 0 :::35343 :::* 108 12584 -
udp6 0 0 :::546 :::* 0 14344 -
pi@rpi4b-node1:~ $ docker info
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Docker Buildx (Docker Inc., v0.7.1-docker)

Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 20.10.12
Storage Driver: vfs
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
NodeID: tdfutddkxmjyxz3bsudtdwh4z
Is Manager: true
ClusterID: tot9b75kcl2r2judk32qczfkb
Managers: 1
Nodes: 1
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
Data Path Port: 4789
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 192.168.49.101
Manager Addresses:
192.168.49.101:2377
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
runc version: v1.0.2-0-g52b36a2
init version: de40ad0
Security Options:
seccomp
Profile: default
rootless
cgroupns
Kernel Version: 5.10.92-v7l+
Operating System: Raspbian GNU/Linux 11 (bullseye)
OSType: linux
Architecture: armv7l
CPUs: 4
Total Memory: 1.828GiB
Name: rpi4b-node1
ID: GK7Z:XWBG:ZU4K:KWVZ:O5MJ:UUWX:VGIV:F3Y3:Z3DJ:FZU6:LT53:J6XS
Docker Root Dir: /home/pi/.local/share/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpu shares support
WARNING: No cpuset support
WARNING: No io.weight support
WARNING: No io.weight (per device) support
WARNING: No io.max (rbps) support
WARNING: No io.max (wbps) support
WARNING: No io.max (riops) support
WARNING: No io.max (wiops) support
pi@rpi4b-node1:~ $ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

Node

pi@raspberrypi:~ $ docker swarm join --token SWMTKN-1-5at4916cfchiomuwra18jqaw8gs211x6ky48twnsf0b5k1dnuo-6copxcw7vq47hvhkiaixv844a 192.168.49.101:2377

Error response from daemon: rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp 192.168.49.101:2377: connect: connection refused”

Have you checked the log of the Docker service? Probably

journalctl -xe -u docker

What is the output of the following command on the manager?

docker node list

Thank you for a quick reply :slight_smile:

Feb 05 22:45:51 rpi4b-node1 systemd[1]: Starting Docker Application Container Engine...

░░ Subject: A start job for unit docker.service has begun execution

░░ Defined-By: systemd

░░ Support: https://www.debian.org/support

░░

░░ A start job for unit docker.service has begun execution.

░░

░░ The job identifier is 785.

Feb 05 22:45:51 rpi4b-node1 dockerd[5265]: time="2022-02-05T22:45:51.791215754Z" level=info msg="Starting up"

Feb 05 22:45:51 rpi4b-node1 dockerd[5265]: time="2022-02-05T22:45:51.794370451Z" level=info msg="parsed scheme: \"unix\"" module=grpc

Feb 05 22:45:51 rpi4b-node1 dockerd[5265]: time="2022-02-05T22:45:51.794964144Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc

Feb 05 22:45:51 rpi4b-node1 dockerd[5265]: time="2022-02-05T22:45:51.795432412Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}" module=grpc

Feb 05 22:45:51 rpi4b-node1 dockerd[5265]: time="2022-02-05T22:45:51.795851348Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc

Feb 05 22:45:51 rpi4b-node1 dockerd[5265]: time="2022-02-05T22:45:51.799685625Z" level=info msg="parsed scheme: \"unix\"" module=grpc

Feb 05 22:45:51 rpi4b-node1 dockerd[5265]: time="2022-02-05T22:45:51.800263447Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc

Feb 05 22:45:51 rpi4b-node1 dockerd[5265]: time="2022-02-05T22:45:51.800685347Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}" module=grpc

Feb 05 22:45:51 rpi4b-node1 dockerd[5265]: time="2022-02-05T22:45:51.801166411Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc

Feb 05 22:45:51 rpi4b-node1 dockerd[5265]: time="2022-02-05T22:45:51.976702501Z" level=info msg="[graphdriver] using prior storage driver: overlay2"

Feb 05 22:45:51 rpi4b-node1 dockerd[5265]: time="2022-02-05T22:45:51.983444276Z" level=warning msg="Unable to find memory controller"

Feb 05 22:45:51 rpi4b-node1 dockerd[5265]: time="2022-02-05T22:45:51.984003691Z" level=info msg="Loading containers: start."

Feb 05 22:45:52 rpi4b-node1 dockerd[5265]: time="2022-02-05T22:45:52.714349982Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"

Feb 05 22:45:52 rpi4b-node1 dockerd[5265]: time="2022-02-05T22:45:52.969587608Z" level=info msg="Loading containers: done."

Feb 05 22:45:53 rpi4b-node1 dockerd[5265]: time="2022-02-05T22:45:53.106122071Z" level=info msg="Docker daemon" commit=459d0df graphdriver(s)=overlay2 version=20.10.12

Feb 05 22:45:53 rpi4b-node1 dockerd[5265]: time="2022-02-05T22:45:53.106324826Z" level=info msg="Daemon has completed initialization"

Feb 05 22:45:53 rpi4b-node1 systemd[1]: Started Docker Application Container Engine.

░░ Subject: A start job for unit docker.service has finished successfully

░░ Defined-By: systemd

░░ Support: https://www.debian.org/support

░░

░░ A start job for unit docker.service has finished successfully.

░░

░░ The job identifier is 785.

Feb 05 22:45:53 rpi4b-node1 dockerd[5265]: time="2022-02-05T22:45:53.241236784Z" level=info msg="API listen on /run/docker.sock"
pi@rpi4b-node1:~ $ docker node list
ID                            HOSTNAME      STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
tdfutddkxmjyxz3bsudtdwh4z *   rpi4b-node1   Ready     Active         Leader           20.10.12

Please, use the </> button to insert codes, commands and logs because the markdown filter hides potentially important parts of them if you just quote. I edited your post to fix it.

Is the logs from right after you initialized the manager node? I don’t see any swarm log, only how Docker started.

If you remove your manager from the cluster (``docker swarm leave --force`) and try to set the listen address instead of the advertise address, you can make sure you are checking the right port:

docker swarm init --listen-addr=192.168.49.101:2377

Then check the logs and the ports.

Sorry for formatting.

pi@rpi4b-node1:~ $ docker swarm init --listen-addr=192.168.49.101:2377
Error response from daemon: manager stopped: failed to listen on remote API address: listen tcp 192.168.49.101:2377: bind: cannot assign requested address

Only restart things are in the logs. Swarm init with advertise nor listen are mentioned there.

Can you run it on any port?

The same result with another ports.

Then is it really the IP address of the node? Can you run a simple python HTTP server for example on any port using that IP address?

python3 -m http.server --bind 192.168.49.101 8080

If you can, try with port 2377 too.

python3 -m http.server --bind 192.168.49.101 2377

Webserver works on the same ports.

    1. 2022 10:51:16, Ákos Takács via Docker Community Forums <docker@discoursemail.com> napsal:

I have the same issue as you bobik - did you get this resolved?

1 Like

Hello @keithleecouk @bobik
Same problem for me, did you get this resolved ?

I’m having the same issue too. I have a raspberry pi host and cannot connect to it from another pi or from my Mac. The biggest red flag that swarm is failing to listen on that port is that I’m able to run the python webserver on 2377 even when it’s supposedly waiting for connections.

I have exactly the same problem. I have installed Docker and activated rootless setup. After some research, I found out that swarm mode and rootless mode are not possible together.

From the documentation:

When you initialize a swarm or join a Docker host to an existing swarm, an overlay network called ingress is created on that Docker host, which handles the control and data traffic related to swarm services.

Run the Docker daemon as a non-root user overlay networks are not supported.

more under

2 Likes

Thanks, that answers my other question :slight_smile: