Error: manager stopped: can't initialize raft node: rpc error: code = DeadlineExceeded desc = context deadline exceeded

Using 18.09.1 on Red Hat.

When I run

docker swarm join --token

I get

Error response from daemon: Timeout was reached before node joined. The attempt to join the swarm will continue in the background. Use the “docker info” command to see the current swarm status of your node.

When I run

docker info

I get

Error: manager stopped: can’t initialize raft node: rpc error: code = DeadlineExceeded desc = context deadline exceeded

Has anyone else had a problem?
I have even added them as workers and they add no problem, but when i promote the worker to manager it errors out.

Are you running a ntp-client on all nodes? Raft is quite sensitive to time drift.
Also make sure that following ports are not blocked between nodes:

  • TCP port 2377 for cluster management communications
  • TCP and UDP port 7946 for communication among nodes
  • UDP port 4789 for overlay network traffic

How I fixed it was adding all the nodes private IP addresses to the NO_PROXY list.