The node that initialized a swarm becomes a manager node. Other docker hosts can be joined as either manager or worker nodes.
You might want to check on each additional docker host you want to join the swarm, whether the Swarm mode is already enabled:
docker info | grep Swarm
if the output is “Swarm: active”, the node indeed joined a swarm, and needs to leave it before it can join a new swarm cluster. On a fresh installed host, or a host that never initiated or joined a swarm, it should be inactive.