For Docker Swarm with two nodes, one manager and one worker, what happens if the manager node goes out?
Or is it better to have two managers?
Have you tried the forum search? It is safe to say this has been discussed a couple of times now.
Yes, there is a post on this. I just wanted to confirm if the manager node goes out in a two-node system, does the worker node become a manager node? Raft consensus says that it maintains being a candidate until two more nodes come back online to create a tie-breaker majority. In this case, we only have the old manager node that can come back online, which means the worker node would be in perpetual candidate status.
On a separate note, if the manager node comes back online, does it maintain its manager status?
It does not. A node must be explicitly promoted to manager, to become a manager. Though, in a two node cluster with a single manager, it requires the manager node to be healthy, in order to promote the worker to manager node.
Please share the source, so we can see in which context this statement was embedded.
It does. Though, the cluster will be headless until then. No swarm scoped resources can be created/deleted/updated in a headless cluster. Of course plain docker commands will continue to work, as they are neither affected by swarm, nor affect swarm in any way.
1 Like
Oh, I forgot to mention that it’s a terrible idea to run the cluster with 2 manager nodes, as the cluster will be headless when one of the two nodes becomes unhealthy.
1 Like
I was just playing around with a raft visualizer that does not necessarily speak to how docker handles things.
Thank you for the clarity.
Yes, I saw that one everywhere.
In my opinion the minimum cluster size can consist of 3 manager nodes (but not less) without any worker node. If the cluster grows, just add more worker nodes and be good.
1 Like
The docs are kind of talking around the minimum number:
If the swarm loses the quorum of managers, the swarm cannot perform management tasks. If your swarm has multiple managers, always have more than two. To maintain quorum, a majority of managers must be available. An odd number of managers is recommended, because the next even number does not make the quorum easier to keep. For instance, whether you have 3 or 4 managers, you can still only lose 1 manager and maintain the quorum. If you have 5 or 6 managers, you can still only lose two.
Even if a swarm loses the quorum of managers, swarm tasks on existing worker nodes continue to run. However, swarm nodes cannot be added, updated, or removed, and new or existing tasks cannot be started, stopped, moved, or updated.
For HA, you need to setup 3 managers to continue regular operations, even when 1 manager fails. Note that managers can run regular worker workloads.