What if the leader is down in Swarm cluster and what about histories

Hello,

This is what I understood from a Docker Swarm cluster with multiple managers:

manager1 ==> leader
manager2 ==> manager
manager3 ==> manager

When I say manager2 to run a command, it sends this command to the leader and the leader runs that command.

Now suppose manager1 is down now, and manager3 is the new leader.

  1. How can we check which one was the last leader?
  2. How to check the history of commands in the last leader?
  3. What if I leave manager1 from the managers and make him a worker? Are there any histories to show what it has done when he was the leader?

Swarm uses the raft consensus algorithm for quorum. If no leader is present, the nodes will elect a new leader. As long as (floor n/2)+1 manager nodes are healty the cluster will be able to hold quorum, elect leaders and change state for swarm scoped objects… State changes will always be forwarded to the leader and then distributed to all nodes using an encrpyted raft log.

May I ask why those questions mater? If you are curious about what happens on each node, just attach yourself to the even stream and handle the events you want to know about - you are able to receive the node local and swarm scoped events for all manager nodes.

1 Like

I’m just curious about the process.

So, when the last leader is accessible again, it sends the history to the current leader.
Suppose last leader is being reboot, and it may take 5 minutes.
Is that possible to check the previous logs on new leader?

I have 6 VMs in my laptop (3 managers - 3 workers) and am curious about how to check them.

Though, you did understand the part that the state is present on all nodes due the raft consensus, didn’ you?

The only situation where the history is send is when a node was unhealthy and became healthy again, it will catch up to the current state of the raft log and be in sync with the other manager nodes again.

There is no such thing as the leader is the only one storing the logs. What sense would it make to have a single point of faile for your state if you ihave a consensus based state amongst 3 nodes?!

You might want to read more about how consensul algorithms in general and Raft in particular work.

1 Like