Can't understand Docker Recover from losing the quorum

The docker official documentation said

However, a swarm cannot automatically recover if it loses a quorum. Tasks on existing worker nodes continue to run, but administrative tasks are not possible, including scaling or updating services and joining or removing nodes from the swarm.

When I have a cluster which has four nodes and two of them are managers, and then I demote one of the managers and remove it, and the swarm loses the quorum, but I am still able to scale the service in the leader node. According to the quote above, I can’t scale the service as the swarm loses the quorum. So do I understand right?

Loss of a quorum refers to a cluster that has gone from an odd number of managers to an even number of managers. Quorum is usually required in order for a cluster to decide on updates (e.g. 2 or 3 out of 3 managers agree on an update).

In your example, with 2 managers, if both agree on an update, a quorum is established. However, if the two disagree on state, a quorum with respect to that state has not been reached. A swarm with two managers is not a recommended configuration for high availability, since there is no tie breaker manager in the event two managers disagree on state.

Scaling your cluster down to a single node is actually a quorate cluster, since only one manager exists, there are no disagreements on state.

For some general guidance on what quorum means in the context of high availability and clustering, here are a couple of articles I found on my favorite search engine. You can check these for a deeper discussion on the topic


https://blogs.msdn.microsoft.com/clustering/2011/05/27/understanding-quorum-in-a-failover-cluster/

It’s a common enough topic in the literature that there are many other sources you can check if neither of these provide sufficient background.

Thank you very much!