High Availability in Docker Swarm

Hi,

I have 2 Ubuntu servers and I need to configure high availability cluster with docker.

Is it best recommended to go with swarm ? The requirement is to keep the application from both the Ubuntu servers always available for end users through a virtual IP.

I have configured a swarm with 2 nodes and added an nginx service across 2 node.

Here When worker was made down the container moved to manager node. But when Manager node was made down containers from manager node never failed back to worker node.

Question:1 - Is there any specific configuration which needs to be applied to make the fail back happen automatically. ?

Question:2 - Will there be any Split brain kind of condition in Docker swarm similar to Sun/Veritas Cluster? If Yes, how to handle it ?

Question:3 - During any planned maintenance is there any mechanism to manually moved the application to other node ?

Please assist with feedback for the above.

Regards.,
Viswanathan

Using a light weight Kubernetes distribution like K3s could be a better idea. People are looking for features in Swarm existing in Kubernetes which will probably not be implemented since as far as I know it is not actively developed and improved, but whatever you are using two managers will not be enough. You must have an odd number of managers. If you have two, then one fails an the other will not work either and there is nothing to schedule your containers to another machine. This is mentioned in the documentation of swarm as well

I can’t really answer the other questions confidentally as I don’t use Swarm and I never used Sun/Veritas, but the above linked documentation also mentiones the “drain” mode. If it wors as in Kubernetes, that is what you are looking for in your 3rd question.

Since Swarn am Kubernetes use the raft consensus algorithm, those aspects are identical for both. Raft requires low latency network connections amongst the nodes, and floor(n/2)+1 healthy manager nodes to reach quorum. Without quorum the cluster is headless and can not perform any state changes to any(!) resource, until the required number of nodes are available again…

Regarding split brain:

  • let’s assume in a cluster of 3 manager nodes, one gets separated. From the perspective of the two nodes, the cluster is able to reach quorum - and they will reconcile ressources to the desired state - though, the separated node, will continue to run its containers. As soon as the node rejoins the cluster, the node catches up and reconciliation kicks in, to get rid of over-provisioned resources.

  • now let’s assume you have 4 manager nodes, and two get separated. Then both cluster partitions will be headless, and the cluster won’t be able to reconcile any resources. Though, running containers will be still running, but won’t be restarted if they fail.

The drain mode will drain all swarm services taks, but will not touch any (non-swarm) containers.

Fulfilling this requirement is quite simple, actually.

Active/Passive:

  • run two independend docker hosts
  • run the application at all time on both servers
  • configure a virtual-ip with a vrrp implementation like keepalived, carp or whatever solution you have in your company. Make it point the virtual-ip to the active host, and make it switch to the standbye host, when it detects that the active host is not available.

Loadbalancer :

  • run two independent docker hosts
  • run the application at all time on both servers
  • configure a loadbalancer in front that balances the load to containers on both docker engines (configure sticky sessions if necessary). Make sure to check the health of the lb targets, and only send traffic to healthy targets.
1 Like

For a HA Docker Swarm cluster, you need 3 manager nodes (which can run workloads).

But making them available with a failover IP is a different topic. We use a managed load balancer in front of the nodes.

You can manually manage the failover IP with something like keepalived.

I feel k8s/k3s is waaay more complex, but it has some solutions for failover IP.

Ok, seems I sent this too late :wink:

Hi, Thank you for your response. But in our setup it only has 2 servers in prod env and one in dev env. So these 3 couldn’t be on a 3-node swarm. When we go with Kubernetes it was mentioned to disable swap so that Kubernetes can handle resource allocation effectively and if that should be the case I’m afraid I cant go with this option too.

Regards.,

Hi meyay,

Thanks for your response. So can I safely assume that split brain condition wont occur in my case (with only 2 node and no quorum). ?

Regards.,

Hi meyay,

Thanks for the response. I will try this ways and update if I run into any challenges.

Regards.,

Hi bluepuma77,

Thanks for your response. Yes you are right, for Swarm minimum of 3 nodes needed. But our setup is with only 2 nodes in prod and one in dev. So application is required to be made high available only within those 2 nodes.

Correct me if I’m wrong here. Using Load balancer in front of these 2 prod servers will need another pair of IP’s is that correct ?? Then Load balancer is actually another set of servers. Is that right ?

The point here is we need to automate the failover between 2 nodes instead of manual management.
I have gone thru a few blogs and there is a IP reassign script in python which does the thing.
Anyways will try with keepalived and come back in case of any challenges.

Regards.,

Regards.,

For “professional” high availability, you need 3 servers. k8s (and the smaller k3s distribution) and Docker Swarm need 3 nodes for the raft algorithm. If you only have 2 nodes, you can’t reliably run those systems.

Instead you can just run a server and a hot standby - if your application and database allow such a setup. You need to manage a failover IP, which is the target of your domain DNS entry, it needs to be switched over when the “active” server fails.

In a datacenter, you would usually get an additional IP for this. You then need a system like keepalived that switches the IP - either by telling the API of the datacenter or by actively assigning the IP to the passive server. But it really depends on your environment.

Hi bluepuma77,

If I use keepalived and set a Virtual IP to failover to passive when active node fails, the connectivity is established to passive node. But with application data mount points (or file systems) that is available in the active node, will that also failover to passive node ?

For example,
Active node IP is x.x.x.20
Passive node IP is x.x.x.30
Virtual IP is x.x.x.10
Application mountpoint is /app

I have a FS mounted on Active Node. Now active node is down and with keepalived Virtual IP fails to passive node.

Along with this, will the /app mountpoint will become mounted and available in Passive Node so that the data inside the FS is accessible from passive node server??

Regards,

By just using a failover IP, no file system is HA, you need a separate solution. Clustered file server, continuous replication, etc.

Yes… But that is what I am trying to implement thru docker or kubernetes and with only 2 nodes.

Is that possible in docker with a 2-node setup??

All official HA systems need at least 3 nodes/servers.

You can try to alternatively run a active/standby setup, which you need to manage yourself. IP re-assignment, SQL database master/slave, own file replication. There is no “simple” solution, whole books have been written about this topic.

Make sure, after failure, to also think about the recovery process. For example when files have been uploaded to standby (active down) how to sync to old active before taking over again.

1 Like

The whole topic stopped making sense to me, and it seems to still miss relevant details.

Remember: answers can only be as good as the questions and the details shared in them.

Hi Metin,

At what point this topic stopped making sense to you? And what other relevant details you feel is still missing.??

I’m not expecting good answers and im not an expert either in this area. What I have is a scenario and with the points posted here im raising what if and why not kind of questions from a layman’s point of view.

Remember : There is no thing as a good question. Many crazy and dumb questions only have lead to great ideas. :wink:

Feel free to stop giving “good” answers if the topic doesnt make sense to you anymore :blush:

Regards.,

You insist to use two nodes, but we explained multiple times that HA with official solutions need 3 nodes, as they use raft (link) for consensus.

Hi bluepuma77,

Yes You gave ur solution and an alternative too, thanks for that :slightly_smiling_face::+1:but my last response was not for you.

I know, but that was the point that doesn’t make sense :wink: