Data(base) persistence in docker swarm mode

General
1

I’m looking into using the new docker 1.12 swarm mode but am at a loss when it comes to the high availability features in combination with stateful containers (like a database container).

Ordinarily I would use a volume to store the data for a database on a node which would be found again when I recreate a container on the same node (a named volume).

Now consider the following situation:

I have a docker swarm mode cluster of 1 node and start a mongo service with a mount like this:

docker service create --mount type=volume,source=mongodata,target=/data/db,volume-driver=local --replicas 1 -p 28001:27017 --name mongomain mongo --auth

The cluster will fire up a service with a volume (now called mount) which works just fine, remove and recreate the service and the data will be there.

However, what would happen if I create this service on a multi node cluster? If the first node drops and the cluster fires up a new task on a different node, I presume it won’t have the same volume there. So I will have lost my data.

And what if I’d like to use replication? a swarm with 3 nodes and 3 instances of mongodb in replication mode. Is that even possible with the redistribution of tasks done by swarm mode?

I’m currently thinking the only real solution would be to just running 3 classic containers on these nodes and just use swarm mode for my stateless (frontend etc.) tasks.

Is there a better way to do this using the swarm mode?

1 Like
2

Does no one else have this problem? Or do people just not use DB’s in docker like this?

3

I have the same issue with swarm mode. I believe you can add a constraint to the service which runs the database task on a specific node. In my case, I have 3 mysql databases configured as 1 master and 2 slaves with a monitor to failover to the most current slave in the event of a failure of the master. Seems I would need to define 3 services with each service having 1 task fixed to a specific node, if I wanted to run in swarm mode.

But, it seems that the swarm scheduler really isn’t designed for deploying databases in high availability mode. It seems to me that the admin should be able to specify mount of a local host directory (local volume) and have the scheduler automatically schedule a 1 task service on the host node that contains the local volume. But, as of now, it seems we need to use node constraints to fix the task to a particular node, as a work around.

For now, I am not using swarm mode because of this issue. Rather, I just configure each node with a set of containers and use proxies for connecting the containers on various nodes to each other. I think the focus of swarm at this point is to run services that don’t use persistent volumes, unless you are willing to use some volume plugin like flocker to access your volumes over a network connection. In my case, I require the local databases to be on local NVMe SSD storage for maximizing the performance of mysql. I hope the Docker team puts some thought into how to enhance swarm mode to support such database clusters in the future. In the mean time, I don’t think swarm mode offers enough benefits over just running persistent databases using fixed containers with monitoring/proxying for failover and high availability.

4

I am in the same conundrum now. I have a neo4j database and I also have a seed data folder (neo4j/data), that I point the neo4j container and it automatically has the initial set of data that I need for my api. Currently, when I start the ne4j container, I map the volume that contains the seed data , however that is fine with one container.
Now I plan to deploy the full stack (ui, api and db) to docker swarm, so as to leverage HA, load balancing and clustering benefits.

Any directions here please? Until then I am going to assume that data stores are going to be managed in a VM/baremetal/classic fashion for production deployment and docker containerization is NOT going to help much.

5

I’m currently exploring whether it would be possible to mount virtual hard drives as volumes and have these hard drives then stored on some kind of network storage. But I’m still researching and don’t know whether a good I/O performance can be achieved through this.

6

Same questions here. I’m thinking of just running Mongo outside of swarm, set up native Mongo clustering/replication between the nodes and use Swarm for the stateless front end.

What is the best practice for this type of configuration w/Docker?

7

for some basic persistent storage you can use NFS , but for databases it might be too slow.

Here is a blog post and code examples with a solution using NFS
https://www.vip-consult.solutions/post/persistent-storage-docker-swarm-nfs#content

I am also working on an easy percona cluster as a docker swarm service that supports scaling without any external discovery service requirement.
It will run as a global service so each node will have a dedicated volume storage that won’t be lost between restarts, but the sync between the databases will be done on a software level.
I will post it here when ready , but it also might be merged with the official percona cluster image on the docker hub

8

I find my self having the same issue, constraint the server service to a node where is the mount (volume), is this a good solution? And how I do this? --constraint ‘node.hostname==cloud12’ ?

9

For setting up a multi-master, multi-source MariaDB cluster, I used container labels combined with service constraints. II used the local-persist-volume-plugin to let the config files and database itself reside on the physical node.

I initially wanted to create one service for all mariadb instances, but that was too complicated. I would need to set it up with three replicas, but if one of the three nodes went down, I would be left with one node with two replicas running on the same volume.

In the end I found it better to create three services, one for each MariaDB instance, and add the MaxScale MySQL proxy into the mix, create one service for that one, and let the proxy handle the fail over to the three mysql services. Docker can move the proxy around if it fails, or I can even create multiple instances of it.

10

FireCamp is a project aims to solve data persistence in docker swarm mode. The installation wiki shows how easy it is to install a swarm cluster on AWS via CloudFormation and deploy such as 3 node MongoDB ReplicaSet across 3 availability zones. Please take a look to see if it helps for your cases.

11

Hello everybody,
what about using GlusterFS at Docker host level to replicate the whole /var/lib/docker/volumes/ directory?
I’ll try just this approach this evening, with a two-node GlusterFS+Docker Raspberry environment.

12

I’m replying to myself, just to state that:

  • I tried to replicate via GlusterFS the /var/lib/docker/volumes/ and (as I should have known even before trying) this didn’t work (Docker not starting anymore)

  • Setting up replication for another directory works well, but you’ll need to abandon the usage of named volumes, switching to bind mounts instead

13

Using GlusterFS is a good option. You may need to pay attention to the possible corner case. For example, if there are some bug that the old container is not stopped and a new container is started on a new node, 2 db instances may be running at the same time and write to the same db file. This will corrupt the db file, and you may have data loss. This kind of corner case will not happen for like AWS EBS. EBS ensures the volume could only be attached to one instance. So the new container on the new node will not be able to attach the EBS volume, as EBS volume is still attached to the old node.

1 Like