Percona MySQL cluster using docker swarm and service

General
,
1

Hi,

I am trying to create mysql cluster using docker swarm. I have 3 host servers and I want on each one of them to run only one docker mysql container. I was following the guide from here:

I followed the steps from the tutorial above and I have done this so far:

  1. Installed and configured etcd on each host and its cluster health is ok

  2. Installed docker on each servers and service is started

  3. On one of the nodes initiated swarm mode and joined the other 2 nodes to the swarm. docker node ls shows all nodes and their state is ok

  4. Created overlay network

  5. Created 2 directories on each node that will be mounted to docker containers for permanent storage solution:
    /etc/docker-configs/mysql-config
    /var/lib/mysql

  6. I try to create a service
    docker service create
    –detach=false
    –name mysql-cluster
    –replicas 3
    –network my-net
    –mount type=bind,src=/var/lib/mysql,dst=/var/lib/mysql
    –mount type=bind,src=/etc/docker-configs/mysql-config,dst=/etc/my.cnf.d
    –secret mysql_root_password
    –env MYSQL_ROOT_PASSWORD_FILE=/run/secrets/mysql_root_password
    –env DISCOVERY_SERVICE=xxx.xxx.xxx.xxx:2379,xxx.xxx.xxx.xxx:2379,xxx.xxx.xxx.xxx:2379
    –env XTRABACKUP_PASSWORD_FILE=/run/secrets/mysql_root_password
    –env CLUSTER_NAME=pxc
    perconalab/percona-xtradb-cluster:5.7

Now here problems happen. When i run the service create command all 3 containers never start and I get following errors in logs:

Jul 19 10:12:37 bs1 dockerd[15147]: time=“2017-07-19T10:12:37.740949992+02:00” level=error msg=“fatal task error” error=“task: non-zero exit (1)” module=“node/agent/taskmanager” node.id=pg5nwd5ws0eq38te3f0t1o27a service.id=ib6osl3b8k8hsh59p8i7irk77 task.id=ygee5jvdqtwq6sx20rt2rkgif

This problem happens even if i don’t use mount options and use ephemeral storage.

Can you give me some advice how to solve this problem? I need to have permanent storage for each container on each host to have a healthy and good working cluster.

Thank you in advance!

Best regards,
Ali Nebi

2

Found the problem. Docker image that I use doesn’t support _FILE variables and like that mysql is not starting. If i pass plain passwords using MYSQL_ROOT_PASSWORD and XTRABACKUP_PASSWORD cluster is working ok.

4

Hi,

After reading several posts about clustering I saw that best practice in this situation is to use mode=global instead of telling how many replicas the service to have. Like this docker always creates only one container on each node. You can additionally use constraints to tell on which nodes you want the containers to be created.

I tested that method and it worked ok for me. When I put down one of the instances it is recreated on same one without problem and re-sync from other healthy servers. Cluster state is ok. About mounting, yes, I also use mount-points to persists the data.

So far the cluster worked ok.

Regards,
Ali