Docker Community Forums

Share and learn in the Docker community.

Using etcd cluster for docker swarm kv store

Hi all, I am configuring a Docker Swarm cluster (not “swarm mode”, the “original” swarm) using etcd as the kv store.
I installed an etcd daemon on each swarm master (2) as well as workers (3).
To avoid pointing to a single etcd node, each docker engine daemon uses the local etcd instance; the idea is that the docker daemon does not care about the etcd cluster topology, it just uses the local etcd instance.

Using this approach, I am experiencing an inconsistent behaviour: sometimes the swarm manager gets inconsistent data: a “docker ps” in the swarm manager does not always reflect the whole list of running containers on all workers.
Using docker-compose to start several services, some service fails to find the overlay/bridge network which compose creates at start.

I also found that configuring all of the docker daemons to use a single etcd instance (the one local to the swarm master node) everythings works fine.

Is this related to the etcd cluster eventual consistency? If this is the case, is there a different, suggested practice to configure the kv store in order to achieve high availability?

Best Regards
Roberto

After a few days of troubleshooting, it appears the problem was due to the clocks of the nodes/managers not properly in sync.