Resolved: High availability swarm manager leadership issue


I’m trying to set up a swarm cluster with a high availability swarm manager as documented here:

I have a consul cluster up and running and am able to register services with it successfully.
I have some docker swarm nodes which run a consul agent and I can see them listed as nodes in the consul ui.

The problems I’m having are with the swarm manager leadership elections. Basically, a manager is able to acquire leadership and keeps it for about a minute, then loses it. I can see in the consul KV store that the leader key (nodes/docker/swarm/leader) value has been set.

Looking at the logs from the swarm containers, I see the following output:

time="2015-09-24T15:19:40Z" level=info msg="Leader Election: Cluster leadership acquired"
time="2015-09-24T15:20:39Z" level=info msg="Leader Election: Cluster leadership lost"
time="2015-09-24T15:20:39Z" level=error msg="Lock already held"
time="2015-09-24T15:20:49Z" level=info msg="Leader Election: Cluster leadership lost"

I can’t seem to get a swarm manager to keep leadership, even if I use a single manager!

Can anyone help?
I can provide more info if required, just let me know what you need.



How did you resolve this?