2 mutally exclusive swarm nodes with swarm/shipyard

Docker Engine Swarm
1

Hi All,

I am seeing odd behavior between 2 of the 4 nodes on my swarm/shipyard cluster, let’s call them A and B.
The swarm manager is running on host C.

If node A is present, the containers from B are not listed in shipyard.
In this state, Node B is listed under the "nodes’ tab with the correct information (# of containers, CPUs, RAM, etc), but no containers from that node are listed under the containers tab.

Now If I remove node A from the cluster, the containers from B appear under the containers tab.
Then if I restart node A then I have to remove B to make A’s containers appear.

A and B were happily co-existing for many days up until last week.

I also have a forth node D that is the same configuration as A and B.
D can connect anytime and those containers are always listed.
Also the containers from C, the swarm manager, are also listed.

I am using swarm and docker on CentOS7 w/ TLS and etcd in a configuration like here:
https://shipyard-project.com/docs/deploy

Any ideas on how to troubleshoot this?
I’ve moved the nodes around, tried the swarm manger between them, checked the certs, etc.
Any advice would be very appreciated.

-jk

2

Hi all,

I figured this out -

Each node has an ID associated with it. That ID, at least on my CentOS7 systems, is in /etc/docker/key.json.

If that ID clashes with another node, say… because you copied the docker folder from another host for some reason, bad things happen.

If you are having this issue, you can delete key.json and it gets recreated when the docker daemon is restarted.

-jk

see : https://github.com/docker/swarm/issues/1406