Cluster Depends on the IP Address of the Initial Manager Node

General
1

Expected behavior

Nodes can be added to the cluster after losing the initial manager node.

Actual behavior

Nodes that are brought online using the Auto Scaling Groups fail to join.

Additional Information

The issue seems to occur because the first manager node that comes online registers it’s IP address in DynamoDB and it’s never updated even if the first manager node is lost and replaced by the Auto Scaling Group. When new instances are launched my the Auto Scaling Groups, they pull the first manager’s IP address from DynamoDB and try to join the cluster using it. If that instance is not accessible, or has been replaced and has a new IP address, the new nodes fail to join the cluster.

Steps to reproduce the behavior

  1. Launch Cloudformation Stack
  2. Find the IP address of the initial manager by inspecting the DynamoDB table
  3. Terminate the instance that has that IP address
  4. Wait until the manager Auto Scaling Group will replace it
  5. Once the new manager is online, verify that it is not part of the swarm: docker node ls
  6. Confirm that it attempted to join using the now terminated instances IP: docker logs $(docker ps --all --filter ancestor=docker4x/init-aws:aws-v1.12.0-beta4 --quiet)
2

Thanks for reporting this!

We’re working on changes to how the manager quorum is managed - stay tuned.

1 Like