Docker Community Forums

Share and learn in the Docker community.

BUG: "Unreachable" Nodes Break Stack Redeploy


(Vidsy.co (Charlie)) #1

Context

Problem

  • Nodes scaled down (terminated) at ~9pm UTC.
  • They remained in "Unreachable" state.
  • New nodes started at ~7am UTC as the ASG scaled back up.
  • New nodes tried to redeploy stack.
  • Redeploy failed as Docker Cloud tries to redeploy services that it thinks are still running on the "Unreachable" nodes.

Manual Solution

  • Log into web app and terminate each of the "Unreachable" nodes.
  • Redeploy stack again.

Do "Unreachable" nodes ever get removed automatically?

@revett


(Vidsy.co (Charlie)) #2

Answering my own question after a chat w/ @maxime_tutum on the Tutum Slack.

  • "Unreachable" nodes will never be removed automatically.
  • Best workaround is to have another service remove "Unreachable" nodes via the API.
  • Or, terminate nodes via the API so that they do not become "Unreachable".

https://docs.docker.com/apidocs/docker-cloud/#terminate-a-node

@revett