Docker Community Forums

Share and learn in the Docker community.

Nodes have disappeared

(Vanderkerkoff) #1

Hi everyone

I just updated a kernel on one of our AWS nodes hooked into docker cloud and rebooted the node using AWS.

I then got a message from docker cloud, saying lost contact with node, to be expected.

Now however, I can’t see any nodes in the docker-cloud GUI, and the node that lost contact isn’t working.

The other node is however, anyone got any ideas?

(Vanderkerkoff) #2

That was hell on earth.

I’d sudo apt-get updated and sudo apt-get upgraded and sudo apt-get dist-upgraded both of my nodes.

Both of them went offline, I had to purge docker-cloud-agent software, reinstall using the docker-cloud cli as the web site wasn’t showing anything. Then I had to re label the nodes once they had finally finished deploying, took about 3 goes each, total disaster, 15-30 mins downtime.

Obviously my fault, but if anyone else gets into this state, all the best.

(Lizter) #3

The same happened here! Only I didn’t reboot, but just randomly lost contact. It is still marked unreachable, but has never actually been down.

The Node list in Docker Cloud fails with Unhandled rejection TypeError: Cannot read property 'content' of undefined

I inspected the API call, and the data is there. Seems like the API changed, but the changes are not reflected in the frontend code.

Integration tests? :slight_smile:

(Vanderkerkoff) #4

Hi lizter

I think what happened with me is I fell into the AUFS trap, and my apt-get upgrade/dist-upgrades caused this to happen.

Note: If you remove, upgrade, or install over the version of Docker Engine bundled with the Docker Cloud Agent, your nodes may not maintain a connection with Docker Cloud correctly. You will have to reinstall a compatible Engine version (currently version 1.11.2-cs5). Contact Docker Support for more information.

From here

I ‘think’ if I’d upgraded to currently version 1.11.2-cs5 prior to doing any of this, I may have been ok, not entirely sure though.

Time to get my important sites load balanced across nodes properly, well, at all would be a start.

Integration tests? hahahahahaha

(Lizter) #5

Hello! :slight_smile:

I hope the information you provided can be of help to others in the same situation, but I don’t have any issues related to upgrading at the moment.

My beef was merely with the Docker Cloud GUI not listing my nodes since yesterday with a JavaScript error in my client-browser.

(Vanderkerkoff) #6

Yeah, me too, hopefully it will help someone.

I usually tend to use the docker-cloud cli when I run into any sort of problems with the GUI lizter.

To be honest, I’m leaning towards only using the docker-cloud cli from now on, save me some headaches.

Thanks for replying, nice to know there’s others out there!!