Docker Community Forums

Share and learn in the Docker community.

Sequential deployment - No downtime deployments

Hi,

We want to achieve “No-Downtime” deployments, but this is currently not possible with Docker Cloud. The option “Sequential deployment” deploys containers sequentially, but it doesn’t check at all if the deployed container is ready (e.g. web application is up and running) and continues with the next in the list. That means currently we always run into downtimes during deployments.

Is there anything planned to support “No-Downtime” deployments in the future?

Cheers Sven

10 Likes

I’d be very interested in that too…
It’s actually the only thing holding me fro moving to docker cloud

Not sure if the team’s opinion has changed but I think if you want 100% zero-downtime then you need to use the API.

I wrote this hack 6+ months ago https://gist.github.com/revett/99d8a5143c0bfeddfc92

@revett

1 Like

Zero downtime is far better suited to blue-green strategy then a sequential container deployment.

Hi Sven,

As some have suggested already, blue-green deployments are a common solution for people wanting to achieve 100% zero-downtime.

You can read more about how to do it here (although it references Tutum, it still works if you just substitute in Docker Cloud): http://blog.tutum.co/2015/06/08/blue-green-deployment-using-containers/

1 Like

Things that hardened deployments for us:

  • tune your images for fast container start times. Get ALL building/compiling out of the way during the container build. (see http://12factor.net/disposability for rationale)
  • run min 3 containers even if you only require 2
  • run in high-availability strategy. The extra time to pull images on each server should give your prior container plenty of time to startup.

Hello @kickingthetv,

After having read that article, I decided to write a Capistrano gem in order to automatise this blue/green deployment procedure (Yes, I’m a Rails dev, and I like to “just” call deploy, and wait to get it done well).

I have implemented exactly (as far as I can see) what’s written in that article, and the deployment is working, but with 5 seconds downtime when redeploying the load balancer.
(Here is the Github repo: https://github.com/YourCursus/capistrano-docker-cloud).

To summarise what’s done:

  1. After having checked some required information I’m looking for a stack with the stage name (conversion over configuration). Here I’m deploying with cap staging docker_cloud:deploy so the gem is looking for a stack named ‘staging’.
  2. The gem check in this stack that no other service is running the image and tag we’re going to deploy
  3. Then the gem check if there is a load balancer in the stack (jn order to finish the deployment, otherwise it make no sense)
  4. Now starts the deployment: the gem creates a new service using the docker image name (without the namespace) then the tag with dots and underscores replaced by hyphens. The service configuration is copied from the service already existing running the same image but with a different tag (otherwise it uses the default values).
  5. The gem starts the servie
  6. The gem waits until the service is in the “Running” state, and then waits until it gets a 200 from all the containers of the service
  7. Finally the gem update the load balancer links and redeploy

And here is the moment when it breaks, when redeploying the load balancer.

Can you please tell me what am I doing wrong which causes this downtime ?

Thank you.

zedtux

This will probably come out of the box with Docker Swarm in 1.12.
This is assuming that Docker Cloud is moving towards a Swarm architecture.

1 Like

OK, nice, but anyway, I don’t like to have unfinished stuff and I don’t like to not understand why things are going wrong.

I only have an issue with the LB redeploy which were supposed, based on the article, to switch the traffic from one service to another.
It should work, right?

Hello @borja. As you wrote the article, maybe you could help on this topic? I’m so close to get this zero downtime deployment.

@yourcursus the LB (dockercloud/haproxy) should update links by querying the Docker Cloud API without the need for it to be redeployed. Have you tried this? Simply keep everything the same but do not redeploy the LB as the last step. After you have changed its links, see if the LB starts sending traffic to the new service. Let me know how that works in this scenario.

Awesome, thank you @borja I’ll give it a try.

Awesome @borja. Thank you, it’s working perfectly.

I just need to do some code cleaning and I’ll release my gem.

Under what circumstances does dockercloud/haproxy update links automatically? I was able to get it to work when the load balancer and the services being swapped out were all in the same stack, but I can’t seem to get it to work when the lb (in its own stack without other services) is swapping out links from services in one stack to services in another stack.

Apart from the advise from @willrstern with regards to tuning the images for fast container start time, a rolling redeployment can be fixed by the mentioned blue/green deployment. I automated this with a shell script when a new Docker image is pushed to the repo: https://medium.com/@duizendnegen/blue-green-rolling-redeployments-in-docker-cloud-with-harrow-io-4c492161ad41