Chaos Monkey for Docker?

General Discussions
1

Is there a tool similar to Chaos Monkey that can do chaos testing for Docker containers? For example, if you have an application composed of many microservices, each running in their own container, and you want to test that they are resilient in the face of random failures.

Thanks.

2

Hi, would also be interested. If there’s nothing already it might be a good idea to start it and since the ChaosMonkey is for ec2 I would suggest to start a fork for ecs.

3

Use Docker Swarm and run this script as a daemon :wink: :smiley: :stuck_out_tongue:

#!/bin/bash

while true; do
    docker rm -f $(docker ps -aq | sort -R | head -n 1)
    sleep "$RANDOM"
done
1 Like
4

Hi, if you are still interested, I’ve created a small tool that does it: chaos testing for Docker.
Please, take a look https://github.com/gaia-adm/pumba

6

I tried pumba on Docker Cloud. I set my containers to autorestart=always, but when I kill a container with pumba the container does not restart. Is this a Docker Cloud issue or a pumba issue. I tried the same test locally on my laptop killing containers that were set to restart and it did not work… Again, is this a pumba issue or am I not understanding docker restart?

7

The problem is with Docker misleading restart policy names. Pumba just invokes docker kill command, but Docker container started with restat=always does not actually restart container when it stopped or killed. I think it will restart the container only if the main process exits with non-zero code.
One possible “solution” would be to handle some STOP signal an exit with non-zero code, then use pumba kill --singal <STOPSIG> to send this signal to the main process within Docker container.

8

Thanks so much for your quick response. I just did a quick test and created a container that would sleep for a few seconds then exit 1. When I run this test container with --restart always it works so that just confirms what you are saying. Not sure if I want to handle the STOP signal but I will consider it.

I really think my issue is with Docker Cloud, not pumba. I was expecting Docker Cloud to be a declarative scheduler and if I scale a container to 5 instances and there are only 4 running, regardless of why, I want Docker Cloud to start up a 5th container. I think if I were using DC/OS or K8S then it would be possible to use pumba as a chaos monkey.

Knowing more what I am looking for if you have any suggestions let me know. I really like pumba, keep up the good work.