TLDR; I am looking for a docker build and/or test framework/runner that is more maintainable than a bash script that can orchestrate tests for the docker images. Here are some unanswered questions my team has about the current solution I will describe in detail below.
- Does anyone know of a better way to manage intra-image dependencies?
- Has anyone found a test runner that works well with docker test containers (Containers that use your production containers as a base and runs tests against them)?
- Has anyone found a tool that is useful for building and maintaining your images (above and beyond what docker-compose provides)?
The long version:
I am reaching out to the community to see if others have hit the same challenges that I have recently hit, and if so, what you have done to overcome them. My team is responsible for maintaining a set of docker images that we use to execute long running processing jobs in AWS Batch. The batch framework and the pattern we follow to create these images works quite well, but up until recently, each image was handcrafted, and in order to update images, the person needed a deep understanding of the project, how each image ran, and how certain images depended on other images to be built. Each image has a set of bash scripts that would build, push, and if you were lucky, run a simple test. Essentially we lacked proper automation and testing tooling.
The challenges we are trying to overcome:
- Make building images easier (no more having to go into each image’s directory to build and push them separately).
- Make the dependencies between images clear and easy to manage.
- Make it clear which images are common and which are the actually batch job images.
- Have an easy way to test the images so we can plug this all into our Jenkins build.
What we have done so far:
To help solve the build and build dependency problem I turned to docker-compose. Docker-compose lets me have a series of compose files at the top level directory to define all the images and make it very clear what version each image is on, as well as clearly see the build context. We have a docker-compose.yml file that contains the batch jobs and a docker-compose.common.yml for the common/shared images
Example docker-compose.yml:
version: '3'
services:
video-metadata:
build:
context: video-metadata
args:
FFMPEG_IMAGE: "aws/batch/ffmpeg:4.1"
image: aws/batch/videometadata:3
This compose file makes it very easy to see that the metadata extraction job depends on ffmpeg:4.1 and that it is currently on version 3. There is no more going to several different files to get this information, it is all here and easy to read.
Example docker-compose.common.yml:
version: '3'
services:
ffmpeg:
build:
context: ffmpeg
args:
FFMPEG_FULL_VERSION: "4.1"
image: aws/batch/ffmpeg:4.1
The common compose file tells me that this image is not a job itself (and in some cases may not even be pushed to a repository), and clearly lays out all the configuration options for the ffmpeg image. I can also easily see that this image may be shared across multiple other jobs.
So far, these two files helped solve three of the challenges I first mentioned, but does nothing to solve the testing problem and introduces another challenge. The new challenge is that docker-compose has no way of determining what order to build images in. Lucky for us, the dependency list for our jobs is very short. If we just build all the common images first, then build the batch-job images, the build order is taken care of. This is a bit frightening for any complex project though. Does anyone know of a better way to manage intra-image dependencies?
To help manage these different docker-compose files I created a simple bash script to wrap common docker-compose commands. The script is quite simple and exposes the following functionality
./compose build #builds common and then the aws-batch images
./compose build {image-name} #builds specified image
./compose push #pushes only aws-batch images
./compose push {image-name} #pushes specified image
./compose test #uses docker-compose up to execute all images in docker-compose.test.yml
./compose test {image-name} #uses docker-compose run to run single test container.
The only new command here is test which under the covers uses the following series of docker-compose commands
docker-compose --file docker-compose.test.yml build
docker-compose --file docker-compose.test.yml up
##Use `docker-compose --file docker-compose.test.yml ps` to get exit codes of containers
docker-compose --file docker-compose.test.yml down
and a docker-compose.test.yml that looks like below
Example docker-compose.test.yml:
version: '3'
services:
video-metadata-test:
build:
context: video-metadata-test
args:
SUT_IMAGE: "aws/batch/videometadata:3"
environment:
AWS_CREDENTIALS: ${AWS_CREDENTIALS}
In this file I can see we have a test called video-metadata-test that tests the videometadata:3 image. It is very simple and easy to read.
And with that I have covered all of our problems by using docker-compose in a clever way, and creating a small wrapper bash script to simplify the command line user experience. My fear is that docker-compose really wasn’t built for this type of workflow, and a simple change to how it functions could bring this entire system down. My research on this topic lead me to two resources:
- https://alexei-led.github.io/post/docker_testing/
- https://github.com/GoogleContainerTools/container-structure-test
The first walks through testing with docker and arrives at the same conclusion that I did, which is use test containers. The second is a testing tool by google which is more geared towards ensuring a container is structured properly, and from what I can see, would be painful to configure for running the types of tests we are running here (full blown integration tests in some cases). I really think there is a hole in the docker workflow, and it mainly revolves around tests, which is why I am making this post. Has anyone else encountered similar problems, and what did you do to get around them? Specifically, do you have a test runner that you have found works well with docker test-containers? Have you found another tool that is useful for building and maintaining your images?
Thanks for reading,
Mark Haley