Forwarding filesystem through multiple docker-compose instances using volumes

In a CI-setup I’m working on we’re using kubernetes to run a docker container which runs docker-compose which is performing end-to-end testing. As I was working on this setup, I noticed I couldn’t mount the filesystem through to the innermost container. I’ve set up a repo which reproduces the issue with just docker-compose here:

In the example above I’m basically running two layers of docker-compose and trying to set up a chain of volumes in order to pass the imagined source code through to the innermost container.

file system ---> system docker-compose ---> docker-compose image ---> container

The first volume seems to work as I can list the file before running the next instance of docker-compose. The second volume definition though seems to believe there are no files on the host (in the docker-compose container). I assume this has to do somehow with the fact that the docker-runtime is shared between the system and container docker-compose and that those files are the result of a volume.

I would be very happy if someone could help me understand what’s going on and what my options are at this point! :sweat_smile:

Tested on

  • Docker for Mac: 18.03.1-ce-mac65 (24312)
  • Engine: 18.03.1-ce
  • Compose: 1.21.1

How to try it yourself

git clone https://github.com/dnjstrom/bug-multi-docker-compose-volumes.git
cd bug-multi-docker-compose-volumes
docker-compose run --rm test

My output

The files in /outer and /inner should theoretically be the same, but they’re not.

❯ docker-compose run --rm test
Creating network compose_default with the default driver
Outer container:
/outer
total 16
drwxr-xr-x    6 root     root           192 Jul 17 16:06 .
drwxr-xr-x    1 root     root          4096 Jul 17 16:17 ..
-rw-r--r--    1 root     root           158 Jul 13 14:16 README.md
-rw-r--r--    1 root     root           292 Jul 17 16:13 docker-compose.test.yml
-rw-r--r--    1 root     root           168 Jul 17 16:17 docker-compose.yml
Inner container:
/inner
total 4
drwxr-xr-x    2 root     root            40 Jul 17 16:13 .
drwxr-xr-x    1 root     root          4096 Jul 17 16:17 ..

The issue here stems from the fact that reads for mounted volumes and the mounted files are separated in different layers. So what’ll end up in the 2nd container is whatever was in the corresponding location in the 1st container before the host fs was mounted in.

The way I got around this was to run Docker-in-Docker (DIND) rather than forwarding the host docker socket. This has a number of serious tradeoffs though that might not make it a viable solution for everyone.

Checkout this new container runtime (runc) from Nestybox; it enables Docker to deploy containers that can run Docker (and Docker compose) inside without resorting to unsecure privileged containers and in total isolation from the Docker on the host. It’s useful for CI setups as well as for Docker sandboxing. The Nestybox blog has several examples.