volumes_from not working as expected

I’m embarrassed to say that after spending a extremely long time trying to get a data-only container configuration working with Compose without any success. I am attempting to work with the following files, all of which are located in the same directory:

docker-compose.yml

bower:
  build: .
  dockerfile: Dockerfile.bower
  env_file:
    - env-common.env
  links:
    - npm
  volumes_from:
    - data
composer:
  build: .
  dockerfile: Dockerfile.composer
  env_file:
    - env-common.env
  volumes_from:
    - data
data:
  build: .
  env_file:
    - env-common.env
  dockerfile: Dockerfile.data
  volumes:
    - ./tmp:/tmp
    - ./www:/var/www/html
db:
  env_file:
    - env-common.env
  image: mariadb:latest
  ports:
    - "3306:3306"
  volumes_from:
    - data
gulp:
  build: .
  dockerfile: Dockerfile.gulp
  env_file:
    - env-common.env
  links:
    - bower
    - npm
  volumes_from:
    - data
npm:
  build: .
  dockerfile: Dockerfile.npm
  env_file:
    - env-common.env
  volumes_from:
    - data
wordpress:
  env_file:
    - env-common.env
  image: wordpress:latest
  links:
    - bower
    - composer
    - db:mysql
    - gulp
    - npm
    - wordpress-cli
  ports:
    - 8080:80
  volumes_from:
    - data
  volumes:
    - ./uploads:/var/www/html/wp-content/uploads
  working_dir: /var/www/html
wordpress-cli:
  build: .
  dockerfile: Dockerfile.wordpress-cli
  env_file:
    - env-common.env
  volumes_from:
    - data
  working_dir: /var/www/html

Dockerfile.bower

FROM node:latest

MAINTAINER Karl <karl@madeupemail.com>

RUN npm install -g bower

WORKDIR /tmp

ADD .bowerrc .bowerrc

ADD bower.json bower.json

ADD scripts/bower/* scripts/bower/

RUN bower install

RUN mkdir -p /var/www/html

RUN cp -a /tmp/ /var/www/html/

ENTRYPOINT ["bower"]

Dockerfile.composer

FROM composer/composer:latest

MAINTAINER Karl <karl@madeupemail.com>

WORKDIR /tmp

ADD composer.json composer.json

ADD composer.lock composer.lock

ADD scripts/composer/* scripts/composer/

RUN composer install

RUN mkdir -p /var/www/html

RUN cp -a /tmp/www/ /var/www/html/

ENTRYPOINT ["composer", "--ansi"]

Dockerfile.data

FROM buildpack-deps:latest

MAINTAINER Karl <karl@madeupemail.com>

RUN mkdir -p /var/www/html

RUN echo "hello" > /tmp/hello.txt

Dockerfile.gulp

FROM node:latest

MAINTAINER Karl <karl@madeupemail.com>

RUN npm install -g gulp

USER $user

WORKDIR /tmp

ADD gulpfile.babel.js ./

ADD gulp_tasks gulp_tasks

ADD *.json ./

ADD *.yml ./

ADD .* ./

ADD app app

RUN gulp all

RUN mkdir -p /var/www/html

RUN cp -a /tmp/www/ /var/www/html/

ENTRYPOINT ["gulp"]

Dockerfile.npm
FROM node:latest

MAINTAINER Karl <karl@madeupemail.com>

WORKDIR /tmp

ADD package.json package.json

ADD scripts/npm/* scripts/npm/

RUN npm install

RUN mkdir -p /var/www/html

RUN cp -a /tmp/ /var/www/html/

ENTRYPOINT ["npm"]

Dockerfile.wordpress-cli

FROM buildpack-deps:latest

MAINTAINER Karl <karl@madeupemail.com>

RUN curl -0 https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar > /usr/bin/wp && chmod +x /usr/bin/wp

When running docker-compose up, the issue becomes evident when “RUN gulp all” of the gulp service is run as I have assumed that the relevant files in the node_modules folder would be available to the gulp service, having previously been installed by the “RUN npm install” command in the npm service. Running the two commands directly from bash on the host machine gives me the result that I expect in that gulp finds its dependencies in the node_modules folder.

The actual issue reported is as follows

[15:40:56] Failed to load external module babel-core/register
[15:40:56] Failed to load external module babel/register
[15:40:56] Local gulp not found in /tmp
[15:40:56] Try running: npm install gulp
ERROR: Service 'gulp' failed to build: The command '/bin/sh -c gulp all' returned a non-zero code: 1

The way in which the docker-compose.yml file and the Dockerfiles are authored seems very clear to me. I’ve read and re-read the documentation, various blog posts, many forum posts and GitHub issues in the hope of learning about some subtle setting that will suddenly allow me to get this all working but to no avail.

The various Dockerfiles do not currently do much and may exhibit other areas where I need to add in further configuration, however they have “evolved” to this state over the course of simplifying them and trying everything that I have been able to think of trying in order to get the configuration working. For example, the “RUN mkdir -p /var/www/html” command in most of the Dockerfiles is a hacky way to be able to compensate for the lack of an available volume in order that I can proceed to the next step.

I have used docker-inspect on the data container, which returns the following:

[
{
    "Id": "55f94baa6505b112f5a582f892963c47c048188ed38d7d72a5a5d206ed35452e",
    "Created": "2015-12-02T13:11:57.238243474Z",
    "Path": "/bin/bash",
    "Args": [],
    "State": {
        "Status": "exited",
        "Running": false,
        "Paused": false,
        "Restarting": false,
        "OOMKilled": false,
        "Dead": false,
        "Pid": 0,
        "ExitCode": 0,
        "Error": "",
        "StartedAt": "2015-12-02T15:37:35.042980443Z",
        "FinishedAt": "2015-12-02T15:37:35.449058804Z"
    },
    "Image": "3b2c037dff66ff40e76eb9a7bc5bbbed55a86203f71e5e5eacfb4a5e0acd8a42",
    "ResolvConfPath": "/mnt/sda1/var/lib/docker/containers/55f94baa6505b112f5a582f892963c47c048188ed38d7d72a5a5d206ed35452e/resolv.conf",
    "HostnamePath": "/mnt/sda1/var/lib/docker/containers/55f94baa6505b112f5a582f892963c47c048188ed38d7d72a5a5d206ed35452e/hostname",
    "HostsPath": "/mnt/sda1/var/lib/docker/containers/55f94baa6505b112f5a582f892963c47c048188ed38d7d72a5a5d206ed35452e/hosts",
    "LogPath": "/mnt/sda1/var/lib/docker/containers/55f94baa6505b112f5a582f892963c47c048188ed38d7d72a5a5d206ed35452e/55f94baa6505b112f5a582f892963c47c048188ed38d7d72a5a5d206ed35452e-json.log",
    "Name": "/madeupsite_data_1",
    "RestartCount": 0,
    "Driver": "aufs",
    "ExecDriver": "native-0.2",
    "MountLabel": "",
    "ProcessLabel": "",
    "AppArmorProfile": "",
    "ExecIDs": null,
    "HostConfig": {
        "Binds": [
            "/Users/karl/Sites/madeupsite.com/www:/var/www/html:rw",
            "/Users/karl/Sites/madeupsite.com/tmp:/tmp:rw"
        ],
        "ContainerIDFile": "",
        "LxcConf": null,
        "Memory": 0,
        "MemoryReservation": 0,
        "MemorySwap": 0,
        "KernelMemory": 0,
        "CpuShares": 0,
        "CpuPeriod": 0,
        "CpusetCpus": "",
        "CpusetMems": "",
        "CpuQuota": 0,
        "BlkioWeight": 0,
        "OomKillDisable": false,
        "MemorySwappiness": null,
        "Privileged": false,
        "PortBindings": {},
        "Links": null,
        "PublishAllPorts": false,
        "Dns": null,
        "DnsOptions": null,
        "DnsSearch": null,
        "ExtraHosts": [],
        "VolumesFrom": [],
        "Devices": null,
        "NetworkMode": "default",
        "IpcMode": "",
        "PidMode": "",
        "UTSMode": "",
        "CapAdd": null,
        "CapDrop": null,
        "GroupAdd": null,
        "RestartPolicy": {
            "Name": "",
            "MaximumRetryCount": 0
        },
        "SecurityOpt": null,
        "ReadonlyRootfs": false,
        "Ulimits": null,
        "LogConfig": {
            "Type": "json-file",
            "Config": {}
        },
        "CgroupParent": "",
        "ConsoleSize": [
            0,
            0
        ],
        "VolumeDriver": ""
    },
    "GraphDriver": {
        "Name": "aufs",
        "Data": null
    },
    "Mounts": [
        {
            "Source": "/Users/karl/Sites/madeupsite.com/www",
            "Destination": "/var/www/html",
            "Mode": "rw",
            "RW": true
        },
        {
            "Source": "/Users/karl/Sites/madeupsite.com/tmp",
            "Destination": "/tmp",
            "Mode": "rw",
            "RW": true
        }
    ],
    "Config": {
        "Hostname": "55f94baa6505",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "MYSQL_ROOT_PASSWORD=madeuppassword2",
            "MYSQL_PASSWORD=madeuppassword1",
            "MYSQL_USER=wordpress",
            "MYSQL_DATABASE=wordpress",
            "user=build",
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
        ],
        "Cmd": [
            "/bin/bash"
        ],
        "Image": "madeupsite_data",
        "Volumes": {
            "/tmp": {},
            "/var/www/html": {}
        },
        "WorkingDir": "",
        "Entrypoint": null,
        "OnBuild": null,
        "Labels": {
            "com.docker.compose.config-hash": "268b6526775a5592fa46cd2eb1d81f509009177825fe603729d2def318333e5e",
            "com.docker.compose.container-number": "1",
            "com.docker.compose.oneoff": "False",
            "com.docker.compose.project": "madeupsite",
            "com.docker.compose.service": "data",
            "com.docker.compose.version": "1.5.1"
        }
    },
    "NetworkSettings": {
        "Bridge": "",
        "SandboxID": "",
        "HairpinMode": false,
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "Ports": null,
        "SandboxKey": "",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null,
        "EndpointID": "",
        "Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "IPAddress": "",
        "IPPrefixLen": 0,
        "IPv6Gateway": "",
        "MacAddress": "",
        "Networks": {
            "bridge": {
                "EndpointID": "",
                "Gateway": "",
                "IPAddress": "",
                "IPPrefixLen": 0,
                "IPv6Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "MacAddress": ""
            }
        }
    }
}
]

Both the Binds and Mounts keys in the output look promising to me but equally I don’t know enough about docker to know if that means that everything looks to be configured properly.

Running docker-inspect on the npm container shows what the volumes in Binds and Mounts also:

[
{
    "Id": "00443963c692391971678a71f95cf42d31d30f79f62618062a770a134d8f9425",
    "Created": "2015-12-02T15:39:17.734699249Z",
    "Path": "npm",
    "Args": [],
    "State": {
        "Status": "exited",
        "Running": false,
        "Paused": false,
        "Restarting": false,
        "OOMKilled": false,
        "Dead": false,
        "Pid": 0,
        "ExitCode": 1,
        "Error": "",
        "StartedAt": "2015-12-02T15:39:18.205907539Z",
        "FinishedAt": "2015-12-02T15:39:32.908821534Z"
    },
    "Image": "ecff9a27227f64e523bb666b88e338b4a491d90ffe7ef6e7c860865246091f03",
    "ResolvConfPath": "/mnt/sda1/var/lib/docker/containers/00443963c692391971678a71f95cf42d31d30f79f62618062a770a134d8f9425/resolv.conf",
    "HostnamePath": "/mnt/sda1/var/lib/docker/containers/00443963c692391971678a71f95cf42d31d30f79f62618062a770a134d8f9425/hostname",
    "HostsPath": "/mnt/sda1/var/lib/docker/containers/00443963c692391971678a71f95cf42d31d30f79f62618062a770a134d8f9425/hosts",
    "LogPath": "/mnt/sda1/var/lib/docker/containers/00443963c692391971678a71f95cf42d31d30f79f62618062a770a134d8f9425/00443963c692391971678a71f95cf42d31d30f79f62618062a770a134d8f9425-json.log",
    "Name": "/madeupsite_npm_1",
    "RestartCount": 0,
    "Driver": "aufs",
    "ExecDriver": "native-0.2",
    "MountLabel": "",
    "ProcessLabel": "",
    "AppArmorProfile": "",
    "ExecIDs": null,
    "HostConfig": {
        "Binds": [],
        "ContainerIDFile": "",
        "LxcConf": null,
        "Memory": 0,
        "MemoryReservation": 0,
        "MemorySwap": 0,
        "KernelMemory": 0,
        "CpuShares": 0,
        "CpuPeriod": 0,
        "CpusetCpus": "",
        "CpusetMems": "",
        "CpuQuota": 0,
        "BlkioWeight": 0,
        "OomKillDisable": false,
        "MemorySwappiness": null,
        "Privileged": false,
        "PortBindings": {},
        "Links": null,
        "PublishAllPorts": false,
        "Dns": null,
        "DnsOptions": null,
        "DnsSearch": null,
        "ExtraHosts": [],
        "VolumesFrom": [
            "55f94baa6505b112f5a582f892963c47c048188ed38d7d72a5a5d206ed35452e:rw"
        ],
        "Devices": null,
        "NetworkMode": "default",
        "IpcMode": "",
        "PidMode": "",
        "UTSMode": "",
        "CapAdd": null,
        "CapDrop": null,
        "GroupAdd": null,
        "RestartPolicy": {
            "Name": "",
            "MaximumRetryCount": 0
        },
        "SecurityOpt": null,
        "ReadonlyRootfs": false,
        "Ulimits": null,
        "LogConfig": {
            "Type": "json-file",
            "Config": {}
        },
        "CgroupParent": "",
        "ConsoleSize": [
            0,
            0
        ],
        "VolumeDriver": ""
    },
    "GraphDriver": {
        "Name": "aufs",
        "Data": null
    },
    "Mounts": [
        {
            "Source": "/Users/karl/Sites/madeupsite.com/www",
            "Destination": "/var/www/html",
            "Mode": "",
            "RW": true
        },
        {
            "Source": "/Users/karl/Sites/madeupsite.com/tmp",
            "Destination": "/tmp",
            "Mode": "",
            "RW": true
        }
    ],
    "Config": {
        "Hostname": "00443963c692",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "MYSQL_ROOT_PASSWORD=M3rcr3d1",
            "MYSQL_PASSWORD=m1ttw0ch13",
            "MYSQL_USER=wordpress",
            "MYSQL_DATABASE=wordpress",
            "user=build",
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "NPM_CONFIG_LOGLEVEL=info",
            "NODE_VERSION=5.1.0"
        ],
        "Cmd": null,
        "Image": "madeupsite_npm",
        "Volumes": null,
        "WorkingDir": "/tmp",
        "Entrypoint": [
            "npm"
        ],
        "OnBuild": null,
        "Labels": {
            "com.docker.compose.config-hash": "75c65c4e1ddce4943e47a65718ea24b77fbe0f8add58439be496a387e887d3cd",
            "com.docker.compose.container-number": "1",
            "com.docker.compose.oneoff": "False",
            "com.docker.compose.project": "madeupsite",
            "com.docker.compose.service": "npm",
            "com.docker.compose.version": "1.5.1"
        }
    },
    "NetworkSettings": {
        "Bridge": "",
        "SandboxID": "",
        "HairpinMode": false,
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "Ports": null,
        "SandboxKey": "",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null,
        "EndpointID": "",
        "Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "IPAddress": "",
        "IPPrefixLen": 0,
        "IPv6Gateway": "",
        "MacAddress": "",
        "Networks": {
            "bridge": {
                "EndpointID": "",
                "Gateway": "",
                "IPAddress": "",
                "IPPrefixLen": 0,
                "IPv6Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "MacAddress": ""
            }
        }
    }
}
]

This is not true of the gulp service however, although I assume that is because the Dockerfile caused an exit state that wasn’t 0:

[
{
    "Id": "fc84161dc9ef507d2460280e74861a11bcf8fae81ec3bc062001ba987e66120d",
    "Created": "2015-12-02T15:40:55.91570302Z",
    "Path": "/bin/sh",
    "Args": [
        "-c",
        "gulp all"
    ],
    "State": {
        "Status": "exited",
        "Running": false,
        "Paused": false,
        "Restarting": false,
        "OOMKilled": false,
        "Dead": false,
        "Pid": 0,
        "ExitCode": 1,
        "Error": "",
        "StartedAt": "2015-12-02T15:40:56.014825258Z",
        "FinishedAt": "2015-12-02T15:40:56.358674836Z"
    },
    "Image": "a0fb55324ac38d64125b869549d1fc9af4fb6fb03ef123c63a8e6dc65d176f11",
    "ResolvConfPath": "/mnt/sda1/var/lib/docker/containers/fc84161dc9ef507d2460280e74861a11bcf8fae81ec3bc062001ba987e66120d/resolv.conf",
    "HostnamePath": "/mnt/sda1/var/lib/docker/containers/fc84161dc9ef507d2460280e74861a11bcf8fae81ec3bc062001ba987e66120d/hostname",
    "HostsPath": "/mnt/sda1/var/lib/docker/containers/fc84161dc9ef507d2460280e74861a11bcf8fae81ec3bc062001ba987e66120d/hosts",
    "LogPath": "/mnt/sda1/var/lib/docker/containers/fc84161dc9ef507d2460280e74861a11bcf8fae81ec3bc062001ba987e66120d/fc84161dc9ef507d2460280e74861a11bcf8fae81ec3bc062001ba987e66120d-json.log",
    "Name": "/stoic_jepsen",
    "RestartCount": 0,
    "Driver": "aufs",
    "ExecDriver": "native-0.2",
    "MountLabel": "",
    "ProcessLabel": "",
    "AppArmorProfile": "",
    "ExecIDs": null,
    "HostConfig": {
        "Binds": null,
        "ContainerIDFile": "",
        "LxcConf": null,
        "Memory": 0,
        "MemoryReservation": 0,
        "MemorySwap": 0,
        "KernelMemory": 0,
        "CpuShares": 0,
        "CpuPeriod": 0,
        "CpusetCpus": "",
        "CpusetMems": "",
        "CpuQuota": 0,
        "BlkioWeight": 0,
        "OomKillDisable": false,
        "MemorySwappiness": null,
        "Privileged": false,
        "PortBindings": null,
        "Links": null,
        "PublishAllPorts": false,
        "Dns": null,
        "DnsOptions": null,
        "DnsSearch": null,
        "ExtraHosts": null,
        "VolumesFrom": null,
        "Devices": null,
        "NetworkMode": "default",
        "IpcMode": "",
        "PidMode": "",
        "UTSMode": "",
        "CapAdd": null,
        "CapDrop": null,
        "GroupAdd": null,
        "RestartPolicy": {
            "Name": "",
            "MaximumRetryCount": 0
        },
        "SecurityOpt": null,
        "ReadonlyRootfs": false,
        "Ulimits": null,
        "LogConfig": {
            "Type": "json-file",
            "Config": {}
        },
        "CgroupParent": "",
        "ConsoleSize": [
            0,
            0
        ],
        "VolumeDriver": ""
    },
    "GraphDriver": {
        "Name": "aufs",
        "Data": null
    },
    "Mounts": [],
    "Config": {
        "Hostname": "11aafb97cfeb",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "NPM_CONFIG_LOGLEVEL=info",
            "NODE_VERSION=5.1.0"
        ],
        "Cmd": [
            "node"
        ],
        "Image": "a0fb55324ac38d64125b869549d1fc9af4fb6fb03ef123c63a8e6dc65d176f11",
        "Volumes": null,
        "WorkingDir": "/tmp",
        "Entrypoint": null,
        "OnBuild": [],
        "Labels": {}
    },
    "NetworkSettings": {
        "Bridge": "",
        "SandboxID": "",
        "HairpinMode": false,
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "Ports": null,
        "SandboxKey": "",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null,
        "EndpointID": "",
        "Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "IPAddress": "",
        "IPPrefixLen": 0,
        "IPv6Gateway": "",
        "MacAddress": "",
        "Networks": {
            "bridge": {
                "EndpointID": "",
                "Gateway": "",
                "IPAddress": "",
                "IPPrefixLen": 0,
                "IPv6Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "MacAddress": ""
            }
        }
    }
}
]

I have tried to create the volumes without mounting them on to the host and my experience is the same. Also, if I add a file or folder into the mount point from the host then that file or folder is reported by the ls command from within the docker container, however any file or folder written to the folder from any container is not visible within the host folder.

I am obviously fundamentally misunderstanding something about the configuration or the way in which volumes_from works but as it all seems quite clear, I have no idea what stone I have left unturned in my trial and error approach to getting this working.

I am on OS-X El Capitan on an early 2009 Mac Pro, running docker version 1.9.1 (build a34a1d5) , docker-compose version: 1.5.1 and docker-machine docker-machine version 0.5.1 (HEAD). All 3 commands were installed from Homebrew. I have deleted and recreated the VirtualBox VM using docker-machine to ensure that boot2docker is at the latest version.

Any insight that anyone has would be greatly appreciated please.