Inconsistent volume mounting of OS X directories in Docker for Mac

Docker Desktop
1

Expected behavior

Assume a Docker environment with no currently existing, running or stopped containers (verified with docker ps -a) and no currently existing volumes (as noted by docker volume ls). When running the following command, either using the terminal or a different wrapper to call the docker executable, you would expect a host directory to be mounted within a container:

docker run -d -v /Users/someuser/path/to/some/dir:/etc/mysql/conf.d --name=myname mysql:5.6.28

Actual behavior

Somehow a condition can be introduced whereby the host directory is not mounted in the container volume. By itself, even removing the container and the volume and creating new instances (using the same name and path values) does not relieve the issue for subsequent docker run invocations with the same parameters.

Strangely, there is a way to reliably remove the condition, although it is unclear why it has this effect. If you remove the prior container and volume in question, create and run a container using the same name, and have this container mount /Users to some (seemingly) arbitrarily named container directory, this volume is mounted successful. Upon removing this container and volume, the problem condition appears to be cleared.

Information

  • OS X Version 10.11.4 (build: 15E65)

  • Docker Version 1.11.1-beta10 (build: 6662)

  • Observed in particular in the official mysql:5.6.28 image

    • Note that /etc/mysql/conf.d is an existing directory in the image

  • From pinata diagnose -u

    • Most specific failure is: No error was detected

  • The problem condition can be consistently brought about when using scripted Ant macros that stop and removing containers as part of an automated build (see below)

    • The macros utilize Ant’s exec task to stop and remove containers

      • This usage should be equivalent to manually running the stop and remove container commands directly from the terminal

    • The macros have been adapted to run for either a native Docker app or with the Docker Toolbox, and there were previously no issues with them using Toolbox

    • The macros appear to work, in that containers are stopped and both containers and volumes are removed (as verified with docker ps -a and docker volume ls)

    • Once the problem condition has been introduced, directly running commands via the terminal does not produce different results or relieve the issue

    • Other containers, using other images, with other host bind mounts, are created and cleaned up using the same macros and do not experience issues with the volumes

Steps to reproduce the behavior

  1. Create and run a container with some name and some host-directory as a volume successfully mounted in the container
  2. Run an Ant build to stop and remove this container
  3. Verify that the container and volume no longer appear in docker ps -a and docker volume ls respectively
  4. Observe that subsequent attempts to create/run a container using the same name and host-directory volume mount do not mount the directory
2

@rbartel can you replicate this issue without using Ant? e.g by just using the docker CLI commands?

3

I have not yet been able to reliably enter the state wherein the error occurs, except via the Ant build, although I haven’t yet spent a lot of time trying. After I have entered the error state, I do consistently see the error even when exclusively using CLI commands from the terminal to create/start and stop/remove subsequent containers and volumes.

Also, to be clear, I am not using a custom Ant Docker task. I have exclusively been using Ant's exec task to call the appropriate docker CLI commands. I presume that by “e.g by just using the docker CLI commands?” you really mean “e.g by just using the terminal?” I have, strictly speaking, been using the docker CLI command (i.e., calls to /usr/local/bin/docker) for all operations, even those within Ant builds.

4

I can confirm the same behavior without Ant. Also I appreciate the info on the workaround. In my case for development purposes I’m exposing a python module as a volume. Unfortunately the container mount point creates a directory but files are not visible, and although it is mounted rw any attempt to create a file in the directory also fails. This persists even after reinstalling the Docker daemon, and am forced to mount to a different path.

I have found that either changing either the host volume path (simply renaming the directory), or the container mount path, will both result in a successful mount. Furthermore, changing any component of the path of either one will result in a successful mount. And after cycling through several path changes, the original intended volume mount will start working again, although as I can now no longer reproduce, I’m uncertain where in the sequence of changes it cleared. For example:

mymodule directory created, but empty. Container rm and docker reinstall does not fix:
/Users/foo/bar/mymodule:/usr/local/lib/python3.4/dist-packages/mymodule

Following sequence of changes:
/Users/foo/bar/mymodule:/usr/local/lib/python3.4/dist-packages/mymodule2 [Works]

/Users/foo/bar/mymodule2:/usr/local/lib/python3.4/dist-packages/mymodule [Works]

/Users/foo/bar2/mymodule:/usr/local/lib/python3.4/dist-packages/mymodule [Works]

And going back to the original, now it works:
/Users/foo/bar/mymodule:/usr/local/lib/python3.4/dist-packages/mymodule [Works]

5

@kaellis could you confirm that you are running Beta 13? If the issue exists in Beta 13, could you please post the unique id output by pinata diagnose -u?

6

Further occurrences have been rare in the last few weeks. This may have been partially influenced by a change to the design of the working environment, which now makes use of Docker Compose. The automation is still done via Ant builds, but now the containers are managed directly with a Compose config and calls to the docker-compose command.

However, I did see the same (or a very similar) problem again yesterday:

  • Docker version was 1.12.0-rc2-beta16 (build: 9493)
  • the first volume listed in the docker-compose.yml excerpt below would not mount
  • only the first volume was observed having issues; several other volumes were directly confirmed as mounting appropriately
  • only the problem volume was configured using an absolute path (which may have been incidental)
  • several attempts were made to stop and restart the container using docker-compose -f pathToFile down -v followed by docker-compose -f pathToFile up -d serviceName (both called within Ant exec task invocations), with this having no affect
  • configuration changes, first to the use a relative path, and then to the order of the volumes in the configuration, also did not have any affect
  • the problem was resolved after stopping and restarting the Docker application

Here is the volume config excerpt:

volumes:
     - /Users/rbartel/dev/sites/appmgr/build/docker:/var/www/docker
     - ../../conf/docker/apache/apache2.conf:/etc/apache2/apache2.conf
     - ../../conf/docker/apache/default-ssl.conf:/etc/apache2/sites-enabled/000-default.conf
     - ../../public_html:/var/www/public_html
     - ../../includes:/var/www/includes
     - ../../fonts:/var/www/fonts
     - ../../cronjobs:/var/www/cronjobs
     - ../../secure:/var/www/secure
     - ../../parsing:/var/www/parsing
     - ../../calendar:/var/www/calendar
     - ../../conf/deploy:/home/appmgr/deploy
     - ../../conf/test:/home/appmgr/test
     - ../../build/test:/home/appmgr/testbuild
7

I think I ran into this issue twice in the past few weeks. For me, renaming the mount dir inside the container fixes the problem. Renaming the dir back to the original name makes the issue re-appear.

Using Docker Desktop 4.22.0 (117440), Engine 24.0.5.