Docker volume mounting shows empty directory with a linux mounted directory?

General Discussions
#1

I have a linux volume, dev/sdh, that’s mounted at /media/ebs_volume. When I try to mount it into a docker container using -v /media/ebs_volume:/tmp/foo:ro, the mounted directory in /tmp/foo is empty! ls -al /tmp/foo from inside the container is empty but ls -al /media/ebs_volume on the host has plenty of stuff in it.

I’ve tried mounting the volume on the host in different locations, and mounting just a single specific file from the host instead of the entire directory. If I mount something from the home directory that seems to work without any issue.

Anyone have any ideas what might be happening here or best way to debug this?

Possibly useful output with some edits:

ec2-user@ip-10-0-2-50 ~]$ ls -al /media/ebs_volume/
total 28
drwxr-xrwx 4 root     root      4096 Jan 20 21:34 .
drwxr-xr-x 3 root     root      4096 Jan 29 23:18 ..
drwx------ 2 root     root     16384 Jan 20 21:29 lost+found
drwxrwxr-x 3 ec2-user ec2-user  4096 Jan 20 21:38 stuff
[ec2-user@ip-10-0-2-50 ~]$ docker run -it --rm -v /media/ebs_volume:/tmp/foo:ro nginx /bin/bash
root@7bd154b9aa1c:/# ls -al /tmp/foo/
total 4
drwxr-xr-x 2 root root 4096 Jan 29 23:18 .
drwxrwxrwt 3 root root   16 Jan 30 17:33 ..

[ec2-user@ip-10-0-2-50 ~]$ mount
proc on /proc type proc (rw,relatime)
/dev/xvda1 on / type ext4 (rw,noatime,data=ordered)
/dev/xvdh on /media/ebs_volume type ext4 (rw,relatime,data=ordered)
...

[ec2-user@ip-10-0-2-50 ~]$ ps aux | grep docker
root      2950  0.0  4.4 719812 45416 ?        Sl   Jan29   0:22 /usr/bin/dockerd
root      2956  0.0  1.0 229364 11136 ?        Ssl  Jan29   0:01 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim docker-containerd-shim --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --runtime docker-runc
root      3086  0.0  1.4 114664 15044 ?        Ssl  Jan29   0:00 docker events

[ec2-user@ip-10-0-2-50 ~]$ docker version
Client:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   7392c3b/1.12.6
 Built:        Fri Jan  6 22:16:21 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   7392c3b/1.12.6
 Built:        Fri Jan  6 22:16:21 2017
 OS/Arch:      linux/amd64

[ec2-user@ip-10-0-2-50 ~]$ cat /proc/version
Linux version 4.4.41-36.55.amzn1.x86_64 (mockbuild@gobi-build-60008) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Wed Jan 18 01:03:26 UTC 2017

[ec2-user@ip-10-0-2-50 ~]$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
1436d3223b27        nginx               "/bin/bash"              47 seconds ago      Up 46 seconds       80/tcp, 443/tcp     happy_leakey

[ec2-user@ip-10-0-2-50 ~]$ docker inspect 1436d3223b27
[
    {
        "Id": "1436d3223b270f78654d26682c469f4d7bdd24766c60f4052cb1a3fb60c86c12",
        "Created": "2017-01-30T17:34:12.430135675Z",
        "Path": "/bin/bash",
        ...
        "Image": "sha256:cc1b614067128cd2f5cdafb258b0a4dd25760f14562bcce516c13f760c3b79c4",
        ...
        "Driver": "devicemapper",
        "MountLabel": "",
        ...
        "HostConfig": {
            "Binds": [
                "/media/ebs_volume:/tmp/foo:ro"
            ],
            "ContainerIDFile": "",
            ...
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            ...
        },
        ...
        "Mounts": [
            {
                "Source": "/media/ebs_volume",
                "Destination": "/tmp/foo",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            ...
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "NGINX_VERSION=1.11.9-1~jessie"
            ],
            "Cmd": [
                "/bin/bash"
            ],
            "Image": "nginx",
            "Volumes": null,
            ...
        },
        ...
    }
]
#2

Hey, did you get the solution for this problem. I also seem to have the same problem with Linux mounted directories.

#3

Not the finest of answers, but I found restarting the docker service, docker service restart seemed to do the trick.
As I was on elasticbeanstalk, I found adding the following to a .config file to work:

...

commands:
    ...
    02-restart-docker:
        command: service docker restart && docker restart ecs-agent
    ...