Path external hard disk passed by env variable to volume in compose yml

Docker Engine Compose
1

An env variable $BLOCK containing the path to an external hard disk is defined in Ubuntu 20.04 as such:

echo $BLOCK 
/mnt/103bae77-3b0b-4bdd-8573-d2c5b8734ab5/BLOCK

The compose yml, see below, should use it to define a volume but the container fails to bind it. If the absolute path is used, then binding works. $HOME also works. $(readlink) in turn fails.

How to modify:
- ${BLOCK}/SimNet/examples:/examples
to bind an external hard disk to the container?

Any ideas are highly appreciated. Thank you!

# compose-simnet.yml

version: '2.1'

services:

  simnet-service:

    shm_size: '1g'

    ulimits:
      memlock: '-1'
      stack: '67108864'

    volumes:
      - /mnt/103bdk77-3b0b-4baa-8473-d2c5b5384ab5/BLOCK/SimNet/examples:/examples
      #- ${BLOCK}/SimNet/examples:/examples
      #- $(readlink -f ${HOME}/BLOCK/)/SimNet/examples:/examples
      - ${HOME}/Docker/volume-simnet:/home/volume-simnet

    runtime: 'nvidia'

    image: "simnet:21.06"

    command: tail -F /dev/null
2

Seems like you didn’t export the BLOCK variable :slight_smile:

Preparation:

me@swarm1:~/docker-tests/test-env$ cat docker-compose.yml
version: '2.1'
services:
  simnet-service:
    shm_size: '1g'
    ulimits:
      memlock: '-1'
      stack: '67108864'
    volumes:
      - ${BLOCK}/SimNet/examples:/examples
    runtime: 'nvidia'
    image: "simnet:21.06"
    command: tail -F /dev/null

Test1: variable declaration

me@swarm1:~/docker-tests/test-env$ BLOCK=/mnt/dummy
me@swarm1:~/docker-tests/test-env$ docker-compose --file docker-compose.yml config
WARNING: The BLOCK variable is not set. Defaulting to a blank string.
services:
  simnet-service:
    command: tail -F /dev/null
    image: simnet:21.06
    runtime: nvidia
    shm_size: 1g
    ulimits:
      memlock: -1
      stack: 67108864
    volumes:
    - /SimNet/examples:/examples:rw
   version: '2.1'

Test2: exported variable declaration

me@swarm1:~/docker-tests/test-env$ export BLOCK=/mnt/dummy
me@swarm1:~/docker-tests/test-env$ docker-compose --file docker-compose.yml config
services:
  simnet-service:
    command: tail -F /dev/null
    image: simnet:21.06
    runtime: nvidia
    shm_size: 1g
    ulimits:
      memlock: -1
      stack: 67108864
    volumes:
    - /mnt/dummy/SimNet/examples:/examples:rw
version: '2.1'
1 Like
3

Yes, I can reproduce the substitution using config. However, when I enter the container, go to /test and ls, then no files are visible as opposed to providing the full path in .yml. Having parsed the files, that should not be possible. Looks like a bug.

Here is a simpler container to reproduce the problem:

# compose-ubuntu.yml
version: '2.1'
services:
  ubuntu-service:
    image: ubuntu
    volumes:
      - ${BLOCK}/test:/test   
    command: tail -F /dev/null

using these commands:

docker-compose --file compose-ubuntu.yml config
services:
  ubuntu-service:
    command: tail -F /dev/null
    image: ubuntu
    volumes:
    - /mnt/103bdk77-3b0b-4baa-8473-d2c5b5384ab5/BLOCK/test:/test:rw
version: '2.1'

docker-compose --file compose-ubuntu.yml up

docker exec -ti compose_ubuntu-service_1 /bin/bash
root@f26767ea242e:/# cd /test
root@f26767ea242e:/test# ls -al
total 8
drwxr-xr-x 2 root root 4096 Jul 31 10:09 .
drwxr-xr-x 1 root root 4096 Jul 31 10:09 ..
4

Just to be sure: is the value of $BLOCK correct?

From your first post:

Though above the /mnt/xxx parts look alike, they are not exactly the same? In your last post it seems that $BLOCK was actually /mnt/103bdk77-3b0b-4baa-8473-d2c5b5384ab5/BLOCK, which apparently worked when used in the absolute path as quoted from the first post above (but not in the last post). So I guess the mount name and the value of $BLOCK simply changed during testing, but you may want to make sure.

1 Like
5

Thank you - well spotted :innocent: $BLOCK is identical but was manually modified for the forum; however, that makes no sense, so I started to copy and pasted the real value.

Can some reproduce the problem?

6

I cannot reproduce it.

That is: for a moment I thought I could reproduce your simpler test case, but my mount did not have a /test folder, so then for me ${BLOCK}/test should be empty indeed. Fixing that to use a different folder in the export BLOCK=, I get the expected results. (On a Mac, using /Volumes/xxx rather than /mnt/xxx.)

Why did you think readlink may help?

(Aside: your simpler use case also works for a symbolic link, for me.)

1 Like
7

Problem (mostly) solved - as often, two issues were superimposed:

  1. the external hard disk was mounted as root preventing access, fixed.

  2. then these two commands instead of using Portainer work to get access:

docker-compose --file <docker-compose.yml> up
docker exec -it <container name> /bin/bash

I have never been able to find a combined single command. It would also be interesting to know why Portainer fails.

8

I followed your remark using a symlink called BLOCK in conjunction with the full path:

  - /home/user/BLOCK/SimNet/examples:/examples

which also works. This is also the solution for a Portainer Stack that only substitutes environment variables when linked to a git repository (good to know).