Map a host directory to a Docker container directory

General
1

Hi everybody. I’m relatively newbie to docker so sorry if my question is quite simple.

I’m trying to build a multi-container App which will be launch via a docker-compose.yml

I need that when I launch the App, two of the services get a storage directory located in my host computer, copy its content to the container and that if any of the container adds a new file to this folder it is saved in my computer local directory and also in the other container directory. In other words a need that changes applied to my local directory manually by me or by any of the containers gets reflected in the other ones.

I have been reading some documentations and to achieve this I have tried to declare a volume in the docker-compose and assign it to each service, however I don’t know how to tell the volume itself where the folder it needs to take as a source is located. I need the volume to represent the content of the MABSED/orchestrator/data/ directory.

This is my project tree overall view:

MABSED/
|_ docker-compose.yml
|_ ...
|_ streamer/
|  |_ Dockerfile
|  |_ startStreaming.py
|  |_ credentials.py
|_ orchestrator/
   |_ Dockerfile
   |_ requirements.txt
   |_ tasks.py
   |_ my_sched.py
   |_ data/
   |  |_ streaming/
   |  |_ preprocessed/
   |  |_ results/
   |_ detector/
   |_ filter/
   |_ lemmatizer/

The services which will share the storage are orchestrator and streamer. This is my docker-compose.yml file:

version: '2'

services:
  dashboard:
    build: demo-dashboard/
    ports:
     - "8080:8080"
    environment:
      - ES_ENDPOINT_EXTERNAL=http://localhost:9200
      - http.cors.enabled=true
      - http.cors.allow-origin=ES_ENDPOINT_EXTERNAL
      - http.cors.allow-headers=Content-Type, Access-Control-Allow-Headers, Authorization, X-Requested-With
      - http.cors.allow-credentials=true
    volumes:
     - ./demo-dashboard:/usr/src/app
    networks:
      - dashboard-network

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.7.0
    environment:
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - http.cors.enabled=true
      - http.cors.allow-origin=http://localhost:8080
      - http.cors.allow-headers=Content-Type, Access-Control-Allow-Headers, Authorization, X-Requested-With
      - http.cors.allow-credentials=true
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    mem_limit: 1g
    cap_add:
      - IPC_LOCK
    volumes:
      - esdata1:/usr/share/elasticsearch/data
    networks:
      - dashboard-network
    ports:
      - 9200:9200

  orchestrator:
    image: orchestrator-mabsed
    build: orchestrator/
    environment:
      ES_HOST: 'elasticsearch'
    tty: true
    volumes:
      - socialdata1:/usr/src/app/orchestrator/data

  streamer:
    image: streamer-mabsed
    build: streamer/
    tty: true
    volumes:
      - socialdata1:/usr/src/orchestrator/data

volumes:
  esdata1:
    driver: local
  socialdata1:
    driver: local
    driver_opts:
      type: none
      device: $PWD/orchestrator/data/
      o: bind

networks:
  dashboard-network:
    driver: bridge

This is my Orchestrator service Dockerfile:

FROM python:3.6

WORKDIR /usr/src/app/

COPY requirements.txt ./
RUN pip3 install --user -r requirements.txt

COPY my_sched.py ./
COPY tasks.py ./
COPY detector/ ./detector/
COPY filter/ ./filter/
COPY lemmatizer/ ./lemmatizer/

CMD python my_sched.py

And this is my Streamer service Dockerfile:

FROM python:3.6

RUN pip3 install --user tweepy

WORKDIR /usr/src/app/
COPY startStreaming.py /usr/src/app/
COPY credentials.py /usr/src/app/

CMD python startStreaming.py

If I try to do docker-compose up this is what I get:

ERROR: for mabsed_streamer_1 Cannot start service streamer: error while mounting volume ‘/var/snap/docker/common/var-lib-docker/volumes/mabsed_socialdata1/_data’: error while mounting volume with options: type=‘none’ device=‘/orchestrator/data/’ o=‘bind’: no such file or directory
Creating mabsed_streamer_1 … error

What am I missing?

2

Sorry for spam, but any solution?

3

error while mounting volume with options: type=‘none’ device=’/orchestrator/data/’ o=‘bind’: no such file or directory
Creating mabsed_streamer_1 … error

Seems like $PWD is ignored. I am quite sure, if you use an absolut path this error will be gone.

4

But I want to upload this project to a repository and to work in any system which download and run it, so specifying my own absolute path won’t allow this.

Is there another way to write in the docker-compose.yml the source path of the host directory which feed the volume/mount?

5

Either treat your compose.yml as reference stack and provide enough comments to highlight what needs to be adopted to the own environment. OR externalise the volume to force the users to create a volume from the command line. In both cases you will end up providing additional documentation :wink:

Of course there is one way: remove all your volumes and use a direct bind declaration, like you did for the ‘dasboard’ service.

update: is there a reason that you use three differents volume types? One is a direct bind (on dasboard), One is a named volume with default parameters (esdata1) and one is a named volume with bind (socialdata1).

6

I added the absolute path to the device field as you told me and now the system deployed propertly. However if I navigate in the Streamer container the repository /usr/src/orchestrator/ doesn’t exist, so it seems that the host repository hasn’t mapped to the container with the volume instruction. Is this behavior normal or it should appear there?

7

For socialdata1, any content existing in the host source folder should be visable in the container under /usr/src/app/orchestrator/data and /usr/src/orchestrator/data (did you declare it intentionaly like this?!). Any content written in one of these container folders will be visible on the host source folder.

Fir the source folder is a plain folder, then: Yes, the content should be visible in the container and vise versa.

Though, if the source folder is a mount point, and you mount a remote share into the folder after starting the container, the container will still see the content before the remote share was mountet.