How to stop missing volumes prevent containers starting?

ldexterldesign · October 5, 2020, 10:41am

Hi all,

Hope you’re well

I have a container which won’t start when a volume path is missing (e.g. network path not found)

Until now I’ve been commenting in/out the offending path when it is/isn’t accessible but there must be a better way - would someone be kind enough to point me to a better solution?

Brainstorming, I’m thinking:

Symbolic link(s)
Separate linked container (possible?)
Dynamic/hotswappable volume paths (possible?)

Just reading through docs so hopefully some of these thoughts are possible

Hope to hear back

Sincerely

  jellyfin:
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/London
    image: linuxserver/jellyfin
    ports:
      - 8096:8096
    restart: unless-stopped
    volumes:
#    - /Volumes/synology/gtd/home/reference:/-data/jellyfin/synology
      - ~/Downloads/-data/-config/jellyfin:/config
      - ~/Downloads/-data:/-data/jellyfin/apple

ldexterldesign · October 8, 2020, 4:47am

I guess something like –devices but for network drives would be useful…?

--devices is better documented here IMO

meyay · November 19, 2020, 6:43pm

First of all: you don’t use volumes, you use bind-mounts.

Bind-mounts are using “mount --bind /src/path /target/path” (you can test its behavior on any linux system right away) . Whenever you map a host path (regardless wether its a folder on disk, a mountpoint for a drive, a mountpoint for a remote share) into a container path, the host path will be mounted into the container path. If you dismount/re-mount a drive or remoteshare on the host, the handle for the mount will change, but the container will still point to the old handle. Though, if you bind-mount the parent folder of the mounted drive or network share into the container, you could probably leverage bind propagations to make it work.

ldexterldesign · November 21, 2020, 9:47pm

Thank you so much!

Yes, bind mounts was the solution

I’ve been wanting to solve this issue for months

Sincerely

meyay · November 22, 2020, 10:23am

Since you already used bind-mounts from the beginning, do you mind sharing the exact solution?

If you only switched to mount the parent folder without specifying the correct mount propagation, the problem of “stale” submounts might still effect you. Which mount propagation did you finaly use?

ldexterldesign · November 24, 2020, 12:23am

version: "3"
services:

  jackett:
    container_name: jackett
    environment:
      - PGID=1000
      - PUID=1000
      - TZ=Europe/London
    image: linuxserver/jackett:development
    ports:
      - 9117:9117
    restart: unless-stopped
    volumes:
      - ~/Downloads/-data/-config/jackett:/config

  jellyfin:
    container_name: jellyfin
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/London
    image: linuxserver/jellyfin
    ports:
      - 8096:8096
    restart: unless-stopped
    volumes:
      - ~/Downloads/-data/-config/jellyfin:/config

      - type: bind
        source: /Volumes
        target: /-data/jellyfin/synology
        read_only: true

      - type: bind
        source: ~/Downloads/-data
        target: /-data/jellyfin/apple
        read_only: true

  jellyfin_beta:
    container_name: jellyfin_beta
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/London
    image: jellyfin/jellyfin-vue:unstable
    ports:
      - 8097:80
    restart: unless-stopped
    volumes:
      - ~/Downloads/-data/-config/jellyfin:/config

      - type: bind
        source: /Volumes
        target: /-data/jellyfin/synology
        read_only: true

      - type: bind
        source: ~/Downloads/-data
        target: /-data/jellyfin/apple
        read_only: true

  lidarr:
    container_name: lidarr
    environment:
      - PGID=1000
      - PUID=1000
      - TZ=Europe/London
    image: linuxserver/lidarr:nightly
    ports:
     - 8686:8686
    restart: unless-stopped
    volumes:
      - ~/Downloads/-data/-config/lidarr:/config
      - ~/Downloads/-data:/-data

  radarr:
    container_name: radarr
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/London
    image: linuxserver/radarr:nightly
    ports:
     - 7878:7878
    restart: unless-stopped
    volumes:
      - ~/Downloads/-data/-config/radarr:/config
      - ~/Downloads/-data:/-data

  sonarr:
    container_name: sonarr
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/London
    image: linuxserver/sonarr:preview
    ports:
     - 8989:8989
    restart: unless-stopped
    volumes:
      - ~/Downloads/-data/-config/sonarr:/config
      - ~/Downloads/-data:/-data

Since you already used bind-mounts from the beginning, do you mind sharing the exact solution?

No, I was using volumes for everything before you suggested bind mounts

Which mount propagation did you finally use?

I think I read that there’s limitations around propagation if the host isn’t Linux, which is the case for me because my host is macOS

Thankfully I didn’t need propagation

Sincerely

meyay · November 24, 2020, 6:46am

I can see where the confusion commes from. Let me explain it with an example:

version: "3.8"
services:
  app:
    ...
    volumes:
      - type: volume
        source: realvolume
        target: /data
      - realvolume:/data
      - type: bind
        source: /host/path
        target: /data
      - /host/path:/data
      - type: volume
        source: realvolumebakedbybind
        target: /data
      - realvolumebakedbybind:/data
     ...

..
volumes:
  realvolume:
    ..
  realvolumebakedbybind:
   driver: local
    driver_opts:
      type: none
      o: bind
      device: /host/path

```
    ...

In the volume declaration of app, the first two entries are the long and short syntax for volumes, both have a matching volume declaration underneath the top level section volumes and are listend when docker volume ls is executed. A volume will copy pre-existing content in the container folder back into the volume (unless you configure it to not do so). Some images rely on this mechanism!

The third and forth entries are the long and short syntax for bind-mounts - no matching declartion in the volumes top level section, not listed with docker volume ls. A bind-mount “replace” the target folder in the container, without copying pre-existing data from the container back to the bind-mount.

To make confusion perfect, it is also possible to declare a volume baked by a bind (last two entries). It would be listed with docker volume ls as well. These volume provide the copy pre-existing data mechanism as well.

After having this sorted out, I am even more confused about what actualy brought the solution… Was it just mounting the parent folder?

ldexterldesign · November 26, 2020, 7:07pm

@meyay

Thanks for excellent summary!

Perhaps I wasn’t in the right head space at the time but I felt like I sank a whole evening into trying to understand Docker data storage before our last interaction, and I still didn’t grok it properly. I think I understand it a bit better now.

Firstly, I now acknowledge I was never using docker volumes, as you mention, and was using docker bind mounts all along

Secondly, because I updated my bind mount syntax to the long syntax and specified read_only: true I think this fixed my issue

Lastly, I’m going to read the manage app data docs again but if I don’t understand something would you be willing to help?

Sincerely

ldexterldesign · November 26, 2020, 7:13pm

@nhazlett @pkennedyr

I think a better topic title would be:
How to stop missing bind mounts preventing containers starting

Aside, it would be useful if you allowed topic authors/owners to do this themselves

Sincerely

meyay · November 27, 2020, 7:11pm

In case of trouble, prep your case and post it If the level of provided details is sufficient and the desired outcome is something that sparks my interessent, then I usualy pitch in.

Though some topics realy don’t spark my interesst: like MACVLAN network, systemd, ui-applications, VPN client containers, genereally things that result in anti-patterns.

Months ago I came to the conclusion that his forum is headless. I would be surprised, if someone is going to respond.