VolumeDriver.Mount: docker-legacy: Mount: volume : failed: Error while waiting for storage action to finish

General Discussions
, ,
1

Hi,

I’m having issues regarding volumes and its use with docker swarm. I have created two volumes, which are hosted in Digital Ocean Block storage and are used via rexray. The volumes show fine:

$docker volume ls
DRIVER VOLUME NAME
rexray/dobs:latest grafana
rexray/dobs:latest loki

I want to spin up grafana:

version: "3.5"
networks:
  loki:
    external: false
  traefik-public:
    external: true
  grafana-influx:
    external: true
volumes:
  grafana:
    external: true
  loki:
    external: true
services:
  grafana8:
    image: grafana/grafana:9.5.1
    user: "0"
    container_name: grafana8
    volumes:
      - grafana:/etc/grafana
      - grafana:/var/lib/grafana
      - grafana:/var/log/grafana
    ports:
      - "3000:3000"
    restart: always
    networks:
      - loki
      - traefik-public
    deploy:
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.grafana.rule=Host(`grafana2.domain.net`)"
        - "traefik.http.routers.grafana.entrypoints=websecure"
        - "traefik.http.routers.grafana.tls=true"
        - "traefik.http.routers.grafana.tls.certresolver=lets-encrypt"
        - "traefik.http.services.grafana.loadbalancer.server.port=3000"
        - "traefik.docker.network=traefik-public"
  loki:
      image: grafana/loki:2.6.1
      user: "0"
      ports:
        - "3100:3100"
      command: -config.file=/etc/loki/local-config.yaml
      volumes:
      - loki:/etc/loki
      - loki:/loki
      restart: unless-stopped
      networks:
        - loki
        - traefik-public
      deploy:
        labels:
          - "traefik.enable=true"
          - "traefik.http.routers.loki.rule=Host(`loki2.domain.net`)"
          - "traefik.http.routers.loki.entrypoints=websecure"
          - "traefik.http.routers.loki.tls=true"
          - "traefik.http.routers.loki.tls.certresolver=lets-encrypt"
          - "traefik.http.services.loki.loadbalancer.server.port=3100"
          - "traefik.docker.network=traefik-public"

I have tried both compose up and docker stack up
With docker-compose up:

Executing task: docker compose -f "PLG/docker-compose.yml" up -d --build 

[+] Running 0/2
 ⠹ Container plg-loki-1  Creating                                                                                                                                     26.2s 
 ⠹ Container grafana8    Creating                                                                                                                                     26.2s 
Error response from daemon: VolumeDriver.Mount: docker-legacy: Mount: loki: failed: Error while waiting for storage action to finish

And with docker stack commands the result is similar.

What am I doing wrong?

2

Can you share the output of these commands:

docker volume inspect grafana
docker volume inspect loki

It looks like the Volume Driver Plugin doesn’t work properly. Doesn’t rexray create some sort of logs you can check? Also looking at the docker logs usually provides insights: sudo journalctl -xu docker.service

3

Hi @meyay

I supose the volume has to be mounted?:

[
    {
        "CreatedAt": "0001-01-01T00:00:00Z",
        "Driver": "rexray/dobs:latest",
        "Labels": null,
        "Mountpoint": "",
        "Name": "grafana",
        "Options": {
            "size": "1"
        },
        "Scope": "global",
        "Status": {
            "availabilityZone": "fra1",
            "fields": null,
            "iops": 0,
            "name": "grafana",
            "server": "dobs",
            "service": "dobs",
            "size": 1,
            "type": ""
        }
    }
]

The logs:

May 11 07:22:13 ubuntu-s-1vcpu-512mb-10gb-fra1-01 dockerd[845]: time=“2023-05-11T07:22:13Z” level=error msg="time="2023-05-11T07:22:13Z" level=error msg="Error while waiting for storage action to finish" host="unix:///var/run/rexray/615785418.sock" inner="GET https://api.digitalocean.com/v2/volumes/75af638b-eb41-11ed-9cb7-0a58ac14d288/actions/1784988059: 404 (request \"2cad82aa-f39b-44ba-ad48-84b8df24e263\") could not get volume action: no volume event found" instanceID="dobs=353500070,name=ubuntu-s-1vcpu-512mb-10gb-fra1-01&region=fra1" route=volumeAttach server=daisy-duke-fr service=dobs storageDriver=dobs task=1 time=1683789733896 tls=false txCR=1683789731 txID=dd156507-4fa4-4ace-6573-1016a82eacf2 volumeID=75af638b-eb41-11ed-9cb7-0a58ac14d288 " plugin=a8a1bd9be72924325ed78c2c9fc39a50b85ae4a2320be306d8b36d620e7d453f
May 11 07:22:13 ubuntu-s-1vcpu-512mb-10gb-fra1-01 dockerd[845]: time=“2023-05-11T07:22:13Z” level=error msg="time="2023-05-11T07:22:13Z" level=error msg="error: api call failed" error.volumeID=75af638b-eb41-11ed-9cb7-0a58ac14d288 host="unix:///var/run/rexray/615785418.sock" route=volumeAttach server=daisy-duke-fr time=1683789733897 tls=false txCR=1683789731 txID=dd156507-4fa4-4ace-6573-1016a82eacf2 " plugin=a8a1bd9be72924325ed78c2c9fc39a50b85ae4a2320be306d8b36d620e7d453f
May 11 07:22:13 ubuntu-s-1vcpu-512mb-10gb-fra1-01 dockerd[845]: time=“2023-05-11T07:22:13Z” level=error msg="time="2023-05-11T07:22:13Z" level=error msg="docker-legacy: Mount: loki: failed: Error while waiting for storage action to finish" host="unix:///var/run/rexray/615785418.sock" integrationDriver=linux osDriver=linux service=dobs storageDriver=libstorage time=1683789733909 " plugin=a8a1bd9be72924325ed78c2c9fc39a50b85ae4a2320be306d8b36d620e7d453f
May 11 07:22:14 ubuntu-s-1vcpu-512mb-10gb-fra1-01 dockerd[845]: time=“2023-05-11T07:22:14.005272906Z” level=error msg=“Handler for POST /v1.42/containers/create returned error: VolumeDriver.Mount: docker-legacy: Mount: loki: failed: Error while waiting for storage action to finish”

4

Actually inspect shows that it wasn’t mounted. The property “Mountpoint” would show the local path to the mountpoint. Though, it is only mounted on a node, if the swarm service is actually scheduled on the node. On a single node swarm it would be obvious that something is wrong. On a multi node swarm the task could have been scheduled to a different node.

Though, the error message looks like there is a general problem between Rexray and the backend. I won’t be able to help, as I have no experience with Rexray.