Docker Community Forums

Share and learn in the Docker community.

Mounted volume / wrong df value reported

Hello,

I am currently fighting a weird issue that I can’t seem to resolve : I am trying to mount a volume on / to allow some monitoring, but even though I am using the same version of docker on both, one server reports it correctly, but not the other one

docker-compose.yml (Exactly the same on both servers)

version: '3.4'

networks:
  monitor-net:
    driver: bridge

services:

  nodeexporter-text:
    image: prom/node-exporter:v1.1.2
    #image: quay.io/prometheus/node-exporter:latest
    container_name: nodeexporter-test
    volumes:
            - /proc:/host/proc:ro
            - /sys:/host/sys:ro
            - /:/rootfs:ro
    command:
            - '--path.rootfs=/rootfs'
            - '--path.procfs=/host/proc'
            - '--path.sysfs=/host/sys'
            #- '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
            - '--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)'
            #- "--collector.filesystem.ignored-mount-points"
            #- "^/(sys|proc|dev|host|etc|rootfs/var/lib/docker/containers|rootfs/var/lib/docker/overlay2|rootfs/run/docker/netns|rootfs/var/lib/docker/aufs)($$|/)"
    pid: host
    restart: unless-stopped
    expose:
      - 9100
    networks:
      - monitor-net
    labels:
      org.label-schema.group: "monitoring"
    healthcheck:
      test: ["CMD", "wget", "--tries=1", "--spider", "http://localhost:9100/metrics"]
      interval: 10s
      timeout: 5s

… started exactly the same way on both servers :

$ sudo docker-compose up -d
Starting nodeexporter-test ... done

Server “Jump” : /rootfs seen empty

$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/md3        938G   31G  860G   4% /

$ sudo docker exec -it nodeexporter-test sh -c 'df -h /rootfs'
Filesystem                Size      Used Available Use% Mounted on
/dev/loop1               55.5M     55.5M         0 100% /rootfs
$ sudo docker --version
Docker version 20.10.7, build f0df350

$ sudo docker-compose --version
docker-compose version 1.29.2, build 5becea4c

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.5 LTS
Release:	18.04
Codename:	bionic

Server “API” /rootfs seen correctly

$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/md2        848G  496G  309G  62% /

$ sudo docker exec -it nodeexporter-test sh -c 'df -h /rootfs'
Filesystem                Size      Used Available Use% Mounted on
/dev/md2                847.1G    495.0G    308.9G  62% /rootfs
$ sudo docker --version
Docker version 20.10.7, build f0df350

$ sudo docker-compose --version
docker-compose version 1.29.0, build 07737305

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.5 LTS
Release:	18.04
Codename:	bionic

Any suggestion would be greatly appreciated :slight_smile:

Thanks in advance for your help

PS : I wasn’t sure where to post that in this forum, feel free to indicate a more appropriate section if needed !

1 Like

I guess not, but just in case:

While the Docker versions are the same, the Docker Compose versions are slightly different. Could this somehow affect how the volumes are created? Maybe docker volume inspect can help?

Hello, I tried on another server (same exact docker-compose.yml than copied in the first message), whith exactly the same version of docker-compose, and as you see this time again df -h properly works again.

Dev server (ok : partition size is seen correctly)

co@dev-api:/tmp/mvc$ sudo docker exec -it nodeexporter-test sh -c 'df -h /rootfs'
Filesystem                Size      Used Available Use% Mounted on
/dev/md2                847.1G    692.3G    111.7G  86% /rootfs

co@dev-api:/tmp/mvc$ sudo docker --version
Docker version 20.10.7, build f0df350

corentin@dev-api:/tmp/mvc$ sudo docker-compose --version
docker-compose version 1.29.2, build 5becea4c

co@dev-api:/tmp/mvc$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.5 LTS
Release:	18.04
Codename:	bionic

Monitoring server (KO :red_circle: partition size not seen corrrectly)

co@jump:/tmp/mvc$ sudo docker-compose up -d
Starting nodeexporter-test ... done

co@jump:/tmp/mvc$ sudo docker exec -it nodeexporter-test sh -c 'df -h /rootfs'
Filesystem                Size      Used Available Use% Mounted on
/dev/loop1               55.5M     55.5M         0 100% /rootfs

co@jump:/tmp/mvc$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/md3        938G   34G  856G   4% /

co@jump:/tmp/mvc$ sudo docker --version
Docker version 20.10.7, build f0df350

co@jump:/tmp/mvc$ sudo docker-compose --version
docker-compose version 1.29.2, build 5becea4c

co@jump:/tmp/mvc$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.5 LTS
Release:	18.04
Codename:	bionic

I don’t understand where that 55.5M size for the docker volume comes from…

I guess you noticed the differences for the docker exec command, for okay mounts:

…versus the troublesome ones:

I don’t know what that means but it may help searching?

Also, did docker volume inspect (docs) reveal anything? And do the troublesome mounts show files like expected?