Files ending up in overlay2 instead of being written to volume

Docker Engine General
1

Here’s what part of my docker-compose.yaml looks like:

  grabber:
    build:
      context: .
      dockerfile: ./Dockerfile_grabber
    container_name: grabber
    image: grabber_image
    restart: always
    environment:
      DAGSTER_POSTGRES_USER: 
      DAGSTER_POSTGRES_PASSWORD: 
      DAGSTER_POSTGRES_DB: 'orchestrator'
      DAGSTER_CURRENT_IMAGE: 'grabber_image'
    volumes:
      - type: volume
        source: bronze
        target: /opt/orchestrator/database/bronze_grabber
        volume:
          nocopy: true
    networks:
      - dagster_network

networks:
  dagster_network:
    driver: bridge
    name: dagster_network

volumes:
  bronze:

Pretty standard. I’ve got a volume called bronze and a target path within the container.

When running the container it seems like I’m writing folders and files to the correct path. I printed out some debugging and my Python code says the path being written to is /opt/orchestrator/database/bronze_grabber.

When I go into /var/lib/docker and do a find I do see the files written. Here’s an example of two files:

./overlay2/e57831b14ff91e18d966f0a56f55f2067f2fc0c63e497554fd262dc73fad44c5/diff/opt/orchestrator/database/bronze_grabber/trading_view/COMEX:GC1!/2025/2025-04-26_COMEX:GC1!.parquet
./overlay2/e57831b14ff91e18d966f0a56f55f2067f2fc0c63e497554fd262dc73fad44c5/diff/opt/orchestrator/database/bronze_grabber/trading_view/AMEX:UVIX/2025/2025-04-26_AMEX:UVIX.parquet

The path should be to /var/lib/docker/volumes/. Within that path there is an empty folder with the volume but no files in it.

I tried doing a bind mount but the folder is empty as well.

Everything I see online says my path is incorrect, but I can’t for the life of me see how it is incorrect. Is it because there are nested folders? Maybe it’s a bug in docker itself?

docker info
Client:
 Version:    26.1.3
 Context:    default
 Debug Mode: false
 Plugins:
  compose: Docker Compose (Docker Inc.)
    Version:  2.27.1+ds1-0ubuntu1~24.04.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 13
  Running: 0
  Paused: 0
  Stopped: 13
 Images: 7
 Server Version: 26.1.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 
 runc version: 
 init version: 
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.0-58-generic
 Operating System: Linux Mint 22.1
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 23.42GiB
 Name: Happy
 ID: 884bf545-63ee-40b3-a0cf-30bf9750fe32
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
2

It looks like you are using a not officially supported distribution and possibly a docker package maintained by the distribution maintainers.

https://docs.docker.com/engine/install/#other-linux-distributions

But for now, let’s assume the issue is general Docker issue.

Check the output of

docker container inspect containername

It will show the mounts and volumes. If it was not mounted to the right folder, you will see that.

You can also check the mounts in the container:

docker compose exec grabber cat /proc/mounts

You can try to filter to the target folder too

docker compose exec grabber cat /proc/mounts | grep orchestrator
3

Thanks for the info!

I’m not sure what I’m looking for. Here’s the output from these commands:

docker inspect grabber_image
[
    {
        "Id": "sha256:d09c32af66df67e9dd289354fa34da9a37584d6c6ea98c5af4e70d519f4924fd",
        "RepoTags": [
            "grabber_image:latest"
        ],
        "RepoDigests": [],
        "Parent": "",
        "Comment": "buildkit.dockerfile.v0",
        "Created": "2025-04-26T12:35:29.868653708-07:00",
        "DockerVersion": "",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "4000/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "LANG=C.UTF-8",
                "GPG_KEY=7169605F62C751356D054A26A821E680E5FA6305",
                "PYTHON_VERSION=3.12.10",
                "PYTHON_SHA256=07ab697474595e06f06647417d3c7fa97ded07afc1a7e4454c5639919b46eaea"
            ],
            "Cmd": [
                "dagster",
                "code-server",
                "start",
                "-h",
                "0.0.0.0",
                "-p",
                "4000",
                "-m",
                "grabber"
            ],
            "ArgsEscaped": true,
            "Image": "",
            "Volumes": null,
            "WorkingDir": "/opt/orchestrator",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "com.docker.compose.project": "orchestrator",
                "com.docker.compose.service": "grabber",
                "com.docker.compose.version": "2.27.1"
            }
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 527253199,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/vqndxlfqasj6ac7s911kegiih/diff:/var/lib/docker/overlay2/m0dyp58sc5hysc6bg2se3rpqh/diff:/var/lib/docker/overlay2/nba3qb3ul0bqb0l01fn2a5ac9/diff:/var/lib/docker/overlay2/9n0i1mc08ixcr9azxst52yjy2/diff:/var/lib/docker/overlay2/waprbdgyihd0ae57ob612qajk/diff:/var/lib/docker/overlay2/21f040f5b71ae44cc57958d947a214a7a1ad6865fd485b25bc942f30388ecab4/diff:/var/lib/docker/overlay2/ef18908141abc26991e784231772c8d69ef493507043a188c3615d13d327820b/diff:/var/lib/docker/overlay2/14d4b4d75737730581864abcf6f140d2997dfecf686362d03bdc708d6234adcb/diff:/var/lib/docker/overlay2/7b72c147fdbdd0de5d9e2875caafe2e6b375b0a1d4116f717918ac3d43326f7a/diff",
                "MergedDir": "/var/lib/docker/overlay2/4hbu4j4nde9ajafs6zzob7q53/merged",
                "UpperDir": "/var/lib/docker/overlay2/4hbu4j4nde9ajafs6zzob7q53/diff",
                "WorkDir": "/var/lib/docker/overlay2/4hbu4j4nde9ajafs6zzob7q53/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:ea680fbff095473bb8a6c867938d6d851e11ef0c177fce983ccc83440172bd72",
                "sha256:67c69e32800ff6bdaf973dd90010ca63bb7858f7077bba0abde1680d234e96e4",
                "sha256:e72ee629d8b7663fa594f85a170619785422794b9e02a496f58756f89e6560d1",
                "sha256:68b89356c898a133e6c335725f8fd6aca40f155c84d951481cb087b5d55658a9",
                "sha256:1f24751a03bb6620a669e5024216c76ca88bd6a389730754a263f4462cfb86f6",
                "sha256:9bce4187e167df167fc191ae86ab4e81a1b1add2787dcd281a375a9e364426b7",
                "sha256:a3104f7c59571566f0354f655c07e77e6e13b6b8d46f44a4f3efbeb06873c941",
                "sha256:b5edfbc6b20b8fd18417c1e0da67de4060a504010b076ca52c2a5df2492ab2d3",
                "sha256:e6cd9150f1e7c5040594b8926dedbba40e71cc545991fa4b527d16ed266f4b63",
                "sha256:bd4a58ac34192acdd2633e1f8af69fa94558e7a57268d3ed1b4756695dcccc93"
            ]
        },
        "Metadata": {
            "LastTagTime": "2025-04-26T12:52:00.579471917-07:00"
        }
    }
]

docker compose exec grabber cat /proc/mounts 
overlay / overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/F4NQNQM2D3HIUNGWUCYCCEPLVQ:/var/lib/docker/overlay2/l/IHPZJOYO3J6Q35XSQUM2BU5UID:/var/lib/docker/overlay2/l/CPRMN6VKATX4KLJSUTJXXREDBD:/var/lib/docker/overlay2/l/YVN7N74QBC7JTBPWUFUM6735GN:/var/lib/docker/overlay2/l/2EA6RW2DJILU3IC4C4UZ3NYKAS:/var/lib/docker/overlay2/l/IDNKIEV2YV2M6GKXEU43JIMW7C:/var/lib/docker/overlay2/l/IBOYVQ2BU4DSF63FSNIUM5S7TH:/var/lib/docker/overlay2/l/KGVP2J5BSDCBCST6JT52RLNTAO:/var/lib/docker/overlay2/l/ZHWGRWA5G2IN4RDXM7DPT7S32V:/var/lib/docker/overlay2/l/V2KZOJUUGKG6DGPJYRETGINFPR:/var/lib/docker/overlay2/l/BTCJ3HWTJLCYERBTF544SQVE6S,upperdir=/var/lib/docker/overlay2/615e1e87b2c8158c3d8d8f6b586dcc821410b88a53cf6286628eb1c61a88c3ec/diff,workdir=/var/lib/docker/overlay2/615e1e87b2c8158c3d8d8f6b586dcc821410b88a53cf6286628eb1c61a88c3ec/work,nouserxattr 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev tmpfs rw,nosuid,size=65536k,mode=755,inode64 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666 0 0
sysfs /sys sysfs ro,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup cgroup2 ro,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0
mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0
shm /dev/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k,inode64 0 0
/dev/sda2 /etc/resolv.conf ext4 rw,relatime,errors=remount-ro 0 0
/dev/sda2 /etc/hostname ext4 rw,relatime,errors=remount-ro 0 0
/dev/sda2 /etc/hosts ext4 rw,relatime,errors=remount-ro 0 0
/dev/sda2 /opt/orchestrator/database/bronze_grabber ext4 rw,relatime,errors=remount-ro 0 0
proc /proc/bus proc ro,nosuid,nodev,noexec,relatime 0 0
proc /proc/fs proc ro,nosuid,nodev,noexec,relatime 0 0
proc /proc/irq proc ro,nosuid,nodev,noexec,relatime 0 0
proc /proc/sys proc ro,nosuid,nodev,noexec,relatime 0 0
proc /proc/sysrq-trigger proc ro,nosuid,nodev,noexec,relatime 0 0
tmpfs /proc/asound tmpfs ro,relatime,inode64 0 0
tmpfs /proc/acpi tmpfs ro,relatime,inode64 0 0
tmpfs /proc/kcore tmpfs rw,nosuid,size=65536k,mode=755,inode64 0 0
tmpfs /proc/keys tmpfs rw,nosuid,size=65536k,mode=755,inode64 0 0
tmpfs /proc/latency_stats tmpfs rw,nosuid,size=65536k,mode=755,inode64 0 0
tmpfs /proc/timer_list tmpfs rw,nosuid,size=65536k,mode=755,inode64 0 0
tmpfs /proc/scsi tmpfs ro,relatime,inode64 0 0
tmpfs /sys/firmware tmpfs ro,relatime,inode64 0 0
tmpfs /sys/devices/virtual/powercap tmpfs ro,relatime,inode64 0 0

docker compose exec grabber cat /proc/mounts | grep orchestrator
/dev/sda2 /opt/orchestrator/database/bronze_grabber ext4 rw,relatime,errors=remount-ro 0 0

It looks like you are using a not officially supported distribution and possibly a docker package maintained by the distribution maintainers.

Maybe it’s a Dagster issue. It doesn’t have a ton of documentation and might be doing some magic under the hood. For example the image line in the docker compose yaml file is required, but it’s making that image. I chose the name grabber_image. It’s not downloading anything from docker hub. The Dagster dockerfile I made. If it’s somehow relevant here it is:

FROM python:3.12-slim

RUN pip install uv
RUN uv pip install \
    dagster \
    dagster-graphql \
    dagster-webserver \
    dagster-postgres \
    dagster-docker \
    --system

# Set $DAGSTER_HOME and copy dagster.yaml and workspace.yaml there
ENV DAGSTER_HOME=/opt/dagster/dagster_home/

RUN mkdir -p $DAGSTER_HOME

COPY dagster.yaml workspace.yaml $DAGSTER_HOME

WORKDIR $DAGSTER_HOME
4

I figured it out. Dagster spins up a new docker containers automatically for grpc server mode.

The docker-compose.yaml file I made for my own docker containers is completely separate from the docker containers Dagster spins up itself. This is why no volume is being created on these other containers, because Dagster is creating them not docker-compose.yaml.

One such solution to test is to go into dagster.yaml and under

run_launcher:
  module: dagster_docker
  class: DockerRunLauncher

try adding a volume there. E.g.:

  config:
    container_kwargs:
      volumes:
        - /tmp/database:/opt/orchestrator/database/

Now all docker containers Dagster spins up share that bind volume. This isn’t ideal because you want just the grpc containers to share the same bind volume, but it’s a start to figuring this out.

1 Like