Stale data being read from anonymous directory volume for postgres container

Here’s my docker-compose.yml:

version: "3.3"
services:
    webserver:
        build: 
            context: ./app_react
            args:
                - APP_HOST_URL=$APP_HOST_URL
                - APP_BACKEND_PORT=$APP_BACKEND_PORT
        ports:
            - 80:80
            - ${APP_BACKEND_PORT}:8080
    nodeapp:
        build: ./app_node
        environment:
            - APP_HOST_URL=$APP_HOST_URL
            - POSTGRES_USER=$POSTGRES_USER
            - POSTGRES_PASSWORD=$POSTGRES_PASSWORD
    db:
        image: postgres:14
        environment:
            - POSTGRES_DB=$POSTGRES_DB
            - POSTGRES_USER=$POSTGRES_USER
            - POSTGRES_PASSWORD=$POSTGRES_PASSWORD
        volumes:
            - pg_data:/var/lib/postgresql
            - ./sql:/docker-entrypoint-initdb.d/
        ports:
            - 5432:5432
    nginx:
        image: nginx:1.21.3
volumes:
    pg_data:

There is an anonymous volume which simply maps the ./sql folder of my project to /docker-entrypoint-initdb.d/. I have one script inside it ./sql/init.sql.

I have made edits to ./sql/init.sql yet for some reason, the postgres container is still reading a stale version of the file that no longer exists in my project folder (I have no idea how or why).

I’ve tried pruning/rebuilding my images, pruning my containers, pruning my volumes… I don’t understand why a stale copy of the volume is being read/mounted into the container.

These are the commands I’m using to rebuild and bring the containers back up:

docker-compose -f ./docker-compose.dev.yml build
docker-compose -f ./docker-compose.dev.yml up -V

I thought the -V option would do the trick, but it isn’t.

Minor aside: you are using a “bind mount”, not an “anonymous volume”. But: that is good.

Are you sure the container is not simply skipping the contents of /docker-entrypoint-initdb.d/ because there is already data in the pg_data volume? The postgres image will only use the init scripts if its data folder is empty:

Warning: scripts in /docker-entrypoint-initdb.d are only run if you start the container with a data directory that is empty; any pre-existing database will be left untouched on container startup. One common problem is that if one of your /docker-entrypoint-initdb.d scripts fails (which will cause the entrypoint script to exit) and your orchestrator restarts the container with the already initialized data directory, it will not continue on with your scripts.

And:

If there is no database when postgres starts in a container, then postgres will create the default database for you.

I did try deleting the pg_data volume, several times. These were the commands I was using (from a convenience script I was using):

docker-compose -f docker-compose.dev.yml down --volumes
docker volume rm -f node-react-docker-skeleton_pg_data

docker-compose -f ./docker-compose.dev.yml build
docker-compose -f ./docker-compose.dev.yml up --renew-anon-volumes

For some reason I don’t fully understand, the container was reading a stale copy of ./sql/init.sql. I know that is the case because the postgres container kept complaining about syntax errors which I had long fixed already on the host copy of the file.

I noticed from docker volumes ls that there was a no-name volume in there that just would not delete, so I ran docker system prune -a --volumes which eventually cleared out all of the entries in docker volumes ls.

After I ran the above 4 commands again, the container then had an up to date copy of ./sql/init.sql from the host.

I wonder: are bind mounts implemented via anonymous volume or something?

Once I was able to clear this unnamed volume from my volumes list, it worked.

I certainly was deleting the pg_data volume over and over again (as you can see from the 4 commands in the convenience script I had written), and it would run the ./sql/init.sql script repeatedly, but it kept telling me there are syntax errors, when I had already long fixed it on the host machine.

I don’t know exactly why, but docker system prune -a --volumes did the trick.

I don’t think so. Volumes are fully managed by Docker, bind mounts are “just” mounts from the host into the container. But on Windows (WSL) and Mac (some VM wrapper) bind mounts may need some extra work? Maybe that can be messed up?

Next time it happens, maybe docker exec -it into the running container to see what the contents of /docker-entrypoint-initdb.d/init.sql are. (Although the error messages that PostgreSQL gave you already show what it was seeing. :thinking:)

Or maybe they are, on Windows and Mac:

bind mounts are remoted to macOS or Windows, where the file systems behave slightly differently

No idea how that “remoted to” is implemented.

Are you seeing the anonymous volume re-appearing while using the bind mount?

Sorry, I have acidentally managed to answer two different questions (different topics) in one answer so I deleted the previous answer to try again

On Windows with WSL2 it uses the 9p protocol so you can access everything from the WSL machine. I don’t remember what happened with Hyper-V. On MacOS I can see two ways. A legacy way can be used by unchecking “use gRPC FUSE for file sharing”. I could not start Docker Desktop without gRPC FUSE so I could not test it.

On Windows:

docker run --rm -it -v /:/data bash

monts the the WSL filesystem into the container

docker run --rm -it -v C:/:/data bash

mounts the “C” drive from the Windows host through WSL.

This is similar on MacOS except it does not have drive letters so my guess is it mounts the folder from the host if that folder exists and you have mounted that folder or one of the parents to the virtual machine (settings / Resources / File sharing).

Acidentally mounting files from from the virtual machine could cause a similar issue but in that case init.sql would not have bee there at all. So strange but I don’t see any volume created when I bind mount something.

1 Like