Docker Volume lifetime

General
1

Hi all…

This is probably general confusion around the docker volume lifecycle.

I’ve defined a simple volume for use by a rabbitmq container for persistence.

volumes:
  rabbitmq-data:
    external: true

According to the documentation, “external: true” means that “docker compose up” will not try to recreate the volume. Great.

I create the volume, start rabbit, and make some changes through the rabbitmq console.

If I simply execute

  1. docker compose stop rabbit
  2. docker compose up -d rabbit

It all works as expected. The same volume is used after restart. If, however, I execute

  1. docker compose stop rabbit
  2. docker system prune
  3. docker compose up -d rabbit

Then oddity happens. “docker system prune” doesn’t delete the volume, but it does remove the old rabbit container. When i restart rabbit, however, now the volume gets recreated, and I’ve lost previous changes. This suggests to me that the only safe way to persist across container restarts and cleanups is to use host mounted directories. Is this correct??

2

This is the correct behavior. As the containers have been stopped, docker system prune deleted them. According the documentation of the command, it doesn’t handle volumes at all:

WARNING! This will remove:
- all stopped containers
- all networks not used by at least one container
- all dangling images
- all build cache

This indeed is odd. With an external declared volume, compose should neither delete nor create a volume. According documentation the existing volume should be (re-)used. Though, maybee you have a special character and need to use volumes.${volume name in compose}.external.name: ${volume name in docker engine} (of course you need to use the yaml syntax instead of the flattend format I used) instead to properly address the volume, see 2nd example in Overview | Docker Docs.

3

Can you show how you bound that volume to your service?
That definition you just shared does not mean you actually use that volume. Rabbitmq has a volume definition in the Dockerfile. If you just stop the container (this is what you did) and start it again, the anonymous volume will be used again. When you delete the container, Docker Compose will create a new anyonymous volume and use that. In the meantime your external is not used at all.

4

Of course! I should have included that earlier:

    sm-queue-service:
       ## snip ##
        volumes:
            - rabbitmq-data:/var/lib/rabbitmq
            - rabbitmq-data:/var/log/rabbitmq

this doesn’t seem an unusual setup, I don’t think.

I’ve run docker compose for some time using host mounted folders in this scenario without issue…it is just when I moved to using volumes that I’ve encountered issues which seem related to performing a docker system prune.

5

I consider unusual enough that you used the same volume to mount to two different destination path in the same container. I didn’t know it was even possible, but strange things could happen. Maybe it is related to your problem, but I am not sure. If somehow the logs folder is cleared because you have a new container with new identifiers, I don’t know… then your data would be removed too since that is also in the log folder. I haven’t tested it though.

6

That’s a fair point…rabbitmq persistant storage seems to implicate those two folders, and I don’t particular want/can’t mount the entire /var/lib directory as a volume. I suppose a continuation for me is the try using two different volumes…

7

I see in the rabbitmq dockerfile that there is only one volume specified:
VOLUME $RABBITMQ_DATA_DIR

Based on this information, I tried the same test with only one mount:

        volumes:
            - rabbitmq-data:/var/lib/rabbitmq

but I experienced the same apparent loss of data. However, I noticed that the creation date on the volume is the same, which suggests that this might be rabbitmq behaviour…perhaps it is using the container id as a key somehow for internal data, and when it sees the id change, it wipes the data.

So, probably not a docker quirk after all.

8

Are you sure that it is realy wipped?

You might want to check the folder where data is actualy stored, which you can find out using this command:
docker volume inspect rabbitmq-data --format '{{ .Mountpoint }}'

Maybee the old data is still sitting there, but is just not picked up by the new instance, as the “hostname” or an instance id is part of the path?

On a second thought: I would strongly recommend to investigate and understand the mechanism, as you would end up beeing in the same situation when you change settings that force docker-compose to re-create the container - which realy is a common thing and usualy happens without any impact exapt downtime of the service…

9

Thank you all…I think the problem was changing hostname on rabbit. I will try to use the volume approach going forward. I appreciate your responses and time.