Named volumes overrides non-empty directory in container

I have this docker-compose to create a new named volume and mount it to a directory in the container.
After container initialization, the content of the container directory gets deleted and I don’t understand why.

services:
  tomcat:
	..
    volumes:
     - tomcat:/usr/local/tomcat/webapps/lavagna

volumes:
  tomcat:

The documentation says:

If you bind-mount into a non-empty directory on the container, the directory’s existing contents are obscured by the bind mount.

But that’s for bind-mounts, not a named volume.

Also this stackoverflow says:

If the host volume/mount exists and contains files it will “override” whatever is in the container.

but the host volume DOES NOT contain files…

GitHub Gist link to docker inspect

Is it really deleted? That is, if you remove the volume definition form the compose file and go into the container, is /usr/local/tomcat/webapps/lavagna still empty?

Other documentation says:

If you mount a bind mount or non-empty volume into a directory in the container in which some files or directories exist, these files or directories are obscured by the mount, just as if you saved files into /mnt on a Linux host and then mounted a USB drive into /mnt. The contents of /mnt would be obscured by the contents of the USB drive until the USB drive were unmounted. The obscured files are not removed or altered, but are not accessible while the bind mount or volume is mounted.

So, your quote does not only apply to bind mounts.

Aside, the same documentation says:

If you mount an empty volume into a directory in the container in which files or directories exist, these files or directories are propagated (copied) into the volume. Similarly, if you start a container and specify a volume which does not already exist, an empty volume is created for you. This is a good way to pre-populate data that another container needs.

Maybe that’s what you want, but then ensure that the volume is really, really empty.

Yes, it’s really deleted.
If the volume is removed from the docker compose definition, this directory has the content of the WAR because Tomcat explodes the war.

If you mount a bind mount or non-empty volume

Right, but this is an empty volume, the docker compose creates it, it has nothing in it.

The volume on the host is /var/lib/docker/volumes/tomcat/_data which is empty,…

That by itself could still be the exploded WAR from an earlier run? Maybe some timestamps on subdirectories can tell when it was exploded by Tomcat, so if the contents really get deleted or not. It may not matter for your problem, but I guess you understand that “deleted” and “obscured” are different things.

Any chance you can run without starting Tomcat? Maybe Tomcat is to blame if things are indeed deleted.

Tomcat is 100 not to blame.

That by itself could still be the exploded WAR from an earlier run?

Also I get what you’re saying but I’m testing “scorched earth” style.

Meaning - I’m running docker system prune on each testing, removing all images, volumes and containers, so no leftovers:

docker rm --force $(docker ps -a -q | xargs) ; docker compose down ; docker system prune --volumes --all --force

but I guess you understand that “deleted” and “obscured” are different things.

I do but what I don’t know is how to test whether it’s deleted or obscured

Assuming the image does not have any exploded WAR file in the lavagna folder. Would Tomcat still explode it if it finds the (empty) subfolder lavagna in /usr/local/tomcat/webapps/, after the image was used to start a new container?

What if you use:

volumes:
 - tomcat:/usr/local/tomcat/webapps

I like your thinking, I found a proof that Docker obscures/deletes the container folder…

with volume:

drwxr-xr-x 2 root root     4096 Jun 29 17:50 lavagna
-rw-r--r-- 1 root root 33368197 Jun 29 17:34 lavagna.war

without volume:

drwxr-x--- 14 root root     4096 Jun 29 18:56 lavagna
-rw-r--r--  1 root root 33368197 Jun 29 18:55 lavagna.war

See? Tomcat always explodes the WAR almost immediately after lavagna.war is created

but with a volume - it re-creates the directory when docker compose finishes - I guess that’s when the volume gets created.
A 16 minutes difference it seems…seems like a lot of time though

Ai, I’d guess many would run into problems if the volume gets created after the container’s ENTRYPOINT was started. I cannot explain the time difference (unless clocks are somehow out of sync), but I’d not expect Docker to first run the ENTRYPOINT and only then create the volume.

Created? Isn’t lavagna.war somewhere in the image? Did you re-create the image before starting the container? (That may explain the 17 minutes?)

Also, does the subfolder /usr/local/tomcat/webapps exist in the image? And if yes: does it also have a subfolder lavagna, and what is in there, in the image?

Also, how did you create the file listing for “with volume”? Maybe you should not peek into the Docker-managed /var/lib/docker/volumes/tomcat/_data as maybe it cannot be trusted to show what the container sees?

And most importantly: anything in that lavagna folder when looking in the container?

I’d try to use tomcat:/usr/local/tomcat/webapps to ensure that Tomcat is not confused by seeing the lavagna folder caused by the mount point.

Just to make sure you understand what it does:

Using the original /usr/local/tomcat/webapps/lavagna will make Docker obscure lavagna but not its parent folder. Or, if lavagna is not in the image then it will create an empty folder.

Mounting it one directory higher will obscure webapps. If the lavagna.war is in that folder in the image, then surely my proposal will make things worse as then Tomcat no longer sees the source it needs to expand… In that case you may need to tell Tomcat to use another folder for the exploded webapps and mount the volume to that other folder. (Explained in a next response.)

Bad choice of words. the WAR is part of the image indeed.

Yes, I’m using this image from Docker Hub, it installs Tomcat which creates the webapps folder:
tomcat:8.5.81-jre8-temurin-jammy

it 100% does, I’m using it reliably for other volumes, you can peek into named volumes data if you have root access on the Docker host…

Ah, wrong. As then the initial copying from the container into the empty volume should kick in.

So, assuming /usr/local/tomcat/webapps/lavagna.war is in the image, then mounting your volume into /usr/local/tomcat/webapps/ should make Docker copy all from that container’s folder into that new empty volume. Next, it will mount the volume into that folder and obscure the original folder. And finally, when running ENTRYPOINT, Tomcat should expand lavagna.war into a newly created subfolder lavagna of that mounted volume (assuming that was not already in the image). Profit!

If that works, then I’d say that Tomcat is not expanding lavagna.war if it already sees a subfolder lavagna, regardless if that is empty…

With docker-ce from docker’s repos, the behavior @avbentem described is the way it is.

I must admit I don’t understand in which usecase it would be useful to use a named volume ifor the most relevant part of the deployment (I assume only the layer with the .war file iside changes when a new image is build) in your scenario.

I can understand that people use binds during developments to replace the .war, but I don’t realy understand why somone would use a named volume during development.

I would highly recomend to not use named volumes for that folder for stages that follow development - there is no benefit, but instead there is the risk that an updated war from a new image will be eclipsed by what already exists inside the volume. The argument that the volume gets removed before each run, is a good indicator that it shouldn’t be a volume at all, don’t you agree?

1 Like