Why do Docker volumes include an underscore data (_data) folder?

Possibly a noob question here but… When I create a volume it includes a “_data” folder, which is where data goes if I pass the volume directly into a container. If I create a folder inside the volume and pass that folder into the container, then the data lives there, outside the _data folder. What is the intended purpose of the _data folder in the volume if it’s just an extra storage layer for 100% of the volume contents to go in?

More importantly, if I want to create subfolders in the volume (and pass those individually to the container) it appears to work the same if I place them into the _data folder versus in the root of the volume, so is there some reason it’s better to stay in the _data folder versus a minimal bit of decluttering my Docker run command?

Edit: Just in case this doesn’t apply to all versions of Docker… I am working in Docker in the Photon OS 3.0 VM by VMWare.

This not a general Docker Volume behavior.

Not every Volume Plugin places it’s data in /var/lib/docker/volumes/{volumename}. The local (=default) driver does. If the volume is a remote share (cifs/nfs) it will include the file opts.json containing metadata about the required mount. Data is always written in the _data folder; in case of a remote share, it is previously mounted into the folder. the _data folder is what actualy is mounted into the container using mount-bind.

Block device oriented volumes might store their metadata and content in other locations, e.g. StorageOS uses a etcd cluster to store the metadata and /var/lib/storageos to store its data.

That is the what; I was curious about the why. I created the volume with the “docker volume create” command, not linking to SMB shares or anything fancy.

I was tweaking with a MySQL container and documenting my best approach for future use. I created one volume “mysql” and added two directories “data” and “logs” to that volume. (My use case: I map the mysql data storage and logs to a volume for persistence and remote access to the logs, and I see no good reason to use separate volumes to do this) In case there’s some important functionality involved with the _data folder, I added my subdirectories to the _data folder, but it works just fine if I add them in the volume.

I did have to chown the folders to 999 so the MySQL container could write to them; I suspect I wouldn’t have needed to do this if I’d used two separate volumes.

The why question is best to be answered by the architect of the the local volume plugin AND/OR the architect beeing responsible for the design that allows to map a volume to a container path, but not a volume:/subfolder into a container folder.

The way you intend to use volumes does break the single responsibility principle, doesn’t it? What happes if your volume is backed by a device that is flooded by the database’s logs, preventing the database’s functional data to be persisted?

I favor logging to STDOUT and using a log collector to send the logs to an aggregator or elastic search…

That’s a fair question. The answer to that is perhaps debatable based on one’s interpretation of the principle. Literally it specifies every module, class or function, and if I’m to consider my MySQL container to be the module, then one volume for the MySQL container is compliant with the principle. I also see how one could arguably say that log writing and data storage are the modules and should live in two separate storage units, and indeed I agree with this because I’m storing each into a separate subdirectory. On the other side of the equation, however, when I go to back up the volume, I’d rather script against the single “mysql” volume rather than doing two backups for “mysql_data” and “mysql_logs”.

Also a very good point. This is actually why in general I’ve moved over to adding virtual disks to my VM and mounting them, then passing those mounts instead of volumes if I’m at all worried about filling a drive. Virtual disks are easy to expand if/as needed. As it stands, just creating two separate volumes won’t prevent me from filling a partition because the volumes all live on the same root partition. Adding mounted virtual disks isn’t technically a “volume” per the “docker volume” functionality, and there is no _data directory in that case. My answer is a cost/benefit analysis: when I determine that there is risk of data loss due to logs filling my disk, I’ll probably dump the volumes in favor of mounted VMDKs.

This is all but gibberish to me, but sounds like it’s probably worth investigating as a better approach to log management, so I’ll be Googling this for sure. Thanks for the info!

I assume you refer to -v /host/path:/container/path? If so, this is a bind mount. It will be mounted on top of the folder (like what happens if your mount a remote share into a folder) and it’s pre-existing content becomes invisible for the container. Basicly it is “mount --bind /source/folder /target/folder”.

Depending on your virtualization plattform, there might be a volume plugin that allows to actualy register block devices directly on the virtualization plattform’s storage plane.

Regarding logging: you might want to take a look at the ELK-Stack or GrayLog.

Host is perhaps confusing there for me; I generally refer to the ESXi host as “host”. I use something like “-v /mnt/data:/var/www/html/data” where /mnt/data is a mounted share attached to /dev/sdh1, which is a virtual disk attached to the VM. I am aware many people like to tout container technology as a replacement for virtual machines, but I find a hybrid approach allows for a more scalable and manageable system.

I have in my ESXi host an SSD where all my VMs boot from, but the bulk of my data lives on a handful of RAID1 arrays of spinning disks, so my data VMDKs go into the larger space while my VMs boot from the faster space. This approach works very well; I have a Nextcloud VM with 3TB of storage VMDKs, (and my SSD is nowhere near that big).

You could be right that there might be a volume plugin to register block devices directly (I’m using ESXi 6.0 and 6.7 in different datacenters), but I don’t know if I’m convinced I’ll see any improvements over what I’ve got now? Part of the value of a virtual machine is that it packages an operating system (and all the containers within that allocation) similar to how a container packages an application. I’d hate to add a volume driver to the mix that makes it so I can’t drop the VM onto another physical host.

Will do. Thanks again!

Regarding the volume plugin: no problem. It took me some time to get there as well and frankly some volume drivers actualy behave terribly. Though having a dedicated, managed “blockdevice” per volume has its charme. Every container can have its dedicated volumes, which will only able to fill up the dedicated amount of storage. The risk of bringing down the whole Docker platform or other containers is reduced.

Mybee I am a little bit spoiled from working with AWS and the EBS Volumes, which are attached to whichever (Swarm/K8s) cluster node a container is running. Actualy this is more portable then adding blockdevices to each and every node and more reliable then using nfs/cifs shares

If such a volume plugin does not exist, I would fall back to a cloud native storage like StorageOS or Portworx, which leverages the storage of each node to build a cluster of storages. Both present blockdevices to the volumes, replicate them amongst it’s nodes and bring the volume to the node where the container is running. You are close to local block device speed and still get replication for free… isn’t that something? :slight_smile: Both provide a developer license which is sufficient fo most cases.

If this is not an option, I would opt for a storage cluster like ceph or glusterfs… and then there is the populare fallback NFS v4 (v3 will eventualy make you unhappy with Docker Volumes).

I only use binds if the container runs clustered software, where each replica/member has it’s own state and replication is done within the application itself, e.g. etcd3, consul.

It is all a mater of what you need. Eventualy you will experience if your current approach has unacceptable limitations and switch to an appropriate solution.

Then why _data?
Can I just remove _data and put all in the above folder?,
@cincitech said it works the same.