Volume migration in case of full disk

Hi, I need help and guidance on how to solve this problem.

I have an application running in docker with a very big volume size (2TB).

I need to migrate it to a different host machine because this one run out of space.

The recommended way of migration is to first compress (tar) the volume and then send it to a new machine but I can’t do that as I don’t have enough free space on device to fit the compressed copy of that big volume.

What should I do in this case? can I just stop docker and rsync the whole volume to another machine?

You can access the volumes contents in many ways aside from compressing them to a tar

If you’re running on Linux, the volumes contents should be stored in /var/lib/docker/volumes

If you’re running on Mac or Windows (Or using Docker Desktop on Linux), then the volumes are stored on that path within the Linux VM that Docker Desktop spins up
You can browse that vm rather simply using:

docker run --rm -it --privileged --pid host alpine nsenter --target 1 --mount --uts --ipc --net sh

Though I’m not sure how simple extraction of the files would be then

Still, you could do something like this:

docker create \
  -v volume_name:/volumes/volume_name \
  -v another_vol:/volumes/another_vol \
  alpine

This would create a container with those volumes mounted, from which you can copy files outside using docker cp

If you can connect to your destination from your containers, you can copy directly from it without ever taking up extra space

Using docker cp to copy the volumes “outside” the container will not help me as outside there’s no space left.

If I understand your suggestion correctly I could probably do docker exec -it container bash and then rsync directly from there to a separate machine over the internet - correct?

Why is it considered better to rsync files from inside the volume than just rsyncing the files located in /var/lib/docker/volumes/ themselves ?

No difference, if you’re running on Linux then it is as easy as accessing /var/lib/docker/volumes, it is just not as simple for Mac/Windows, so I elaborated

You could mount a remote folder locally on host and use docker cp.

1 Like

@tanatos: at it’s core this is a pure Linux problem, which you just happen to have in a Docker related context. The core problem though, is not with Docker itself, but instead with the capacity limit of your machine. How would you proceed to solve the main problem if Docker wasn’t in the picture?

I’m not sure it would work with 2TB as I never tried with such a large volume, but you can try:

The docker-volume-copy-remote script sends the archive to the standard output and the standard output then is sent to a remote server through SSH and created as a new volume there.

But before running it, stop the containers that use the volume.

2 Likes

@meyay No, my question is not about Linux. I’m an advanced Linux user.
My question was specifically about Docker. Why does every docker documentation mentions complicated steps to move the volumes instead of just simply moving the /var/lib/docker/volumes/ files?
This gives users the reason to believe that the way those Volume files are stored is somehow connected to the local instance of the containers and that simply moving them will not work properly.
Instead there are much more complicated steps described about either generating the compressed version of volumes using special commands or by copying the files from inside the docker filesystem itself rather than just moving the files in the root system.
Thanks to @deanayalon 's answer I now know that the simplest approach should in fact work without a problem, but this is something that’s not documented and should be added to Docker documentation as the current state suggests such solution is discouraged.

Because everything under the docker data root (/var/lib/docker by default) is managed by Docker and you can easily break something. And because the documentation probably wants to share a way that works in every case. You could have remote volumes without being able to access the remote server over SSH connections. You could have any other special storage driver. The only thing that will be the same is the files inside the container. You could also use Docker Desktop when you have no access to the volumes folders directly as @deanayalon pointed out.

Sometimes you have access to the docker command but you have no (direct) access to the root filesystem and the docker data root is owned by root, but everything you do with the docker commands would work if you can run docker commands and assuming there is no special protection implemented for the docker API calls.

So doing anything in the docker data root is generally not recommended, but can be done in special cases. If you feel the documentation should mention the volumes folder in the docker data root, you can go to the documentation page where you think it should be mentioned and in the top right corner there should be a “Request changes” link. You can click on it and ask for a change.

1 Like

Rimelek’s solution works great for remote copy of volumes. I recommend it to others who have similar problem!