Docker data occupied 102GB when transfer to external

General
1

  • I am using a mini PC based on ARMv8 CPU and Ubuntu 18.04.5. Because my Mini PC have only 16GB storage in totally, and there is 9GB free after reinstall and install docker software. It is too little to do my project. I mount a 128GB micro SD card. So I want to move all docker data to external SD card.

  • I tried moving docker data as followed these 2 tutorials: link1 and link2. All the steps in the tutorials on the internet are the same in moving docker data in /var/lib/docker/ to other directory.

sudo rsync -aqxP /var/lib/docker/ /new/path/docker
  • When I check disk usage of /var/lib/docker/, it used only 1.5GB, but after move docker data by sudo rsync, or even sudo cp -r, in new directory I used to save docker data, there is 102GB data saved. I don’t have any idea about this issue
Airlink:~ ➜ neofetch
            .-/+oossssoo+/-.               admin@airlink
        `:+ssssssssssssssssss+:`           -------------
      -+ssssssssssssssssssyyssss+-         OS: Ubuntu 18.04.5 LTS aarch64
    .ossssssssssssssssssdMMMNysssso.       Host: FriendlyElec SOM-RK3399
   /ssssssssssshdmmNNmmyNMMMMhssssss/      Kernel: 4.4.179
  +ssssssssshmydMMMMMMMNddddyssssssss+     Uptime: 15 days, 14 hours, 40 mins
 /sssssssshNMMMyhhyyyyhmNMMMNhssssssss/    Packages: 1286
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Shell: bash 4.4.20
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   Terminal: /dev/pts/0
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   CPU: (6) @ 1.416GHz
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   Memory: 515MiB / 3845MiB
+sssshhhyNMMNyssssssssssssyNMMMysssssss+
.ssssssssdMMMNhsssssssssshNMMMdssssssss.
 /sssssssshNMMMyhhyyyyhdNMMMNhssssssss/
  +sssssssssdmydMMMMMMMMddddyssssssss+
   /ssssssssssshdmNNNNmyNMMMMhssssss/
    .ossssssssssssssssssdMMMNysssso.
      -+sssssssssssssssssyyyssss+-
        `:+ssssssssssssssssss+:`
            .-/+oossssoo+/-.
2

@rimelek do you have any thoughts on what might be the issue here?

3

What is the filesystem on that SD card? You should format it to use the same filesystem as you had on the original disk. If your SD cards filesystem has a much larger block size and you have many files smaller than that, the data can require more space. The difference looks too big for me, so it may not be the real issue, but it is something that you must check.

If the filesystem is right, you can compare the two directories using diff before you start the docker daemon. Otherwise the data could change. For example install and use ncdu to brows the filesystem. Open one instance in one terminal and another in an other terminal. You will instantly see if the size of one file/directory is different in the new data directory. so you can follow the differences and find what are the actually different files. You could also use the diff command but I couldn’t use it correctly on the docker data dir.

sudo ncdu -x /var/lib/docker
# and in an other terminal
sudo ncdu -x /new/path/docker

I found that every file with “db” extension got bigger. I used stat on the two files and realized the block size is different for the new db file. It looks like rsync does something I could not reproduce or find what parameter is responsible for this. There is a --block-size parameter. I have never used this so I am not sure when it is used.

At the end I tried to use cp instead of rsync and I got the right result:

cp -ax /var/lib/docker /new/path/docker

This way the size of the two folders were the same.

rsync is better when you want to copy only changed files or copy between different machines

PS.: before you copy the files using cp delete everything from the sdcard to start it clean.

1 Like
4

@rimelek thank you so much for your detailed response. we shall try your suggestions and let you know how it goes either way. cheers!