Huge increase in LXC backup

[gusto1]

I am not a docker expert.
I know it is not recommended, but since debian 11/12 I have been using docker in LXC. It works fine and I do not want to change it.
I manage LXC via the Proxmox web gui.
I have installed LXC Debian 12 with docker compose and there are 4 dockers inside
Portainer, Adguard, RustDesk and DrawIO.
I am backing up this LXC with ZSTD compression.
This backed up/compressed LXC is 840MB in size.

I recently moved to Proxmox 9 (debian 13) and I am also reinstalling all LXC on Debian 13.

I am not solving the functionality, because everything works correctly.
However, my backup has grown significantly compared to the previous one.
LXC debian 12 (830MB)
LXC debian 13 (1.35GB)

I solved this situation with AI and supposedly /var/lib/docker/rootfs should have the value 0B
Is this correct?

535M	/var/lib/docker/rootfs
535M	/var/lib/docker
302K	/var/lib/docker/containers
95K	/var/lib/docker/volumes
24K	/var/lib/docker/buildkit
9,5K	/var/lib/docker/network
1,0K	/var/lib/docker/tmp
1,0K	/var/lib/docker/plugins
512	/var/lib/docker/swarm
512	/var/lib/docker/runtimes

I should also use overlay2 and not overlayfs.
sudo docker info | grep “Storage Driver”
Storage Driver: overlayfs

I have nesting and keyctl enabled in LXC.

[rimelek (Ákos Takács)]

Let’s see if I understood the real issue here, because most of your post was about LXC. Let me knoew if I misunderstood anything.

So as far as I know, “rootfs” is only there under /var/lib/docker when the containerd-snapshooter is enabled. It is then default in the latest Docker versions, but you can disable it if it causes any trouble and you need time to prepare your scripts or backups for a new folder structure. If you want to disable it, the “containerd-snapshotter” feature has to be disabled as described in the dockerd reference documentation: https://docs.docker.com/reference/cli/dockerd/

Changing the snapshotter setting also changes the storage driver, since the containerd-snapshotter means containerd will store the images under the following folder by default:

/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs

So when you see the storage driver is “overlayfs”, that is normal when using the containerd snapshotter. The /var/lib/docker/rootfs/overlayfs/<ID> folder is also created which will be the root filesystem of the container.

The fodler under /var/lib/docker/rootfs/<ID> fodler is actually an overlay filesystem that you can see in the output of mount on the host, but every files in that root filesystem can be found under /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/<SNAPSHOTID>/

Now back to the backup size. The rootfs folder will not be empty under /var/lib/docker when you have running containers. If you have runing containers, the size depends on the size of your images and saved files on the container filesystems.

If you don’t want to backup the rootfs, you can check if the backup software supports a way ignoring overlayfs folders or specific folders by name. OR you can disable the “containerd-snapshotter” as I wrote above, but then you would have the overlay2 folder again unde /var/lib/docker.

Of course, if you choose to use the containerd snapshotter, and you want to backup container images, you will need to backup the snapshots I also mentioned.

Now that I wrote this all down, I guess your real problem is that your backup contains the images twice as it is backed up from containerd and also from the rootfs, which should be ignored if possible since those files are physically in another folder.

[meyay (Metin Y.)]

Can you clarify what exactly you backup? The whole file system of the lxc container? Just a specific path?

According to your output, it can not just be /var/lib/docker, as the folder size has 535M, which is smaller than the backup size of LXC debian 12 and LXC debian 13.

Can you share the output of this command?

 sudo du --max-depth=1 --one-file-system --human-readable /var/lib 

[gusto1]

I never backup docker containers themselves. I backup the entire LXC via the web gui
In the terminal it would be like this
vzdump 115 --mode stop --compress zstd --storage local --notes “my_backup_lxc”

I’ll try to turn off containerd-snapshotter and make a new backup.

sudo du --max-depth=1 --one-file-system --human-readable /var/lib

512	/var/lib/dhcp
512	/var/lib/logrotate
1,0K	/var/lib/postfix
1,0K	/var/lib/vim
1,0K	/var/lib/dbus
38M	/var/lib/apt
512	/var/lib/private
1,0K	/var/lib/wtmpdb
25K	/var/lib/ucf
1,0K	/var/lib/python
186K	/var/lib/systemd
16K	/var/lib/pam
512	/var/lib/dhcpcd
1,0K	/var/lib/man-db
512	/var/lib/git
1,2G	/var/lib/containerd
512	/var/lib/misc
475K	/var/lib/docker
28M	/var/lib/dpkg
1,5K	/var/lib/ifupdown2
1,0K	/var/lib/sudo
1,5K	/var/lib/dictionaries-common
1,3G	/var/lib

[meyay (Metin Y.)]

The du output confirms that it’s due to the switch to containerd-snapshotter.

Is it safe to assume that you enabled the containerd-snapshotter in /etc/docker/daemon.json manually? Afaik an update shouldn’t enable it by itself.

[gusto1]

No I didn’t enable anything. In my case the file /etc/docker/daemon.json doesn’t exist

[meyay (Metin Y.)]

That’s a surprise to me. My mental model aligns with the blog post about Docker Engine v29, which doesn’t seem to align with the behavior we see on your system.

Can you share the output of docker info?

[gusto1]

Client: Docker Engine - Community
 Version:    29.1.5
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.30.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.0.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 6
  Running: 5
  Paused: 0
  Stopped: 1
 Images: 5
 Server Version: 29.1.5
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: dea7da592f5d1d2b7755e3a161be07f43fad8f75
 runc version: v1.3.4-0-gd6d73eb8
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.17.4-2-pve
 Operating System: Debian GNU/Linux 13 (trixie)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 4GiB
 Name: debian-13-docker
 ID: fa22c640-b276-41e8-88fd-0f6221c4156b
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Firewall Backend: iptables

[meyay (Metin Y.)]

Never mind, my assumption was that you upgrade the distro and an existing docker engine version. Now that I read your original post, it looks like you used a fresh Debian 13 lxc image, and made a fresh installation of docker v29.

The behavior you see is expected for a fresh installation.

[gusto1]

Yes, I installed LXC debian 13 and docker today. I’m not saying that anything is wrong, everything is working properly.
However, I was concerned about the size of the backup.
I backup LXC regularly and send it to the cloud (gdrive) using rclone.
Gdrive provides a limited size for free, so I try to keep the backups small.

[rimelek (Ákos Takács)]

Small correction to my original post. When the containerd snapshotter is enabled, containerd stores the images twice basically. Once it stores the images on the overlayfs (like Docker would do with overlay2) and second, it stores the images in a compressed format where every image layer is a targz file and the size of those are the size of what you can see in a registry like Docker Hub. So even if the VM backup size doesn’t increase because of the “rootfs” which is just a mounted filesystem, so it shouldn’t increase the backup size when a whole VM is saved, the size of the compressed images can still increase it.

[gusto1]

So how would I solve my problem?

[rimelek (Ákos Takács)]

I think we already discussed your options. My small correction doesn’t change that, I just didn’t want my answer to be incomplete. If you backup a whole VM, you either disable containerd snapshotter or accept the bigger size.

[gusto1]

According to the AI, it turned out pretty bad.
Procedure

sudo docker info | grep -E "Storage Driver|containerd"

 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Runtimes: io.containerd.runc.v2 runc
 containerd version: dea7da592f5d1d2b7755e3a161be07f43fad8f75

sudo systemctl stop docker
Stopping 'docker.service', but its triggering units are still active:
docker.socket

sudo systemctl stop containerd

sudo nano /etc/docker/daemon.json

{
  "features": {
    "containerd-snapshotter": false
  },
  "storage-driver": "overlay2"
}

sudo rm -rf /var/lib/docker
sudo systemctl start containerd
sudo systemctl start docker
sudo docker info | grep -E "Storage Driver|driver-type"
 Storage Driver: overlay2

sudo du -h --max-depth=1 /var/lib/docker | sort -hr

68K	/var/lib/docker
32K	/var/lib/docker/buildkit
13K	/var/lib/docker/image
5,5K	/var/lib/docker/network
5,0K	/var/lib/docker/volumes
1,0K	/var/lib/docker/plugins
1,0K	/var/lib/docker/overlay2
512	/var/lib/docker/tmp
512	/var/lib/docker/swarm
512	/var/lib/docker/runtimes
512	/var/lib/docker/containers

I rebuilt my 4 dockers

sudo du -h --max-depth=1 /var/lib/docker | sort -hr
1,3G	/var/lib/docker/overlay2
1,3G	/var/lib/docker
2,0M	/var/lib/docker/image
282K	/var/lib/docker/containers
75K	/var/lib/docker/volumes
32K	/var/lib/docker/buildkit
14K	/var/lib/docker/network
1,0K	/var/lib/docker/tmp
1,0K	/var/lib/docker/plugins
512	/var/lib/docker/swarm
512	/var/lib/docker/runtimes

My LXC backup is 1.86GB hmmm

[meyay (Metin Y.)]

@meyay

--one-file-system

Didn’t you forget to specify this argument? Without it, du is not suitable to show the real size.,