Good Morning everyone,
first of all I want to apologise to give you such an amount of Log files, but I’m totaly new in docker and don’t know how to start.
My current setup is running Proxmox VE and on an unprivileged LXC Container running my DNS Server with docker compose. It noe happened several times that in the middle of the night I see that the docker daemon is not running anymore. So I gathered all related journalctl files and found them:
Now I’m sitting here and try to figure out what is going on. A simple systemctl start docker did start the engine again, but this can’t be the solution to notice in my morning my DNS Server is not available and I have to ssh to my machine.
Can anybode give me some insights hwo I’m able to debug it even more?
I hope someone using lxc containers will come around and respond to your question.
In the meantime, please provide details about the lxc image you use as base for your lxc container and provide the output of docker version. Did you install docker from the official Docker repositories? Or did you install docker the os distro repos or even from snap?
I took a look at the pastbin logs, and I am curious which storage driver your docker engine uses, as neither overlay nor overlay2 or fuse-overlay seem to work. Can you share the output of docker info as well?
I tried docker on pve/lxc using a Ubuntu lxc image long time ago, but never really used it. From what I remember, I had problems running docker in an unprivileged lxc container without the “nesting” feature enabled. Though, it has been so long that I don’t remember the actual problems I had without nesting.
768MiB RAM: might be a little too low for OS, Docker and the containers
VFS Storage Driver: is not recommended for production environment (and wastes a lot of filesystem storage)
I have PVE 7.3 running, and the kernel comes with overlay2 support (lsmod | grep overlay). I started the docker-test container (which was created either with PVE7.1 or 7.2), and it turns out it uses overlay2 as storage driver out of the box as well.
Usually the reason is that the kernel lacks features or the docker data root is not on a suitable backing file system:
If your storage pool for the lxc containers uses zfs, then according Google Consulting the vfs storage driver is used - people don’t seem to be able to use the zfs storage driver (I didn’t take a closer look).
In this situation people seem to use fuse-overlayfs (update: ignore all posts except the ones from c-goes, actually his 2nd post should be everything necessary)
I found this post in GH discussion that describes an alternative to fuse-overlayfs.
Best I can do is provide assistance in the detective work, as I don’t use docker in lxc containers.
It is up to you to follow these leads!
Update: I added a comment for the fuse-overlay link AND forgot to mention something important: if the storage driver is changed, existing images and containers will disappear, as they are not migrated to the changed storage driver. Thus, images need to be re-pulled, containers to be re-created.
and did not let manage docker the volumes! For that I don’t lose my data! If anybody use the volume management from docker, please backup you files first before docker compose down! AND IMPORTANT see what happend to me in #13
Well, Now I have to repead this step for my 20 LXC Container But huge thanks to you @meyay for the detective work! Without I wouldn’t be ableto make that switch. Maybe then I don’t waste a lot of space now and habe a more stable system
I would like to give @meyay the solution checkmark, but since I wrote it down on thos post I would checkmark this post here for other people searching for that solution.
I got an IMPORTANT Info for everyone! On one of my LXC Container had, for whatever reason, overlay2 as storage driver with ZFS as backend FS enabled. Why did I noticed it? Because it crashed my complete Proxmox Node!