Percent memory usage of containers after reboot

General Discussions
1

Hi,
before to restart my raspberry, the memory of containers is

NAME MEM USAGE / LIMIT MEM %
fail2ban 10.12MiB / 25MiB 40.50%
homebridge 121.4MiB / 240MiB 50.58%
zwave-js-ui 73.72MiB / 160MiB 46.07%
domoticz 156.4MiB / 360MiB 43.45%
mosquitto 1.203MiB / 6MiB 20.05%

after reboot

NAME MEM USAGE / LIMIT MEM %
fail2ban 9.66MiB / 25MiB 38.64%
homebridge 120.3MiB / 240MiB 50.12%
**zwave-js-ui 108MiB / 160MiB 67.48%**
domoticz 48.39MiB / 360MiB 13.44%
**mosquitto 3.219MiB / 6MiB 53.65%**

i don’t unserstand because the containers take more memory after reboot
and when i restart only the containers, i have this

NAME MEM USAGE / LIMIT MEM %
fail2ban 10.85MiB / 25MiB 43.39%
homebridge 31.86MiB / 240MiB 13.27%
zwave-js-ui 31.3MiB / 160MiB 19.56%
domoticz 29.11MiB / 360MiB 8.09%
mosquitto 2.031MiB / 6MiB 33.85%

Do you have a explcation .?

2

Maybe the processes inside the containers are doing something that requires more memory when the host is not fully started yet. You hhighlighted some of the containers, but some containers use even less memory than before, so it should depend on what the processes do.

3

how can i detect or debug the problem ?

4

I don’t know. Maybe someone who knows those applications can give you some idea what it could do differently when it starts.You can try enable verbose/debug logs in the containers and hope it logs more information. You can try to delay starting Docker jut to test what happens if it has more time before starting. You would need to add a sleep in the Systemd service:

This is the service file

/etc/systemd/system/multi-user.target.wants/docker.service

Then you will need a systemctl daemon-reloadand systemctl restart docker to test if the configuration is correct before you reboot the machine.

5

i execute docker builder prune and i tried different tests
stop container , stop docker service and reboot same problem
i test the different parameter in the docker.service

ExecStartPre=/bin/sleep 10
ExecStartPost=/bin/sleep 10
ExecStop=/bin/sleep 10
but same problem

next reboot, i launched sudo docker restart $(sudo docker ps -q) and the ram is ok, i’m searching already another way

i tried also
sudo systemctl disable docker.service
sudo systemctl disable docker.socket
and reboot same problem after to launch manualy docker service

the problem is the reboot,@rimelek you say : maybe. the processes inside the containers are doing something that requires more memory when the host is not fully started yet
i reactivated the swap , but same problem

perhaps is it the cgroup_enable=memory?
perhaps after reboot the system doesn’t refresh the memory value

6

A another test to stop the log2ram service, same problem

7

Sorry, but wasn’t around much. I can’t give you more ideas. Docker would not just “give” processes more memory. If you see in the statistics that the applications used more memory, then some state is different which makes the processes use more memory. What it could be caused by, I don’t know. If it is related to the containerized environment, maybe it has to do something with the overlay filesystem, but I can’t imagine how. If the additional memory usage is caused something that is not fully running yet when Docker starts, that is hard to debug. Sleep was just an idea, but apparently it didn’t help. If you find the answer, I appreciate if you share it. Until that, is this problem critical for you at the moment?

8

it’s not critical
i saw in the log this , perhaps the problem is the container cleanup ?

Apr 14 18:41:13 raspberrypi systemd[1]: docker-2f81854e0b4e552de46317636bf577549c118754d515fe5dd58cbafcc149891c.scope: Succeeded.
Apr 14 18:41:13 raspberrypi systemd[1]: docker-2f81854e0b4e552de46317636bf577549c118754d515fe5dd58cbafcc149891c.scope: Consumed 7.720s CPU time.
Apr 14 18:41:13 raspberrypi containerd[427]: time="2024-04-14T18:41:13.447097844+02:00" level=info msg="shim disconnected" id=2f81854e0b4e552de46317636bf577549c118754d515fe5dd58cbafcc149891c
Apr 14 18:41:13 raspberrypi containerd[427]: time="2024-04-14T18:41:13.447769248+02:00" level=warning msg="cleaning up after shim disconnected" id=2f81854e0b4e552de46317636bf577549c118754d515fe5dd58cbafcc149891c namespace=moby
Apr 14 18:41:13 raspberrypi containerd[427]: time="2024-04-14T18:41:13.447981904+02:00" level=info msg="cleaning up dead shim"
Apr 14 18:41:13 raspberrypi dockerd[567]: time="2024-04-14T18:41:13.450279449+02:00" level=info msg="ignoring event" container=2f81854e0b4e552de46317636bf577549c118754d515fe5dd58cbafcc149891c module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Apr 14 18:41:13 raspberrypi containerd[427]: time="2024-04-14T18:41:13.525346260+02:00" level=warning msg="cleanup warnings time=\"2024-04-14T18:41:13+02:00\" level=info msg=\"starting signal loop\" namespace=moby pid=2595 runtime=io.containerd.runc.v2\n"
Apr 14 18:41:13 raspberrypi dockerd[567]: time="2024-04-14T18:41:13.536992684+02:00" level=warning msg="ShouldRestart failed, container will not be restarted" container=2f81854e0b4e552de46317636bf577549c118754d515fe5dd58cbafcc149891c daemonShuttingDown=false error="restart canceled" execDuration=3m23.815416028s exitStatus="{0 2024-04-14 16:41:13.293148608 +0000 UTC}" hasBeenManuallyStopped=true restartCount=0
Apr 14 18:41:13 raspberrypi containerd[427]: time="2024-04-14T18:41:13.970259174+02:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Apr 14 18:41:13 raspberrypi containerd[427]: time="2024-04-14T18:41:13.971263598+02:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Apr 14 18:41:13 raspberrypi containerd[427]: time="2024-04-14T18:41:13.971465941+02:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Apr 14 18:41:13 raspberrypi containerd[427]: time="2024-04-14T18:41:13.973386404+02:00" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/2f81854e0b4e552de46317636bf577549c118754d515fe5dd58cbafcc149891c pid=2614 runtime=io.containerd.runc.v2
Apr 14 18:41:14 raspberrypi systemd[1]: run-docker-runtime\x2drunc-moby-2f81854e0b4e552de46317636bf577549c118754d515fe5dd58cbafcc149891c-runc.BoEP0a.mount: Succeeded.
Apr 14 18:41:14 raspberrypi systemd[1777]: run-docker-runtime\x2drunc-moby-2f81854e0b4e552de46317636bf577549c118754d515fe5dd58cbafcc149891c-runc.BoEP0a.mount: Succeeded.
Apr 14 18:41:14 raspberrypi systemd[1]: Started libcontainer container 2f81854e0b4e552de46317636bf577549c118754d515fe5dd58cbafcc149891c.
Apr 14 18:41:24 raspberrypi dockerd[567]: time="2024-04-14T18:41:24.626113378+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=4079b4b8dcb93a020d93c27cef46c6d33eafca6835ab04b6b2edce1512a7cd47 spanID=790e289cebb26f6c traceID=11910c00fd2298c613b85260931d360c
9

Cleanups have nothing to do with how much resources working containers use. If there is some indirect connection, I have no idea what. When a container is killed forcefully after 10 seconds, that means the process inside doesn’t handle signals correctly. That is something you should fix, because it makes stopping containers slower and it could lead to data corruption, but I don’t think it makes other containers use more memory.