Systemctl daemon-reload reset docker container cgroup

Docker Engine General
1

We are seeing the following behavior on one of our AWS instance:

root@ip-10-8-40-48:/sys/fs/cgroup/system.slice/docker-7c27c485ca518ce503bcde8c3565a3fbffa0abee9bed8994951bc2bd1e8a84bf.scope$ cat memory.swap.max

0

root@ip-10-8-40-48:/sys/fs/cgroup/system.slice/docker-7c27c485ca518ce503bcde8c3565a3fbffa0abee9bed8994951bc2bd1e8a84bf.scope# systemctl daemon-reload

root@ip-10-8-40-48:/sys/fs/cgroup/system.slice/docker-7c27c485ca518ce503bcde8c3565a3fbffa0abee9bed8994951bc2bd1e8a84bf.scope# cat memory.swap.max

max

Seems the systemctl daemon-reload reset the docker container cgroup as a side effect.

Version infor as following:

root@ip-10-8-40-48:/sys/fs/cgroup/system.slice/docker-7c27c485ca518ce503bcde8c3565a3fbffa0abee9bed8994951bc2bd1e8a84bf.scope# docker info

Client: Docker Engine - Community

Version: 27.3.1

Context: default

Debug Mode: false

Plugins:

buildx: Docker Buildx (Docker Inc.)

Version:  v0.17.1

Path:     /usr/libexec/docker/cli-plugins/docker-buildx

compose: Docker Compose (Docker Inc.)

Version:  v2.29.7

Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:

Containers: 12

Running: 12

Paused: 0

Stopped: 0

Images: 87

Server Version: 27.3.1

Storage Driver: overlay2

Backing Filesystem: extfs

Supports d_type: true

Using metacopy: false

Native Overlay Diff: true

userxattr: false

Logging Driver: json-file

Cgroup Driver: systemd

Cgroup Version: 2

Plugins:

Volume: local

Network: bridge host ipvlan macvlan null overlay

Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog

Swarm: inactive

Runtimes: io.containerd.runc.v2 nvidia runc

Default Runtime: runc

Init Binary: docker-init

containerd version: 88bf19b2105c8b17560993bee28a01ddc2f97182

runc version: v1.2.2-0-g7cb3632

init version: de40ad0

Security Options:

seccomp

Profile: builtin

cgroupns

Kernel Version: 6.8.0-1019-aws

Operating System: Ubuntu 22.04.5 LTS

OSType: linux

Architecture: x86_64

CPUs: 16

Total Memory: 62.1GiB

Name: ip-10-8-40-48

ID: 30bb0ae0-1f2e-4803-a179-83bbc6a4a8bf

Docker Root Dir: /prestage/docker

Debug Mode: false

Experimental: false

Insecure Registries:

127.0.0.0/8

Live Restore Enabled: false

Default Address Pools:

Base: 192.168.0.0/16, Size: 24

WARNING: bridge-nf-call-iptables is disabled

WARNING: bridge-nf-call-ip6tables is disabled

os version as following:

root@ip-10-8-40-48:/sys/fs/cgroup/system.slice/docker-7c27c485ca518ce503bcde8c3565a3fbffa0abee9bed8994951bc2bd1e8a84bf.scope# uname -a

Linux ip-10-8-40-48 6.8.0-1019-aws #21~22.04.1-Ubuntu SMP Thu Nov 7 17:33:30 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

2

Docker is currently at 28.5.2. I would update first, Your Docker version is over a year old, probably many improvements have been applied in the meantime.

3

I don’t understand why, but I could reproduce this. I don’t know why daemon-reload would reset anything when systemd is the cgroup driver of Docker. I don’t know how it could be fixed in Docker. Maybe it can be defined somehow which could not be reset, or it is a systemd bug.

Since you haven’t shared how you created the containers, I first suspected you changed the values after creating the containers. That took me some extra time to reproduce it, but I share how I did it in Docker CE 28.5.2

docker run -d --name test --memory-swap=50m  --memory=50m nginx

This creates the container with memory.swap.max set to 0. Then I ran systemctl daemon-reload as you and indeed, the value was changed back to max until I restarted the container.

You could try reporting it in the moby repository

Then the developers can decide if this can be fixed in Docker or it is a systemd bug. I used Ubuntu 24.04 by the way, but also tied it. on Ubuntu 22.04. I have no other environments at the moment, so I don1t know if it happens only on Ubuntu or in general.

update:

I also made this test script:

id=$(docker run -d --memory-swap=50m  --memory=50m nginx)

di() { docker inspect "$id" --format '/sys/fs/cgroup/system.slice/docker-{{ .ID }}.scope/memory.swap.max' | xargs -- cat; }

di
systemctl daemon-reload
di

docker rm -f "$id"
1 Like
4

Hi @rimelek Thanks for looking, I was creating the docker container the same way as you did, with the –memory and –memory-swap.

I have created the ticket in moby: Systemctl daemon-reload reset docker container cgroup · Issue #51446 · moby/moby · GitHub

Will let you know if I got anything.

5

I tested on latest Ubuntu and Docker, still reset, added info to Github.

6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.