Docker Containers Not Utilizing Dynamic Memory in Hyper-V Ubuntu VM

Docker Engine General
, , ,
1

Docker containers inside our Hyper-V Ubuntu VM are not utilizing dynamic memory allocation as expected.

Hyper-V Dynamic Memory: The VM is configured with dynamic memory, and Hyper-V is allocating additional memory as needed.
Docker Memory Usage: Despite more memory being available to the VM, Docker containers remain restricted to the initial startup memory.
Lack of Dynamic Scaling: Containers do not scale their memory usage dynamically based on the additional memory assigned to the VM.

2

How did you test it: There is no memory limit by default for any container, so the container (the isolation) can use as much memory as available on the host. The only thing that could restrict memory usage are control groups, but in case of memory, it means if the process tries to use too much memory, it is killed by the OS. Limits in control groups can be set with docker commands, but there is no limit by default

What I can imagine is that the application starts in the container and detects the available memory and configures itself to use as much memory as possible at the moment, but it doesn’t check it again until restarting.

3

Thanks for your response!

Testing Details:

  • My Hyper-V VM (Ubuntu) is set to Dynamic Memory, and I confirmed that Hyper-V is assigning more memory to the VM up to the maximum limit set for the VM when needed.
  • Inside the VM, free -m and /proc/meminfo show that additional memory is being assigned.
  • However, Docker containers inside this VM continue to use the initial startup memory as their limit, even after increasing the VM’s memory.

What I Have Checked:

  • cgroups are already set to max, so no restrictions there.
  • Restarting the container does not update the memory limit (docker stats still shows the old limit).
  • Even when putting memory stress inside the VM (stress-ng), Docker containers still do not exceed the original memory allocation.

Issue:

  • The application inside the container calculates available memory only once at startup, but even restarting the container does not allow it to recognize the new memory.
  • I suspect that Docker is somehow still limiting the memory to the startup allocation, even though no explicit memory limits were set.
4

What cgroups? Can you be omore specific? Did you for example check the output of this command for example?

cat /sys/fs/cgroup/system.slice/memory.max

At this point I also have to ask about how you installed Docker. Can you also share the output of the following commands?

docker info
docker version
snap list docker
dpkg -l ‘docker*’ | grep ‘^ii’
5 
cat /sys/fs/cgroup/system.slice/memory.max
max

docker info
 
Client:  
Context:    default  
Debug Mode: false  
Plugins:  
  app: Docker App (Docker Inc., v0.9.1-beta3)  
  buildx: Docker Buildx (Docker Inc., v0.9.1-docker)  
Server:  
Containers: 26  
  Running: 23  
  Paused: 0  
  Stopped: 3  
Images: 24  
Server Version: 20.10.22  
Storage Driver: overlay2  
  Backing Filesystem: extfs  
  Supports d_type: true  
  Native Overlay Diff: true  
  userxattr: false  
Logging Driver: json-file  
Cgroup Driver: systemd  
Cgroup Version: 2  
Plugins:  
  Volume: local  
  Network: bridge host ipvlan macvlan null overlay  
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog  
Swarm: inactive  
Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux  
Default Runtime: runc  
Init Binary: docker-init  
containerd version: 9ba4b250366a5ddde94bb7c9d1def331423aa323  
runc version: v1.1.4-0-g5fd4c4d  
init version: de40ad0  
Security Options:  
  apparmor  
  seccomp  
   Profile: default  
  cgroupns  
Kernel Version: 5.15.0-133-generic  
Operating System: Ubuntu 22.04.4 LTS  
OSType: linux  
Architecture: x86_64  
CPUs: 3  
Total Memory: 14.64GiB  
Docker Root Dir: /var/lib/docker  
Debug Mode: false  
Registry: https://index.docker.io/v1/  
Labels:  
Experimental: false  
Insecure Registries:  
  127.0.0.0/8  
Live Restore Enabled: false

docker version
 
Client:  
Docker Engine - Community  
Version:           20.10.22  
API version:       1.41  
Go version:        go1.18.9  
Git commit:        3a2c30b  
Built:             Thu Dec 15 22:28:04 2022  
OS/Arch:           linux/amd64  
Context:           default  
Experimental:      true  
Server:  
Docker Engine - Community  
Engine:  
  Version:          20.10.22  
  API version:      1.41 (minimum version 1.12)  
  Go version:       go1.18.9  
  Git commit:       42c8b31  
  Built:            Thu Dec 15 22:25:49 2022  
  OS/Arch:          linux/amd64  
  Experimental:     false  
containerd:  
  Version:          1.6.14  
  GitCommit:        9ba4b250366a5ddde94bb7c9d1def331423aa323  
runc:  
  Version:          1.1.4  
  GitCommit:        v1.1.4-0-g5fd4c4d  
docker-init:  
  Version:          0.19.0  
  GitCommit:        de40ad0

snap list docker
 
error: no matching snaps installed
 
dpkg -l ‘docker*’ | grep ‘^ii’
 
dpkg-query: no packages found matching ‘docker*’
6

Your Dockr seems to be a bit old, but that should matter as Docker is not responsible for control groups directly. Since I never tried dynamic memory in Hyper-V, I can only guess something like that Linux distribution or more likely in the kernel doesn’t support using the increaed resources in containers. Containers are ran by a container runtime. The default is containerd which also has a low level runtime, called “runc”.

Creating containers using runc is not too user friendly, but you can try if you experience the same with containerd. A docker-like client for it is nerdctl

GitHub - containerd/nerdctl: contaiNERD CTL - Docker-compatible CLI for containerd, with support for Compose, Rootless, eStargz, OCIcrypt, IPFS, ....