One of our Docker machines is currently running 183 containers.
The dockerd process on this machine (by itself) has a virtual size of 5.3GiB. It’s tempting to say “Oh, that’s just virtual address space, it isn’t really used.” But it is. The system memory reflects it and once that’s exhausted, it digs into swap. At this point, all but 42MiB of dockerd has been swapped out. (And yes, this is real usage, not unused allocations; the pages are almost all dirty and enough swap space is consumed to fully back this allocation.)
A docker-containerd process has a further 3.6GiB of virtual size, of which 4MiB is resident, and the rest is (again) swap-backed.
The docker-containerd process referenced here has the command line:
docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim docker-containerd-shim --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --runtime docker-runc
This appears to be some sort of “system” container; it isn’t one of ours, and all of the rest of the docker processes are children of this one.
All told, this system has 12GiB of RAM and 12GiB of swap. Of that 24GiB, about 18GiB is used (e.g. the machine is about 6.5GiB into swap). Pretty much that whole 6.5GiB appears attributable to these two processes.
By some back-of-the-envelope calculations, about 57% of that 18GiB, or about 10GiB, is attributable to processes with “docker” in the name.
For 183 containers, this seems excessive. The RAM usage does appear to be roughly linear with the number of containers. It happens immediately after containers start and stays roughly static after that so it doesn’t appear to be a gradual leak.
This is Docker 1.12.1 on Ubuntu 16.04 LTS, but this behavior has been observed on each of the handful of Docker servers we set up over the course of many versions.
This isn’t the “Lightweight, uses less RAM!” that docker.com talks about, and at the same time, it doesn’t seem like this is a common complaint.
So, is this level of overhead expected, just something we have to plan for? (Mainly by adding tons of extra swap space.) Or is it (more likely) that we’re doing something dumb to cause this?
Thanks for any advice!