We run docker on 4.14.219
(hence cgroups v1).
docker stats
reports insanely high memory usage, although memory.stat
contains much lower rss
.
The problem is high dentry
and ovl_inode
usage (we found it in slabinfo
)
This fact affects both memory.usage_in_bytes
and memory.kmem.usage_in_bytes
After echo 2 > /proc/sys/vm/drop_caches
everything becomes OK.
Docker doc says:
On Linux, the Docker CLI reports memory usage by subtracting cache usage from the total memory usage.
I assume Docker uses memory.usage_in_bytes
as “total memory usage”, am I right?
Kernel documentation says:
If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP)
value in memory.stat(see 5.2).
https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt
So, my question is: why use obscure usage_in_bytes
instead of RSS+SWAP?
In any case, big number of dentry and ovl_inode slabs isn’t something that should affect docker stats
, should it?
How to reproduce.
- On the container run
du --inodes -d1 /
- On the host run
docker stats
: you should see lots of memory used - Check slabs:
sed 1,2d memory.kmem.slabinfo | awk '{RES=$2*$4; print RES"\t"RES/1024/1024" MB\t"$1}' | sort -n
(in my case alsmost 10 megabytes occupied byovl_inode
- Reclaim memory (via
2 > /proc/sys/vm/drop_caches
or0 > memory.force_empty
) - Check
docker stats
again