We run docker on 4.14.219 (hence cgroups v1).
docker stats reports insanely high memory usage, although memory.stat contains much lower rss.
The problem is high dentry and ovl_inode usage (we found it in slabinfo)
This fact affects both memory.usage_in_bytes and memory.kmem.usage_in_bytes
After echo 2 > /proc/sys/vm/drop_caches everything becomes OK.
Docker doc says:
On Linux, the Docker CLI reports memory usage by subtracting cache usage from the total memory usage.
I assume Docker uses memory.usage_in_bytes as “total memory usage”, am I right?
Kernel documentation says:
If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP)
value in memory.stat(see 5.2).
https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt
So, my question is: why use obscure usage_in_bytes instead of RSS+SWAP?
In any case, big number of dentry and ovl_inode slabs isn’t something that should affect docker stats, should it?
How to reproduce.
- On the container run
du --inodes -d1 / - On the host run
docker stats: you should see lots of memory used - Check slabs:
sed 1,2d memory.kmem.slabinfo | awk '{RES=$2*$4; print RES"\t"RES/1024/1024" MB\t"$1}' | sort -n(in my case alsmost 10 megabytes occupied byovl_inode - Reclaim memory (via
2 > /proc/sys/vm/drop_cachesor0 > memory.force_empty) - Check
docker statsagain