We have an Amazon EC2 instance with CentOS running java application inside a docker along with another docker of fluent. Till Docker version “19.03.6-ce, build 369ce74” we haven’t faced any issues.
But once we switched to “Docker version 19.03.13-ce, build 4484c46” we have been experiencing the java process inside the container is killed by the OS because of OOM.
After running the application for > 10 days the two containerd-shim processes are eventually ending up taking more memory and when it is reaching beyond OS limits the OS is killing the Java process inside the docker which is taking 50% of the memory. The instance type is c5.xl, the Java process was taking 5GB and each of the containerd-shim process was taking 1.5 GB each. After some more time when containerd -shim goes beyond 1.5GB the OS is killing the java process.
When we switch back to “19.03.6-ce, build 369ce74”, we don’t see the containerd-shim is consuming any memory, and the issue went away.
The OS info is:
NAME=“Amazon Linux”
VERSION=“2”
ID=“amzn”
ID_LIKE=“centos rhel fedora”
VERSION_ID=“2”
PRETTY_NAME=“Amazon Linux 2”
ANSI_COLOR=“0;33”
CPE_NAME=“cpe:2.3:o:amazon:amazon_linux:2”
HOME_URL=“https://amazonlinux.com/”