I have a host machine with 16 GB RAM, on which I have deployed some containers with docker-compose.
I did not set any memory limit for the containers.
One of the containers contains a single java process which executes a task and generates statistics regularly with Quartz. I noticed that this process no longer worked correctly because I no longer had any generated statistics. The process logs show the error java.lang.OutOfMemoryError: Java heap space.
But despite this the container is still running, it has not been killed and restarted by Docker (and the java process inside the container no longer responds anymore).
I think I understood that the container would have been killed by Docker if the memory usage had reached the memory limit of the container, whereas in my case the error occurred because the process reached the memory limit of the JVM.
Is it possible to kill and restart a process when an OOM error occured on JVM heap space, even if the memory usage has not reached the memory limit of the container ?
A heap space OOME is something that can be cought and handled in your code, so potentialy there is no reason for the JVM to stop its process. If you don’t want to handle the exception yourself, you might want to check the available options for the JVM version you use. I am quite sure that we used -XX:+ExitOnOutOfMemoryError (note there is also -XX:+CrashOnOutOfMemoryError which we did not use) in the past to force the container to die and let the swarm/k8s scheduler create a new container.
Depending on which Java version is used and what parameters you provided, the JVM may or may not support and respect cgroup limits and configures 1/4 of the hosts total memory as heap, regardless of ressource constraints on the container. If you google it, you can find plenty of good blog posts that explain the situation in detail.
For a permanent fix, you definitly will want to configure ressource constraints, enable the cgroup related featres and use system monitoring to identify the sweatspot.