I’ve got a process that is a Java app that runs another Java app. Most of the time, this works great - but some times (<1% of the time), the second app will OOM hard (hotspot crashes and creates a log file, not just getting an OOM exception thrown).
Re-running the app with the same data on ECS will continue to fail consistently.
The log file has this in it (along with a stack trace saying “mmap failed for CEN and END part of zip file” and it appears to be something related to java.util.logging.Logger?
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed. Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f890618b000, 12288, 0) failed; error='Cannot allocate memory' (errno=12) # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory. # An error report file with more information is saved as: # ...my path here...
I’ve tried all manner of things - increasing (and decreasing) the Xmx value. The combined values for the JVMs and the container are now less than half of the total memory available on the EC2 instance, so I’m thinking this is less about the amount of memory needed that it is about the methods used to allocate it.
If I run the same docker container on my desktop (OSX) with the same memory settings, it works 100% of the time.
I’ve been tinkering with this for a couple of weeks and have tried everything I can think of.
Anyone have any ideas?