Docker container memory usage not going down

Hi, I have a flask web server running with docker-compose. When the container first starts, it starts by using around 200MB of memory but after some use goes up to 1GB (docker stats). However once it reaches high consumption, even when idle, the memory usage of the container is not decreasing and eventually hitting the limit - causing dead uwsgi workers and stopped processes.
Can someone explain what happens behind the curtain and how to have the container release unused memory?