COPY requirements.txt ./
RUN apt-get update && apt-get install -y default-mysql-client
RUN pip install --no-cache-dir -r requirements.txt
CMD [ "python", "app.py" ]
app.py is a failrly simple lightweight script with no CPU or memory usage, running 24/7. It produces a lot of logs into container as well as writing into the MySQL database. After approximately 15-18 hours, it stops saving logs on around line #47,000 (I can see a last timestamp), it then continue working and saving data to the database for about one more hour and then it stops - no logs (except 47k of successfull runs history), no new data saved to the DB.
When python execution suddently stops, I can attach to the container, see files there, etc. But I cannot stop it - “docker compose stop” keeps counting seconds indefinately without stopping it. The only solution at this point is to do “systemctl restart docker.socket docker.service” which then keeps it going for another 15 or so hours.
As an additional info, exactly same symptoms were happening with unmodified Selenium Hub container. App itself is not responding, nodes not connecting, can’t be stopped unless docker is restarted, but I could attach to it and see filesystem. So I assume it should not be my python application, but rather some incompatability/configuration of docker itself.
The output of docker info looks good to me. We can see that the overlay2 storage driver is used and everything else looks pretty much like on every other Ubuntu 22.04 system.
Next time you have the issue, you might want to check the system load as well (e.g. with top, htop or even uptime). The numbers are average load during the last minute, the last fife minutes and the last fifteen minutes. With 4 cpu’s your system can handle load up to 4.0, everything beyond that point will cause system behavior to slow down. This is just a shot in the dark, but I feel it is worth mentioning it.