Dockerd hangs for ~5m after stopping a long-running container

General Discussions
1

I’m running docker 1.13.0 on Centos 7.3 (with kernel 4.9.6). I’m running several containers, two of which are running wildfly web servers. When I stop them shortly after starting them, there are no problems. However, after the containers have been running for a couple days, then stopping them takes on the order of 5 minutes. When I watch the docker logs output after issuing the docker stop command, the process seems to stop within a few seconds. If I have a shell in the container, I get kicked out after a few seconds. At that point, the docker stop command appears to be hung, and the dockerd process is showing 100% CPU usage. Any other docker commands I attempt to run also hang. After about 5 minutes, the docker stop command completes and other docker commands start working again. If I restart the container, I can stop it and the daemon responds within a few seconds.

It seems maybe the daemon is very busy doing some kind of cleanup for the long running container, but I haven’t been able to figure out where/what. Any ideas of where to look?

2

containers usually log into a jsonfile. This is usually unlimited.
So when your containers run for a while, they collected a lot of trashy logs.
I assume you started your container with --rm=true, so it seems to be then busy with deleting all the trashy logs.

You could use another logdriver like syslog or maybe also like none.
Or you could limit the jsonfile to 10m for example. --log-opt max-size=10m

3

That’s a good point and we probably do need to switch our logging to syslog or something else, but I don’t believe that could be causing my issue… The container was not started with --rm=true and on one that’s been running for 11 days, there is only about 5500 lines of output from docker logs. The container is mostly sitting idle (a load balancer is hitting it for health checks, but that’s about it right now).