Docker hung can someone suggest

krishnananda · February 27, 2020, 3:17pm

Hi ALL,

We have Ubuntu Os where Docker is hosted,where our application is running

My question is

Docker gets hung sometime due to highest cpu load .At same time when we go through the logs we are not able to find any logs regarding this.
when docker goes un responsive ,how can we check apart from system logs what

could you please suggest how we can verfiy the logs what have caused the isisue

meyay · February 28, 2020, 7:37am

You can check docker’s service logs: journalctl -u docker -b.

If the logs provide no clues, enable debugging, restart the daemon and check the logs again: https://success.docker.com/article/how-do-i-enable-debug-logging-of-the-docker-daemon

In the past, I had trouble with corrupted log files causing a huge load on the system. Execute for log in /var/lib/docker/containers/*/*-json.log; do jq < $log 2 >&1 /dev/null || echo $log corrupted;done to check if the logfiles are still valid json. Generally make sure /var/lib/docker has plenty of free diskspace.

krishnananda · February 28, 2020, 11:44am

Hi Metin,

Thanks for your reply.

Can we configure this for existing container. If so any resatart of docker is needed.

meyay · February 28, 2020, 1:49pm

You question confuses me. How would reading the log files of the docker service or checking for corrupt container logs be related to this?

Though, if you find and delete corrupted logs, you will of course lose the existing log data. If the container is running, it will start writing logs again, but the logs before deleting will be lost for this container.

krishnananda · February 29, 2020, 4:05am

Hi Metin

My question is if my docker gets chocked or stuck due to high cpu or memory,how can we verify the logs before it got stuck.

As you said we can configure the logs for existing docker whether we require the restart of docker?

meyay · February 29, 2020, 9:36am

What is “my docker” for you?
So far I assumed docker as in the docker engine. My answers aimed toward that understanding.

I started to get the feeling that you refer with docker to actual containers… well don’t. a container is not a docker.

You can get the container logs with docker logs {container id or name}. Though, this is so basic knowledge… If this realy is what you asked for, then I strongly suggest to spend some time to actualy learn the basics. I can highly recommend this excellent self paced training.

krishnananda · February 29, 2020, 5:26pm

Hi Metin,

I my saying in terms of conatiner terms. If my conatiner get stucks or cannot login how we can check the logs?

krishnananda · March 2, 2020, 6:21am

Hi Metin,

Please find below logs we are getting this continously

Feb 27 13:39:28 PS-N-INVEST-PROD dockerd[1760]: http: TLS handshake error from 172.18.0.37:60600: read tcp 10.128.41.99:2376->172.18.0.37:60600: read: connection reset
Feb 27 13:39:38 PS-N-INVEST-PROD dockerd[1760]: http: TLS handshake error from 172.18.0.37:60738: read tcp 10.128.41.99:2376->172.18.0.37:60738: read: connection reset
Feb 27 13:39:48 PS-N-INVEST-PROD dockerd[1760]: http: TLS handshake error from 172.18.0.37:60868: read tcp 10.128.41.99:2376->172.18.0.37:60868: read: connection reset
Feb 27 13:39:58 PS-N-INVEST-PROD dockerd[1760]: http: TLS handshake error from 172.18.0.37:60998: read tcp 10.128.41.99:2376->172.18.0.37:60998: read: connection reset
Feb 27 13:40:08 PS-N-INVEST-PROD dockerd[1760]: http: TLS handshake error from 172.18.0.37:32906: read tcp 10.128.41.99:2376->172.18.0.37:32906: read: connection reset
Feb 27 13:40:18 PS-N-INVEST-PROD dockerd[1760]: http: TLS handshake error from 172.18.0.37:33066: read tcp 10.128.41.99:2376->172.18.0.37:33066: read: connection reset
Feb 27 13:40:28 PS-N-INVEST-PROD dockerd[1760]: http: TLS handshake error from 172.18.0.37:33212: read tcp 10.128.41.99:2376->172.18.0.37:33212: read: connection reset
Feb 27 13:40:38 PS-N-INVEST-PROD dockerd[1760]: http: TLS handshake error from 172.18.0.37:33334: read tcp 10.128.41.99:2376->172.18.0.37:33334: read: connection reset
Feb 27 13:40:48 PS-N-INVEST-PROD dockerd[1760]: http: TLS handshake error from 172.18.0.37:33486: read tcp 10.128.41.99:2376->172.18.0.37:33486: read: connection reset
Feb 27 13:40:58 PS-N-INVEST-PROD dockerd[1760]: http: TLS handshake error from 172.18.0.37:33620: read tcp 10.128.41.99:2376->172.18.0.37:33620: read: connection reset
Feb 27 13:41:08 PS-N-INVEST-PROD dockerd[1760]: http: TLS handshake error from 172.18.0.37:33760: read tcp 10.128.41.99:2376->172.18.0.37:33760: read: connection reset
Feb 27 13:41:18 PS-N-INVEST-PROD dockerd[1760]: http: TLS handshake error from 172.18.0.37:33890: read tcp 10.128.41.99:2376->172.18.0.37:33890: read: connection reset
Feb 27 13:41:28 PS-N-INVEST-PROD dockerd[1760]: http: TLS handshake error from 172.18.0.37:34038: read tcp 10.128.41.99:2376->172.18.0.37:34038: read: connection reset
Feb 27 13:41:38 PS-N-INVEST-PROD dockerd[1760]: http: TLS handshake error from 172.18.0.37:34168: read tcp 10.128.41.99:2376->172.18.0.37:34168: read: connection reset
lines 3828-3870

krishnananda · March 5, 2020, 5:58am

Hi Metin,
Any update on this?

meyay · March 5, 2020, 9:56am

considering the details you provided so far: I am not able to provide any guidance for you.
Someone else needs to pitch in.

krishnananda · March 5, 2020, 4:53pm

Hi Metin,

Any technically wrong I spoken could please correct me if I am wrong

mrvnmchm · March 5, 2020, 5:55pm

Hi @krishnananda, from the error you provided in the logs, it seems we are not able to make a successful connection from the container to another service/system. Check out this link to see if that may help you. Also, if you could provide any other details on the configuration of these containers, that will be super helpful in helping you.