Docker hanging/zombie process

General
1

Last night we experienced problems stopping and starting containers, and when I ran a ps I saw this:

4 Z root 4986 1 0 80 0 - 0 exit Aug04 ? 00:01:23 [docker]

We rebooted the machine and then all seemed OK. Today I tried to build a new container and it hung. Tried it multiple times and each time it hung in a different place. I was able to run ‘docker ps’ but ‘docker info’ hung. I tired to remove a container with ‘docker rm’ and that hung, then after that ‘docker ps’ and every docker command hung. A ps showed:

4 S root 4895 1 0 80 0 - 305052 futex_ Aug06 ? 00:01:01 /usr/bin/docker -d

I tried to do ‘sudo service docker restart’ and that hung and now docker is again in the zombie state:

4 Z root 4895 1 0 80 0 - 0 exit Aug06 ? 00:01:24 [docker]

What could be the cause of these sudden problems? How can I debug and fix this? We are dead in the water without docker working reliably.

2

Rebooted the machine and still every docker command hangs. When I do a ‘docker rm’ I see this in the system logs:

device-mapper: ioctl: unable to remove open device docker-253:6-1048578-317bb6ad40cded3fbfd752d95551861c2e4ef08dffc1186853fea0e85da6b12b
INFO: task docker:16676 blocked for more than 120 seconds.
Not tainted 2.6.32-573.12.1.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
docker D 000000000000000b 0 16676 1 0x00000080
ffff88035ef13ea8 0000000000000082 ffff88035ef13e70 ffff88035ef13e6c
ffff88035ef13e28 ffff88062fc29a00 0000376c85170937 ffff8800283759c0
0000000000000400 00000001039d40c7 ffff8803000445f8 ffff88035ef13fd8
Call Trace:
[] __mutex_lock_slowpath+0x96/0x210
[] ? wake_up_process+0x15/0x20
[] mutex_lock+0x2b/0x50
[] sync_filesystems+0x26/0x150
[] sys_sync+0x17/0x40
[] system_call_fastpath+0x16/0x1b

3

We still have this happening quite often. The docker daemon hangs and the only option is a reboot. This makes docker somewhat unsuitable for a production deployment. Anyone have any suggestions on what to do about this?

We are running on RHEL 6.7 with docker 1.7.1, build 786b29d. Can or should I install a new version? That is the version yum installs. If I did want a new version how would I install that (sorry if that is a dumb question, I am not a sys admin).

Googling, I found on this a Red Hat site “Red Hat does not recommend running any version of Docker on any RHEL 6 releases.” We have been running docker on RHEL 6 for a few years, so this confuses me. Upgrading to RHEL 7 is not really an option for us right now.

Can anyone shed any light on these issue? We need docker to work reliably without having to reboot often.

4

FWIW, I’m getting similar behaviour on CEntOS7.