Docker daemon hangs, auplink stuck on flush

I am experiencing issues with containers that cannot be killed and removed.

Distribution: Ubuntu 14.04.4 LTS
Kernel: 3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
App version: docker-engine 1.11.0-0~trusty

Rough steps to reproduce:

  • Start container docker run -it --rm option.
  • The program that is being called stops making progress - as far as I can tell it has completed its work and I would have expected for the process to be done, but it seems to hang around
  • docker stop hangs - docker logs will show “failed to exit within 10 seconds of kill - trying direct SIGKILL”
  • docker kill hangs
  • Stop docker service: service docker stop
  • Stop docker service: service docker start
  • docker ps hangs
  • service docker stop hangs
  • kill -9 all docker related processes
  • I notice that a process is stuck on this command: auplink /var/lib/docker/aufs/mnt/84053163ab... flush
  • That process cannot be killed.

It has happened twice already and the first time required a hard reboot of the box.

What is the best way to troubleshoot that further? Would the auplink command not terminating cause the docker daemon to fail to run properly?

The following command by itself won’t start any container.
docker run -it --rm

Which Docker image is used?

I’m using a private image that is based on one of the official java image, the program is a java program that reads large amount of data (GBs) from standard input and writes back to standard output as well as connect to REST services,

It appears that it was probably a manifestation of https://github.com/docker/docker/issues/18180

After upgrading the linux kernel and docker doing a sudo apt-get update and sudo apt-get dist-upgrade and reboot, I have been running 1000+ container with docker run -it --rm without issue.