Docker daemon hangs, auplink stuck on flush

Docker Engine General
1

I am experiencing issues with containers that cannot be killed and removed.

Distribution: Ubuntu 14.04.4 LTS
Kernel: 3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
App version: docker-engine 1.11.0-0~trusty

Rough steps to reproduce:

  • Start container docker run -it --rm option.
  • The program that is being called stops making progress - as far as I can tell it has completed its work and I would have expected for the process to be done, but it seems to hang around
  • docker stop hangs - docker logs will show “failed to exit within 10 seconds of kill - trying direct SIGKILL”
  • docker kill hangs
  • Stop docker service: service docker stop
  • Stop docker service: service docker start
  • docker ps hangs
  • service docker stop hangs
  • kill -9 all docker related processes
  • I notice that a process is stuck on this command: auplink /var/lib/docker/aufs/mnt/84053163ab... flush
  • That process cannot be killed.

It has happened twice already and the first time required a hard reboot of the box.

What is the best way to troubleshoot that further? Would the auplink command not terminating cause the docker daemon to fail to run properly?

2

The following command by itself won’t start any container.
docker run -it --rm

Which Docker image is used?

3

I’m using a private image that is based on one of the official java image, the program is a java program that reads large amount of data (GBs) from standard input and writes back to standard output as well as connect to REST services,

4

It appears that it was probably a manifestation of https://github.com/docker/docker/issues/18180

After upgrading the linux kernel and docker doing a sudo apt-get update and sudo apt-get dist-upgrade and reboot, I have been running 1000+ container with docker run -it --rm without issue.