Docker Community Forums

Share and learn in the Docker community.

Unable to remove filesystem

I’m trying to remove a stopped docker when I’m getting

% docker rm plex
Error response from daemon: unable to remove filesystem for 1407e51fefad6dec47eeb93aedbd0dece7fe35c0a71cc459194ed916a7f19cf1: unlinkat /var/lib/docker/containers/1407e51fefad6dec47eeb93aedbd0dece7fe35c0a71cc459194ed916a7f19cf1: directory not empty

If I try to remove it manually:
rm: cannot remove ‘/var/lib/docker/containers/1407e51fefad6dec47eeb93aedbd0dece7fe35c0a71cc459194ed916a7f19cf1’: Directory not empty

% sudo ls -l /var/lib/docker/containers/1407e51fefad6dec47eeb93aedbd0dece7fe35c0a71cc459194ed916a7f19cf1
ls: cannot access /var/lib/docker/containers/1407e51fefad6dec47eeb93aedbd0dece7fe35c0a71cc459194ed916a7f19cf1/hostconfig.json: No such file or directory
ls: cannot access /var/lib/docker/containers/1407e51fefad6dec47eeb93aedbd0dece7fe35c0a71cc459194ed916a7f19cf1/hostconfig.json: No such file or directory
ls: cannot access /var/lib/docker/containers/1407e51fefad6dec47eeb93aedbd0dece7fe35c0a71cc459194ed916a7f19cf1/hostconfig.json: No such file or directory
ls: cannot access /var/lib/docker/containers/1407e51fefad6dec47eeb93aedbd0dece7fe35c0a71cc459194ed916a7f19cf1/hostconfig.json: No such file or directory
ls: cannot access /var/lib/docker/containers/1407e51fefad6dec47eeb93aedbd0dece7fe35c0a71cc459194ed916a7f19cf1/hostconfig.json: No such file or directory
ls: cannot access /var/lib/docker/containers/1407e51fefad6dec47eeb93aedbd0dece7fe35c0a71cc459194ed916a7f19cf1/hostconfig.json: No such file or directory
ls: cannot access /var/lib/docker/containers/1407e51fefad6dec47eeb93aedbd0dece7fe35c0a71cc459194ed916a7f19cf1/hostconfig.json: No such file or directory
total 0
-????????? ? ? ? ?            ? hostconfig.json
-????????? ? ? ? ?            ? hostconfig.json
-????????? ? ? ? ?            ? hostconfig.json
-????????? ? ? ? ?            ? hostconfig.json
-????????? ? ? ? ?            ? hostconfig.json
-????????? ? ? ? ?            ? hostconfig.json
-????????? ? ? ? ?            ? hostconfig.json

Any ideas?

I would put my money on a corrupt filesystem.

if findmnt --target /var/lib/docker returns that the target is /, a reboot should initiate a filesystem check, if it’s not, the source should be a different drive other than the one for the root filesystem.

In the second case:
– Stop the docker daemon: service docker stop
– unmount the drive: umount /dev/{drive}
– use the check/repair tool for the used filesystem: e.g. fsck.xfs -n /dev/{drive}, fsck.ext4 /dev/{drive} or whatever filesystem you use and however its fsck. needs to be called
– mount the device again: mount -a (assumed it’s registred in /etc/fstab)
– Start the docker deaemon: service docker start
– Delete the container: docker rm plex

Thank you, I ended up rebooting the server, that solved the issue

Docker is unable to remove a container if another container has bind-mounted /var/lib/docker/containers. Only hits with older kernel, e.g. 3.13. Does not happen with newer kernel, e.g. 3.19. Also does not happen with Docker 1.8.

This is a repro carefully crafted to use only the simple alpine image, nothing fancy.

docker version:

Client:
Version: 1.9.0
API version: 1.21
Go version: go1.4.3
Git commit: 76d6bc9
Built: Tue Nov 3 19:20:09 UTC 2015
OS/Arch: linux/amd64

Server:
Version: 1.9.0
API version: 1.21
Go version: go1.4.3
Git commit: 76d6bc9
Built: Tue Nov 3 19:20:09 UTC 2015
OS/Arch: linux/amd64
docker info:

Containers: 1
Images: 93
Server Version: 1.9.0
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 95
Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-55-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 1
Total Memory: 490 MiB
Name: weave-gs-01
ID: LLWG:CZWC:AHZP:EA5W:5MG5:5IOC:C6GD:JKBJ:ZI2V:FMFT:ZNXG:G2OW
WARNING: No swap limit support
uname -a:

Linux weave-gs-01 3.13.0-55-generic #92-Ubuntu SMP Sun Jun 14 18:32:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Environment details (AWS, VirtualBox, physical, etc.):
VirtualBox VM in the above example; get identical symptoms from stock GCE RHEL 7.1.

How reproducible:
100% on older kernel; does not happen on newer kernel.

Steps to Reproduce:

docker run --name c1 -tdi alpine /bin/sh 3b1a8ae179c72c2b411e65a8e4099b61fbfd91f1843f86594bea51f56ca40766 docker run --name c2 -tdi -v /var/lib/docker/containers:/var/lib/docker/containers alpine /bin/sh
1938a9588bfd4708f82294ab9b2be509b6053a9802482565916ab403ae1fe729
$ docker rm -f c1
Actual Results:

Error response from daemon: Unable to remove filesystem for 3b1a8ae179c72c2b411e65a8e4099b61fbfd91f1843f86594bea51f56ca40766: remove /var/lib/docker/containers/3b1a8ae179c72c2b411e65a8e4099b61fbfd91f1843f86594bea51f56ca40766/shm: device or resource busy
Error: failed to remove containers: [c1]