Docker Community Forums

Share and learn in the Docker community.

No left space on device error

Hello. I have 60 containers packed by 4 containers into develop environment, all united by one network. Since yesterday i started to get errors “no left space on device”. I cleaned disk space, cleaned dangling volumes, now i have only 62% in use, but sometimes still get this error.

$ /var/lib/docker/aufs# docker version
Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:33:38 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:33:38 2016
 OS/Arch:      linux/amd64

$ /var/lib/docker/aufs# docker info
Containers: 69
 Running: 59
 Paused: 0
 Stopped: 10
Images: 41
Server Version: 1.12.1
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 542
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-59-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 31.31 GiB
Name: farpost-dev
ID: DDB4:EZTG:KQU5:LH4D:BBDZ:GMMB:AAXE:N5JX:KEJX:UFP7:OG3L:C3Y2
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

And also two different errors in syslog:

Jan 31 08:31:26 dev dockerd[1969]: time="2017-01-31T08:31:26.575456763+07:00" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /var/lib/docker/containers/00da17227973664cfe52166faf78c06b2d96103dfaffe34142fc3bfbf2ee5247/shm: invalid argument"
Jan 31 08:31:26 dev dockerd[1969]: time="2017-01-31T08:31:26.596319882+07:00" level=error msg="Handler for POST /v1.24/containers/00da17227973664cfe52166faf78c06b2d96103dfaffe34142fc3bfbf2ee5247/start returned error: open /var/lib/docker/containers/00da17227973664cfe52166faf78c06b2d96103dfaffe34142fc3bfbf2ee5247/hash297534923: no space left on device"

Jan 31 11:58:38 dev dockerd[1969]: time="2017-01-31T11:58:38.004296665+07:00" level=error msg="libcontainerd: backend.StateChanged(): open /var/lib/docker/containers/86f4510c6d305a2b6fd54ecd90784d3a5358fdc5aa378754f6b7a2ea58d2096b/.tmp-config.v2.json433936343: no space left on device"
Jan 31 11:58:38 dev dockerd[1969]: time="2017-01-31T11:58:38.004391090+07:00" level=error msg="Error saving dying container to disk: open /var/lib/docker/containers/86f4510c6d305a2b6fd54ecd90784d3a5358fdc5aa378754f6b7a2ea58d2096b/.tmp-config.v2.json744200010: no space left on device"

When i delete some containers, error disappears for a while, and then this error strikes again. What is this?