Hello. I have 60 containers packed by 4 containers into develop environment, all united by one network. Since yesterday i started to get errors “no left space on device”. I cleaned disk space, cleaned dangling volumes, now i have only 62% in use, but sometimes still get this error.
$ /var/lib/docker/aufs# docker version
Client:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built: Thu Aug 18 05:33:38 2016
OS/Arch: linux/amd64
Server:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built: Thu Aug 18 05:33:38 2016
OS/Arch: linux/amd64
$ /var/lib/docker/aufs# docker info
Containers: 69
Running: 59
Paused: 0
Stopped: 10
Images: 41
Server Version: 1.12.1
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 542
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-59-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 31.31 GiB
Name: farpost-dev
ID: DDB4:EZTG:KQU5:LH4D:BBDZ:GMMB:AAXE:N5JX:KEJX:UFP7:OG3L:C3Y2
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
127.0.0.0/8
And also two different errors in syslog:
Jan 31 08:31:26 dev dockerd[1969]: time="2017-01-31T08:31:26.575456763+07:00" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /var/lib/docker/containers/00da17227973664cfe52166faf78c06b2d96103dfaffe34142fc3bfbf2ee5247/shm: invalid argument"
Jan 31 08:31:26 dev dockerd[1969]: time="2017-01-31T08:31:26.596319882+07:00" level=error msg="Handler for POST /v1.24/containers/00da17227973664cfe52166faf78c06b2d96103dfaffe34142fc3bfbf2ee5247/start returned error: open /var/lib/docker/containers/00da17227973664cfe52166faf78c06b2d96103dfaffe34142fc3bfbf2ee5247/hash297534923: no space left on device"
Jan 31 11:58:38 dev dockerd[1969]: time="2017-01-31T11:58:38.004296665+07:00" level=error msg="libcontainerd: backend.StateChanged(): open /var/lib/docker/containers/86f4510c6d305a2b6fd54ecd90784d3a5358fdc5aa378754f6b7a2ea58d2096b/.tmp-config.v2.json433936343: no space left on device"
Jan 31 11:58:38 dev dockerd[1969]: time="2017-01-31T11:58:38.004391090+07:00" level=error msg="Error saving dying container to disk: open /var/lib/docker/containers/86f4510c6d305a2b6fd54ecd90784d3a5358fdc5aa378754f6b7a2ea58d2096b/.tmp-config.v2.json744200010: no space left on device"
When i delete some containers, error disappears for a while, and then this error strikes again. What is this?