Issue:
We have a setup where we have several containers deployed within a docker network. The containers communicate with each other through that network. And in those several containers, we have a mongo container, which is used by other containers to save the data in mongodb.
And we have a good amount of space available on root.
When the setup is in running container, all of a sudden writes to mongodb starts failing saying “No space left on device” and the mongo container exits.
The mongo container is created from mongo:3.4.10 image from docker hub.
The output of docker version -
docker version
Client:
Version: 17.12.1-ce
API version: 1.35
Go version: go1.9.4
Git commit: 7390fc6
Built: Tue Feb 27 22:17:40 2018
OS/Arch: linux/amd64
Server:
Engine:
Version: 17.12.1-ce
API version: 1.35 (minimum version 1.12)
Go version: go1.9.4
Git commit: 7390fc6
Built: Tue Feb 27 22:16:13 2018
OS/Arch: linux/amd64
Experimental: false
The output for docker info -
docker info
Containers: 22
Running: 22
Paused: 0
Stopped: 0
Images: 17
Server Version: 17.12.1-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 128
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9b55aab90508bd389d7654c4baf173a981477d55
runc version: 9f9c96235cc97674e935002fc3d78361b696a69e
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-87-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.67GiB
Name: tbbl09vm08
ID: LR5W:2HT2:CSXA:2TCT:BH5C:KZZZ:32ES:ZXIO:BTYK:WHQ3:DBIZ:SWF3
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
We hits this issue every 4-5 days and after several restart of the system it goes away.
How is the space being assigned to the container? How can we know what is causing it to die due to space not available.
Is there any way, by which we could assign space to be used to a container? So that it never runs out of space.