Hey all !
my issue started when kong started with docker compose with nginx consul and konga, crashed.
I decided to do a docker-compose down to stop and remove this containers here the problem started.
I’ve got the errors messages
"Error response from daemon: driver “devicemapper” failed to remove root filesystem …“
I removed the images and did a volume prune but still I wasn’t able to remove any of the container started with the docker-compose file.
I thus decided to stop the docker service (I did so while other forgotten container were running).
Now I am not able to start it again I get
"Job for docker.service failed because the control process exited with error code. See “systemctl status docker.service” and “journalctl -xe” for details.”
systemctl status docker.service gives me:
docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/docker.service.d
└─http-proxy.conf
Active: failed (Result: start-limit) since Mon 2017-10-30 12:13:54 CET; 17s ago
Docs: https://docs.docker.com
Process: 20211 ExecStart=/usr/bin/dockerd (code=exited, status=1/FAILURE)
Main PID: 20211 (code=exited, status=1/FAILURE)
Memory: 144.0M
CGroup: /system.slice/docker.service
systemd[1]: Failed to start Docker Application Container Engine.
systemd[1]: Unit docker.service entered failed state.
systemd[1]: docker.service failed.
systemd[1]: docker.service holdoff time over, scheduling restart.
systemd[1]: start request repeated too quickly for docker.service
systemd[1]: Failed to start Docker Application Container Engine.
systemd[1]: Unit docker.service entered failed state.
systemd[1]: docker.service failed.
and journalctl -xe just repeat:
kernel: device-mapper: block manager: btree_node validator check failed for block 2689
kernel: device-mapper: btree spine: node_check failed: blocknr 0 != wanted 2689
kernel: device-mapper: block manager: btree_node validator check failed for block 2689
kernel: XFS (dm-13): metadata I/O error: block 0xb40002 (“xfs_buf_iodone_callbacks”) error 5 numblks 1
kernel: node_check: 1326 callbacks suppressed
I found out that /var was full the main space was occupied by /var/lib/docker/devicemapper/devicemapper/data
I am using deviecmapper, I read it may lead to corruption what do you think about it ?
I have no idea on what to do to start it back except reboot the system but the system is a server and I can’t easily reboot it
the OS is Centos 7
Have a good day !