Docker not starting container, not enough space


I have Icinga fragmentated on different containers…and I have a Postfix container that works with Icinga for send alert e-mails.

A few days ago the e-mail stopped to be sent, so I checked the state of docker containers and 5 containers were runnng…icinga core, icinga web, icinga php, mariaDB and postfix!

I tried to use the “docker-compose up -d” command for restart containers and it seems to be all ok…but the mariadb was wrong! It seems to be started for 1/2 minutes and then it become "Exited (139)…this is the MariaDB container

Some informations I found (I cannot copy and paste logs because of my remote connection to the server)…

with “du -d1 -h /var7lib/docker/containers | sort -h” I see directories of 40K, 44K, 64K, 112k, 220K ecc ecc…some of this directories refer to some containers exited months ago…

with “docker logs my_container_ID_exited_139” I get a lot of:

[error] InnoDB: preallocating 229376 bytes for file ./icinga2idomysql/icinga_confinfo.ibd failed with error 28
and the last raw of logs is:
[error] [fatal] InnoDB: Out ofspace during rollback. Consider increasing your tablespace.

I have some interesting logs about Postfix too!
Here some lines of logs that I think that can be interesting:

postfix/smtpd[133].warning: not enough space in mail queue: 0 bytes < 1.5]message size limit

postfix/cleanup[134]: warning: 9776G19B7ED: write queue file: No space left on device

and if I use the “df” command I see that are full (used 100%) the following paths:

overlay /var/lib/docker/overlay2/bvqeygre7rvvbytygtgy5…
overlay /var/lib/docker/overlay2/hrthrthrt…
overlay /var/lib/docker/overlay2/yjteyjktku…
overlay /var/lib/docker/overlay2/rkukkuy…

What can I do? I have the possibility do add a few disk space to the VM but I think that it cannot be resolutive…how can I free some space without affecting te production?

I browsed some forums, my Docker book ecc ecc but I am really scared tinking that I can affect production doing something wrong!



Make sure /var/lib/docker has enough capacity and make sure to use volumes or volume binds to persist the application data outside your containers.

ok so if I don’t erase the volume I cannot damage containers? How can I remove some old files for make more space?

Thank you

It is impossible to predict if data corruption/inconsistency didn’t take place already due lack of free space.
It would be even worse, if docker would be operated as a cluster. It pretty much depends on how you applications if there is a data loss or not.

Without doing any further harm, you can execute docker system prune -a to get rid of unused images, left over volumes, networks and stopped(!) containers (warning: this will delete all not running containers). Look for invalid logs find /var/lib/docker/containers/ -name *-json.log -exec bash -c "jq empty {} || echo invalid json : {}" \; and delete those. Additionaly you can check if any log files are bloated ls -lh /var/lib/docker/containers/*/*-json.log and delete those as well.

It should free up enough space to safely shutdown your vm, add more disk capacity and restart your system.

Why would you be afraid of loosing containers? They are ment to be disposable… It is ment to store persistant state of a container in folders that are mapped to a volume or bind-volume that stores the data on the local host or a remote share. If used like that there is no harm in loosing containers. Just recreate them with the same paramaters and be good (hint: I do hope you use docker-compose.yml to manage your containers)

Loosing data when a container is removed is a good indicator that docker is used in a way it is not intended to.