Docker not starting container, not enough space

Hello,

I have Icinga fragmentated on different containers…and I have a Postfix container that works with Icinga for send alert e-mails.

A few days ago the e-mail stopped to be sent, so I checked the state of docker containers and 5 containers were runnng…icinga core, icinga web, icinga php, mariaDB and postfix!

I tried to use the “docker-compose up -d” command for restart containers and it seems to be all ok…but the mariadb was wrong! It seems to be started for 1/2 minutes and then it become "Exited (139)…this is the MariaDB container

Some informations I found (I cannot copy and paste logs because of my remote connection to the server)…

with “du -d1 -h /var7lib/docker/containers | sort -h” I see directories of 40K, 44K, 64K, 112k, 220K ecc ecc…some of this directories refer to some containers exited months ago…

with “docker logs my_container_ID_exited_139” I get a lot of:

[error] InnoDB: preallocating 229376 bytes for file ./icinga2idomysql/icinga_confinfo.ibd failed with error 28
and the last raw of logs is:
[error] [fatal] InnoDB: Out ofspace during rollback. Consider increasing your tablespace.

I have some interesting logs about Postfix too!
Here some lines of logs that I think that can be interesting:

postfix/smtpd[133].warning: not enough space in mail queue: 0 bytes < 1.5]message size limit

postfix/cleanup[134]: warning: 9776G19B7ED: write queue file: No space left on device

and if I use the “df” command I see that are full (used 100%) the following paths:

/ev/mapper/icingasrv—vg-root
overlay /var/lib/docker/overlay2/bvqeygre7rvvbytygtgy5…
overlay /var/lib/docker/overlay2/hrthrthrt…
overlay /var/lib/docker/overlay2/yjteyjktku…
overlay /var/lib/docker/overlay2/rkukkuy…

What can I do? I have the possibility do add a few disk space to the VM but I think that it cannot be resolutive…how can I free some space without affecting te production?

I browsed some forums, my Docker book ecc ecc but I am really scared tinking that I can affect production doing something wrong!

thanks

Giulia

Make sure /var/lib/docker has enough capacity and make sure to use volumes or volume binds to persist the application data outside your containers.

ok so if I don’t erase the volume I cannot damage containers? How can I remove some old files for make more space?

Thank you

It is impossible to predict if data corruption/inconsistency didn’t take place already due lack of free space.
It would be even worse, if docker would be operated as a cluster. It pretty much depends on how you applications if there is a data loss or not.

Without doing any further harm, you can execute docker system prune -a to get rid of unused images, left over volumes, networks and stopped(!) containers (warning: this will delete all not running containers). Look for invalid logs find /var/lib/docker/containers/ -name *-json.log -exec bash -c "jq empty {} || echo invalid json : {}" \; and delete those. Additionaly you can check if any log files are bloated ls -lh /var/lib/docker/containers/*/*-json.log and delete those as well.

It should free up enough space to safely shutdown your vm, add more disk capacity and restart your system.

Update:
Why would you be afraid of loosing containers? They are ment to be disposable… It is ment to store persistant state of a container in folders that are mapped to a volume or bind-volume that stores the data on the local host or a remote share. If used like that there is no harm in loosing containers. Just recreate them with the same paramaters and be good (hint: I do hope you use docker-compose.yml to manage your containers)

Loosing data when a container is removed is a good indicator that docker is used in a way it is not intended to.