Hello,
I have Icinga fragmentated on different containers…and I have a Postfix container that works with Icinga for send alert e-mails.
A few days ago the e-mail stopped to be sent, so I checked the state of docker containers and 5 containers were runnng…icinga core, icinga web, icinga php, mariaDB and postfix!
I tried to use the “docker-compose up -d” command for restart containers and it seems to be all ok…but the mariadb was wrong! It seems to be started for 1/2 minutes and then it become "Exited (139)…this is the MariaDB container
Some informations I found (I cannot copy and paste logs because of my remote connection to the server)…
with “du -d1 -h /var7lib/docker/containers | sort -h” I see directories of 40K, 44K, 64K, 112k, 220K ecc ecc…some of this directories refer to some containers exited months ago…
with “docker logs my_container_ID_exited_139” I get a lot of:
[error] InnoDB: preallocating 229376 bytes for file ./icinga2idomysql/icinga_confinfo.ibd failed with error 28
and the last raw of logs is:
[error] [fatal] InnoDB: Out ofspace during rollback. Consider increasing your tablespace.
I have some interesting logs about Postfix too!
Here some lines of logs that I think that can be interesting:
postfix/smtpd[133].warning: not enough space in mail queue: 0 bytes < 1.5]message size limit
…
postfix/cleanup[134]: warning: 9776G19B7ED: write queue file: No space left on device
and if I use the “df” command I see that are full (used 100%) the following paths:
/ev/mapper/icingasrv—vg-root
overlay /var/lib/docker/overlay2/bvqeygre7rvvbytygtgy5…
overlay /var/lib/docker/overlay2/hrthrthrt…
overlay /var/lib/docker/overlay2/yjteyjktku…
overlay /var/lib/docker/overlay2/rkukkuy…
What can I do? I have the possibility do add a few disk space to the VM but I think that it cannot be resolutive…how can I free some space without affecting te production?
I browsed some forums, my Docker book ecc ecc but I am really scared tinking that I can affect production doing something wrong!
thanks
Giulia