We have space issue on our hosts due to docker daemon repeatedly deletes latest file from set of rotated files without closing it.
Docker version 1.10.3
Red Hat Enterprise Linux Server release 7.2 (Maipo)
lsof -nP +L1
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
ntpd 803 ntp 4r REG 253,0 10406312 0 50614280 /var/lib/sss/mc/initgroups (deleted)
avahi-dae 849 avahi 5r REG 253,0 10406312 0 50614280 /var/lib/sss/mc/initgroups (deleted)
avahi-dae 866 avahi 5r REG 253,0 10406312 0 50614280 /var/lib/sss/mc/initgroups (deleted)
sssd 903 root 17r REG 253,0 10406312 0 50614280 /var/lib/sss/mc/initgroups (deleted)
sssd_be 925 root 19r REG 253,0 10406312 0 50614280 /var/lib/sss/mc/initgroups (deleted)
docker-cu 8342 root 16r REG 253,2 134591072 0 161192 /apps/docker/containers/afa2bda9817a622aafb54541b4bdf94196237e7f3b935541e30ad746c7948619/afa2bda9817a622aafb54541b4bdf94196237e7f3b935541e30ad746c7948619-json.log.4 (deleted)
docker-cu 8342 root 44r REG 253,2 134591123 0 157584 /apps/docker/containers/afa2bda9817a622aafb54541b4bdf94196237e7f3b935541e30ad746c7948619/afa2bda9817a622aafb54541b4bdf94196237e7f3b935541e30ad746c7948619-json.log.4 (deleted)
docker-cu 8342 root 88r REG 253,2 134591511 0 27639 /apps/docker/containers/afa2bda9817a622aafb54541b4bdf94196237e7f3b935541e30ad746c7948619/afa2bda9817a622aafb54541b4bdf94196237e7f3b935541e30ad746c7948619-json.log.4 (deleted)
docker-cu 8342 root 90r REG 253,2 134517765 0 27598 /apps/docker/containers/afa2bda9817a622aafb54541b4bdf94196237e7f3b935541e30ad746c7948619/afa2bda9817a622aafb54541b4bdf94196237e7f3b935541e30ad746c7948619-json.log.4 (deleted)
… >5000 more records for deleted file afa2bda9817a622aafb54541b4bdf94196237e7f3b935541e30ad746c7948619-json.log.4 with the same PID (Docker daemon) but with different File Descriptors. That means that file was open, then deleted, then created new one with the same name and so on without closing files before deleting.
There is our setting of log rotation:
ps aux|grep 8342
root 8342 8.3 1.0 4175376 163756 ? Ssl Aug09 831:15 /usr/bin/docker-current daemon --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd --log-driver=json-file --log-opt max-size=10m --log-opt max-file=5 --storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/rhel-docker–pool --storage-opt dm.use_deferred_removal=true --storage-opt dm.use_deferred_deletion=true
I would appreciate any advice.
Best regards,
Valentyn Sigal