- We’re following 12 factor microservice practice.
- Many of our containers are long running and need daily log-rotation.
- We set our containers to do json logging.
- We’re using thinpool lvm for docker storage but we get corruption with overlay fs as well.
Is it really safe/sane to reach into /var/lib/docker/containers and move json log files with no docker interaction OR should a
docker restart/reload be done at the same time.
We get a lot ‘failed to find device path’ from lsblk/devicemapper if we let logrotate rotate logs without stopping containers. It seems that docker-proxy maintains an open file handle wiring the container’s console/stdout/stderr to the log files in /va/log/docker/containers, correct?
If so, what is the proper way to close it to allow logs to rotated? If not, it must be possible to create a race condition between container stdout/stderr and (re)move the backing json file, right? Or is syslog/UDP the only non-blocking, safe, eternal way to log?
Anyhow, I find a lot of references that use logrotate but that specifically causes us problems.