Log rotation best practice?

General
1
  • We’re following 12 factor microservice practice.
  • Many of our containers are long running and need daily log-rotation.
  • We set our containers to do json logging.
  • We’re using thinpool lvm for docker storage but we get corruption with overlay fs as well.

Is it really safe/sane to reach into /var/lib/docker/containers and move json log files with no docker interaction OR should a docker restart/reload be done at the same time.
We get a lot ‘failed to find device path’ from lsblk/devicemapper if we let logrotate rotate logs without stopping containers. It seems that docker-proxy maintains an open file handle wiring the container’s console/stdout/stderr to the log files in /va/log/docker/containers, correct?
If so, what is the proper way to close it to allow logs to rotated? If not, it must be possible to create a race condition between container stdout/stderr and (re)move the backing json file, right? Or is syslog/UDP the only non-blocking, safe, eternal way to log?

Anyhow, I find a lot of references that use logrotate but that specifically causes us problems.

Please advise.
Chris

1 Like
2

+1

I for one ship every line of logging out of the container, but I would like to hear what people here are doing and what are the best practices.

3

Today I m using a mount volume for application log. I will test fluentd soon

4

I am a big fan of syslog/udp wherever possible. I haven’t gotten a chance to try it out yet, but I like the idea of the gelf log format as well. It also uses udp, but appears to play nicer with things like multiline logs (stacktraces)