Our company has a variety of dockerized services and applications. Currently, we use the syslog driver to log them. We are in the process of setting up a fluentd-elasticsearch-kibana stack to centralize our logging, and we’d like to use the fluentd logging driver to feed into that.
However, among the containers we wish to log in this way are the fluentd, elasticsearch, and kibana containers themselves. While we certainly want them to be logged to fluentd, we need to also have a copy of their logs being recorded to disk locally, as a backup; if there is an error in the FEK stack, we need to be able to see its logs without using the FEK stack.
How do we do this? How do we use the fluentd logging driver, while also saving the logs to disk in some way?
It’s a bit unusual of a request, I don’t know that support for a “log tee” like you mention can be accomplished first class today. However here is one idea.
Let’s say we are configuring logging for fluentd itself (this should be generally applicable to the other two as well).
We could have a docker volume which gets mounted into the fluentd container, and we configure fluentd container to log to disk in this volume.
Then, make another container which shares the Docker volume and whose ENTRYPOINT is tail -f on the fluentd log file(s). This container is configured to use the fluentd log driver and will send its output to fluentd for processing. Boom, fluentd logs routed to fluentd.
With this method, you have an on-disk version in the docker volume, and also a little shim which will send the logs to fluentd itself. You could do the same for ES and for Kibana. Just be careful of infinite loops, e.g. fluentd logging that it received logs from itself and spiraling out of control.