I am very new to docker, so my question might appear silly.
We are running docker on openshift and consuming container logs through fluentd in splunk.
Things work well, but recently we have started getting some concerns from the Splunk team related to the amount of logs being ingested per day, which has a cost impact on our purchased per day license from Splunk.
Now, I have seen multiple threads on this and the documentation here as well - https://docs.docker.com/config/containers/logging/configure/
which talks about the below config. settings
I have 3 questions on how to bring down unnecessary / low information value logs -
1- If we do reduce the max size and max files, what actually happens? for example the above code is equivalent to 3 rotations for 10mb size, what happens when a container logs more than this, for example if it logs say 1 gb? Does the logging stop after 30MB?
2- How does the trimming happen? will applying max file/size limitations impact removal of any useful information?
3- It is possible to leave docker logs as is and limit the logs ingestion in Splunk. Can someone give some pointers on strings / patterns , which I can search in the available Splunk logs and then suggest the Splunk team NOT to ingest those feeds from the container logging? Primarily, we are interested in the application level errors/warnings/ uids etc and not so much on the os/network logs
Once again, I apologize if the post seems very trival, I am very new to docker and am hazy on this.