We have an application that logs a lot to stdout and suddenly began to lock (using 300% cpu the docker daemon)
we did some testing running
- using json logging takes like 30 secs
- using the gelf driver and sending logs to a container running logstash on 127.0.0.1 on the host
it can take easily one hour.
we checked the performance of logstash and the problem is not there, the cpu and memory are barely use, the messages are arriving but very slowly. Also if just stop the logstash container the performance is just the same
We disabled compression and that showed some improvement but still super slow.
is this the expected behavior? I mean we are sure that the gelf driver cannot handle 700 lines/sec just a couple of characters.
What can we do to improve this situation?
P.D: logging is 15X faster using syslog driver (exactly same setup), so there are no issues with our setup… I belive