Gelf driver poor performance (locks container)?

General
1

Hello Guys
We have an application that logs a lot to stdout and suddenly began to lock (using 300% cpu the docker daemon)
we did some testing running
seq 10000000000000

  1. using json logging takes like 30 secs
  2. using the gelf driver and sending logs to a container running logstash on 127.0.0.1 on the host
    it can take easily one hour.
    we checked the performance of logstash and the problem is not there, the cpu and memory are barely use, the messages are arriving but very slowly. Also if just stop the logstash container the performance is just the same
    We disabled compression and that showed some improvement but still super slow.

is this the expected behavior? I mean we are sure that the gelf driver cannot handle 700 lines/sec just a couple of characters.
What can we do to improve this situation?
P.D: logging is 15X faster using syslog driver (exactly same setup), so there are no issues with our setup… I belive

Thanks

2

You could try putting the GELF logging into your app, instead of logging to stdout and using the docker GELF logging driver. You might get better performance.

I use log4j2 which provides a very easy way to configure GELF. I provide the container with the graylog server via an environment variable. I haven’t done performance testing, however.

3

Probably related to

I don’t know when this will be fixed. Handling this in the application seems like sound advice for now.