docker-compose logs dies with "no such file or directory" due to log size even though logs should rotate

General
,
1

I have a couple of docker containers I run and then I run this command which writes all their output to a file:

docker-compose -f docker-compose.yml logs -f -t &> output.log &

The docker-compose.yml looks like this:

version: '3.7'

services:
  pm-a:
    # ...
    environment:
      PYTHONUNBUFFERED: 1
      # ...
    logging:
      driver: "json-file"
      options:
        max-size: "200k"
        max-file: "10"
        mode: "non-blocking"
 # Other similarly configured services...

The problem is that after a while, without killing any of the containers or anything, I find that the logs command died, and the last line in output.log is:

error from daemon in stream: Error grabbing logs: no such file or directory

After experimenting a lot I found that this is because of the logs size. If I increase the logs size the issue takes longer to reproduce, and if I log a bunch of things the issue reproduces more quickly. Now, from my understanding, the logs should rotate, meaning docker-compose logs is supposed to live indefinitely even if the max size of the logs is reached. I also tried inspecting the underlying log files with:

docker inspect --format='{{.LogPath}}'

And I found that the logs have often already reached their max size long before the logs command died, meaning they were able to rotate for a while before it suddenly became an issue.

I’ve tried a bunch of things, including using different logging drivers (such as local), and setting a high COMPOSE_HTTP_TIMEOUT, and googling a bunch, with no luck.

The output of docker --version:

Docker version 20.10.13, build a224086

So my questions are:

  1. Am I wrong in my understanding of what “logs rotation” means? Is it actually expected for the logs command to die when the max size is reached?
  2. Since I’m not well versed with dockers, I wonder if there are things I can do to better debug this, like for instance get more verbose output on why the logs command died.
  3. Of course if anyone just knows how to solve this it will be best!

Thanks

2

Hi,

My best guess is that it rotates the file, and then the follow commands loses its pointer to the old file, and then crashes/ends, this is also the normal behaviour of linux tail command.

I dont think there is a solution to this, but im a bit curious about what your purpose of doing it this way is? Maybe if you can come with some more information we can maybe see if there are better ways of doing what you’re trying to achieve