Docker Community Forums

Share and learn in the Docker community.

Files on the host machine have changed, but the files inside the container are not

I have configured FileBeat on the ELK stack using docker.

  1. On the host machine, elasticsearch, logstash, kibana, and filebeat create images and run containers through docker-compose settings.

  2. HostMachine’s /sample_logs path is mounted on filebeat’s /var/log/server path, and filebeat collects files in the /var/log/server path and delivers them to logstash.

[Problem that occurred]
When filebeat is run as Docker, the files in /sample_logs are normally reflected in /var/log/server in the filebeat container and sent to logstash.
However, even if the files in sample_logs are modified on the host machine afterward, the files in the filebeat container are not changed.

The confirmed phenomenon is as follows.

  1. In Host Machine, echo ‘blar blar…’ >> app.log or chmod 755 app.log is reflected normally when directly modifying the file.

  2. The above phenomenon occurs for files being modified by applications on the Host Machine. (ex: Log file generated by the application when the SpringBoot application is executed with IntelliJ)

The configuration file is placed in the path below.

I need help, Thank you.

In your post you mentioned /sample_logs, however, the docker-compose.yml contains ./sample_logs. Are you sure you write the right folder on the host?

Hello, First, Thank you for your reply.

In this post, /sample_logs was specified, but it was actually coded as ./sample_logs.
All other contents are the same as the main text. However, one thing that has been confirmed is as follows.

  1. Run IntelliJ
  2. Running web applications
    → Create LogFile in sample_logs and start logging
  3. Run ELK and FileBeat container (LogFile is mounted as specified in docker-compose.yml)
    → When executed, the contents of the current logfile are transferred to Logstash
  4. LogFile is additionally recorded in the web application, but the LogFile in FileBeat is not changed, so it is not sent to LogStash.
    5 In the Host Machine terminal, in LogFile
    echo ‘new line…’ >> app.log

When a new line is manually added by inputting a command, FileBeat responds and delivers the contents from offset to the present to LogStash when first executed.

This was expected because the app.log file is managed by the IDE, but I ran FileBeat on the host machine instead of docker.
Works normally when running in the background (nohup filebeat -e -c filebeat.yml after downloading FileBeat binary source).

That’s why we expect that there is a problem with docker.

What setting could be wrong…?

I didn’t realize that the GitHub repository was created by you. I understand now.

I had similar experience only when I mounted a file and not a directory as you did, but in my case the file was changed on the host and not in the container. At least I thought the file was changed on the host but in fact, the file was recreated so it got a new inode on the filesystem with the same name but pointing to a different file. So I decided to mount its parent directory as you id.

When you append a line to a file using >> it writes directly to the end of the file and doesn’t create a temporary file to save it later using the original name.

So it is strange, but I would check those inodes if there is any difference.

ls -i app.log

You can also check the inode of the folder
ls -id ./sample_logs

The answer to my problem was in a surprisingly absurd place. That was the setup in Docker on Mac.
The General settings of Docker on Mac have the following items.

Use gRPC FUSE for file sharing
(Uncheck to use the legacy osxfs file sharing instead.)

After unchecking this item, it was confirmed that the mounted host file and the container file are synchronized normally.

It was the answer I had been looking for for over a week, but the situation is more absurd than cool. :rofl:

Thank you for trying to solve my problem together. :grinning:

1 Like