Long lived `tail -f` on volume mounts

Docker Desktop
,
1

Thank you forums for previous help. That unblocked me to find this new question.

Once I have mounted my host’s /usr/local/ directory so that I can see it as /var/host in the container, I then want to do this (this is just for demo; I will actually consume this tail, but for now we print it to our terminal):

> tail -f /var/host/logs/myapp/debug.log

This dumps the debug.log file and keeps it open, waiting for new lines. Those lines will come from programs running on the host.

This works fine initially, but soon I notice that while the host-file: /usr/local/logs/myapp/debug.log is steadily growing, and tailing it from the host shows all new log lines, I notice that the the container side tail has stalled.

Then I can do this: If I open a shell on the same container and ls -l the directory:

docker exec -it 9e6214851 ls -l /var/host/logs/myapp/

then all new updates to the debug.log are pushed and become visible in the container. For now. But I need to repeat the ls -l for new updates.

Can someone please suggest the flags I need to add to the -v mount command to make the filesystem be “live” ?

thanks

2

Is it safe to assume that you use a bind volume, where a host path ins mounted into a container?
If this is the case than it can’t be “more live” than it currently is, as docker looks up the inode (~=pysical loction) and bind mounts the current inode into the container path. As a result the host path and container path are both accessing the same physical location. Just to be clear: there is no buffer, no cache or anything in between.

3

Thanks for the explanation, @meyay, that makes sense about how bind mounts work. However, it’s still odd that tail -f inside the container stops updating until I run ls -l. Could this be related to file system event propagation between macOS and Docker? I’ve read that inotify events can sometimes be missed or delayed on mounted volumes, especially with Docker Desktop on macOS. Just wondering if this is a known limitation or if there’s a workaround to keep tail -f responsive.

4

@kardoc326 thank you for the reminder that it’s about Docker Desktop, I missed that very relevant part.

What I wrote in my last response is only true for docker-ce on Linux.

With Docker Desktop, the Docker Engine always runs in a utility vm with it’s on Linux kernel. So there can’t be any kernel event propagation between the host’s kernel and utility vm’s kernel. I am not sure which protocol is used on macOS to mount the host folders into the utility vm, but what I wrote in my last post, does not apply to Docker Desktop.

I can imagine the Docker Desktop feature Syncronized file shares might overcome the sync problem. The feature is only available with a paid Pro, Team or Business subscription.

5

I tried all three virtualization modes (Docker VMM, Apple virtualization, Qemu) and I also changed the image store to containerd, but nothing changed the mount behavior. Logs appeared a little later in the container, but it was less then a second delay.

When I first tried, I accidentally used echo "hello" > log.txt, instead of echo "hello" >> log.txt" so I overwrote the first line in the log and tail in the container only showed “o” as a new line while tail on the macOS host showed the whole line I sent to the file.

Then I tried the Synchronized fileshare and I assumed it would not work, because I assumed the inodes would change when syncing happens, and it did. So in fact, the only case I reproduced not updating output of the tail command is when I used the synchronized fileshare., but running ls in another terminal didn’t help, it couldn’t as it would not change the inode that the tail command watches. It is not clear how and where you ran the ls command, but if you stopped the tail command in the same shell in the container, the next tail command would use the new inode which would explain the behavior.

If it is not your case, If you can share exact commands including what image you used , we can try to reproduce the issue your way, but if you have the fileshare enabled, you can disable it in the Docker Desktop settings, on the “Resources” tab, and there on the “File sharing” tab..

In addition to sharing the commands, please share what virtualization method you are using. It is in the settongs on the General tab.

The version of macOS and Docker Desktop can be important too.

6

I apologize for the incomplete bug report, and I want to thank for the time you all have invested. I will be more precise below:

MBPro: M4 Max w/ 36GB. Sequioa 15.5
Docker Desktop: Version 4.40.0 (187762), Engine: 28.0.4, Compose: v2.34.0-desktop.1, Credential Helper: v0.9.3, Kubernetes: v1.32.2
Virtualization: Apple vIrtualization, using rosetta, and virtioFS

experiment.

  1. This will create the log file and start the tail.
> sudo touch /var/log/johan.log
> sudo chmod ugo+w /var/log/johan.log
> echo line 1 >> /var/log/johan.log
> echo line 2 >> /var/log/johan.log
> docker run --rm -it -v /var/log/:/var/hostlog:ro debian:bookworm tail -f /var/hostlog/johan.log
line 1
line 2

I leave the docker run running, and I will report any updates to the tail -f

  1. I also tail the file in another host window: (note the path /var/log/johan.log (host) vs /var/hostlog/johan.log (container)). and leave this running.
> tail -f /var/log/johan.log
line 1
line 2
  1. In a third window, I will add some lines to the host log file.
> echo "line 3" >> /var/log/johan.log
> echo "line 4" >> /var/log/johan.log

These were immediately reported by both the host and container tails. HRM.

  1. Adding some more lines, but without closing the file between each (so like a logger)
> cat >> /var/log/johan.log
line5
lines 6
these are shown in the host, but not the container

So here we see the difference. When catting lines but without closing the file, the host tail -f immediately echoes each line as it is ENTERed (I mean pressing enter), and presumably flushed by some part of the OS / cat. But the container tail -f sees nothing until I exit the cat, and the file is closed. Then it sees all the lines added by the cat. (“line5 … not the container”)

So it seems like whatever mechanism docker deskop uses to propagate files will notice “file close” but not “flush”. And presumably the ls -l somehow emits a similar event.

I wonder if this is bug, something I can fix with better usage pattern, or working as intended

hope that helps!

7

Thank you for the detailed guide to reproduce the issue. This way I managed to do it. The difference was that I always used just echo, never tried streaming data with the cat command without closing the file.

So this way I tried all the virtualization methods again and this is what I found:

  • Regardless of the virtualization method, when I used cat, the tail output in the container was not updated
  • Apple Virtualization:
    I could indeed force the the update by running the ls command which probably forced filesystem syncing, but I could do it also using the sync command:
    sync -d logs.txt
    I tried sync -f too, but that didn’t help.
  • Docker VMM (beta):
    The sync command and the ls command didn’t help at all.
  • Qemu (Legacy)
    • gRPC fuse
      The same as Docker VMM. Nothing updated the file
    • osxf (Legacy)
      The file and the tail output was updated normally without issues.

So it looks like the legacy ways handled file changes better, but I guess the new ways use more cache to be faster and files are updated only when they are closed on the host (Docker VMM, Qemu gRPC fuse) or when forcing the sync in the container (Apple virtualization).

I don’t know how it can be fixed and still keep the improved performance, but you can report this on GitHub if you want

I don’t have a use case wher it causes trouble to me, but if you explain yours in the issue, the developers can prioritize it.

I also moved the topic to the Docker Desktop category, because this is definitely a Docker Desktop issue.

update:

Please, if you create an issue on GitHub, share the link here too so other users can follow it and join the conversation.

3 Likes
8

Thank you. I will test out the legacy virtualization vs running a sync process in a loop.

The use case is for developer development to collect log files for observabilty and report them to a central repostory for developing (both the observability log processing and also using those logs for local debugging).

So this is a situation where we can use some workarounds and also accept non-prod performance levels.

Thanks again!