File system watch does not work with mounted volumes

pmario, do you have a link that documents this is the case? As of samba 3.0.25 inotify events should be supported: https://wiki.samba.org/index.php/Samba_3.0_Features_added/changed

I don’t think the relevant code is in the Linux kernel any longer:

Hi Michael,
In May I was just guessing, how the stuff works, from what I could see, since there was (still is) no accessible source code. https://youtu.be/7da-B3rY9V4?t=2094 here’s the proof. Windows is different :slight_smile:

Have there been any developments on this front? Seems like a pretty critical feature missing from windows for this long.

Sorry, no progress to report. Docker for Windows relies on SMB/CIFS support in Linux, and since propagating filesystem notifications is not supported our options are limited.

I made a really gross hack which just scans a directory of files recursively, over and over, looking at the modified times of files and if they changed since the last scan it fires an event. The file system events don’t fire but the files and their metadata do change successfully, so this technique can work. It’s pretty poor performance but for a dev-time only script it’s a lot better than dockering down then back up just to pickup file changes.

It’s a simple nodejs/gulp script if anyone wants it let me know and I’ll share it.

What programming language and framework are you using? Most dev web servers will have fallback mode that resort to filesystem polling in cases like this.

I have developed the script that sits on Windows host, watches changes in directories mounted by Docker containers and notifies containers once file change occurs. The script is available as pip package (PyPI)

pip install docker-windows-volume-watcher

Then you can just run the script without any arguments:

docker-volume-watcher

The script will inspect all running containers and start notifying containers about changes in mounted directories. The script will also listen container start/stop events and update the list of watched directories.

You can read detailed description of this script in this blog post.

4 Likes

docker-volume-watcher works great for me! It would be great having it integrated inside Docker for Windows. Mikhail, thank you very much for developing this util.

I would love to try your script, has to be better than manually rebuilding all the time. For some reason @merofeev solution just gives me an error.

I used this in gulp4 to work around the issue.

Usage:

import gulp from 'gulp'
import gls from 'gulp-live-server'
import { build } from './build'
import { scan } from './ghetto-watch'

let _server = gls(['-r', 'babel-register', 'index.js'])

function start (callback) {
  _server.start()
  callback()
}

function stop (callback) {
  _server.stop()
  if (callback) callback()
}

function notify (callback) {
  _server.notify()
  callback()
}

function watch () {
  let src = [
    '*.js',
    'api/**/*',
    'site/**/*'
  ]

  let scanner = scan(src)
  let full = gulp.series(
    scanner.stop.bind(scanner),
    build,
    notify,
    stop,
    start,
    scanner.start.bind(scanner)
  )

  scanner.on('changed', (changed) => {
    console.log('changed:', changed)
    full()
  })
  scanner.start()
}

const run = gulp.series(build, start, watch)
gulp.task('start', run)
gulp.task('default', run)
// ghetto-watch.js
import fs from 'fs'
import path from 'path'
import map from 'async/map'
import mm from 'micromatch'
import { EventEmitter } from 'events'

// This should not exist but docker volumes for windows do not trigger file system events
// So instead I scan all the files looking for modify time changes and manually trigger events

let cache = {}
let token = null

function flatten(arr) {
  return arr.reduce(function (flat, toFlatten) {
    return flat.concat(Array.isArray(toFlatten) ? flatten(toFlatten) : toFlatten);
  }, []);
}

function scanFiles (file, callback) {
  fs.lstat(file, (err, stat) => {
    if (err) return callback(err)
    if (stat.isDirectory()) {
      let dir = file
      fs.readdir(dir, (err, files) => {
        if (err) return callback(err)
        let resolved = files.map(f => path.join(dir, f))
        map(resolved, scanFiles, (err, results) => {
          if (err) return callback(err)
          let changed = flatten(results.filter(f => f))
          callback(null, changed)
        })
      })
    } else if (stat.isFile()) {
      let mtime = stat.mtime.getTime()
      let changed = cache[file] != null && cache[file] !== mtime
      cache[file] = mtime
      callback(null, changed ? file : null)
    } else {
      callback()
    }
  })
}

export class Scanner extends EventEmitter {
  constructor (src) {
    super()
    this.running = false
    this.src = src.map(p => path.resolve(p))
  }

  scan (file, callback) {
    if (!this.running) return callback()
    fs.lstat(file, (err, stat) => {
      if (err) return callback(err)
      if (stat.isDirectory()) {
        let dir = file
        fs.readdir(dir, (err, files) => {
          if (err) return callback(err)
          let resolved = files.map(f => path.join(dir, f))
          map(resolved, scanFiles, (err, results) => {
            if (err) return callback(err)
            let changed = flatten(results.filter(f => f))
            callback(null, changed)
          })
        })
      } else if (stat.isFile()) {
        let mtime = stat.mtime.getTime()
        let changed = cache[file] != null && cache[file] !== mtime
        cache[file] = mtime
        callback(null, changed ? file : null)
      } else {
        callback()
      }
    })
  }

  run () {
    if (this.running) {
      this.scan(process.cwd(), (err, changed) => {
        if (err) console.error(err)
        if (mm(changed, this.src).length) this.emit('changed') // a change matching src glob happened!
        setTimeout(() => this.run(), 0)
      })
    }
  }

  start (callback) {
    if (!this.running) {
      this.running = true
      this.run()
    }
    if (callback) callback()
  }

  stop (callback) {
    this.running = false
    if (callback) callback()
  }
}

export function scan (src) {
  return new Scanner(src)
}

Thank you, will see if I get it to work for me :slight_smile:

it’s work for me, thx!

Hi,

Is there any plan to mock this feature, by implementing something like docker-windows-volume-watcher inside docker for windows ?

That’s a pretty big deal for developers

Edit: Apparently LCOW could solve this issue, any news about that ?

how come this isn’t a show stopper?

1 Like

It seems nobody gives a sh*t

Hello, I definitely give a sh*t. However, this person’s python script has saved my bacon: https://github.com/merofeev/docker-windows-volume-watcher

It basically runs in the background and propagates all mounted volume filesystem changes to your running docker containers.

1 Like

Is anyone able to verify if WSL2 solves this problem?

1 Like

Awesome work, thank you @merofeev, you’re a hero! I can confirm that docker-volume-watcher (github) works perfectly on Windows 10 (Docker Toolbox w/ VirtualBox) with an alpine docker container (node:6.17-alpine in my case). This is the best solution; use native inotify file notification events in Linux and then ensure your libraries or build scripts (e.g. gulp) are not using polling at all. The end result is essentially 0% CPU usage vs. 30% CPU load (I’ve got a large number of files to monitor).

Note: From my research, I found that if you’re using grunt (with grunt-watch-contrib like I am) it will not appear to work, because no matter what, it’ll actually be using polling via node’s older fs.watchFile() API. So ensure you’re using a library like chokidar and after it indexes all the files you want to watch, it should sit fully idle and docker-volume-watcher will dutifully notify chokidar in your container using inotify!

EDIT: There does appear to be one limitation, which is that docker-volume-watcher isn’t capable of notifying about file deletion, which could be a potential problem for some. I’ve added an issue about that here: https://github.com/merofeev/docker-windows-volume-watcher/issues/16

I get an error when I try to run docker-volume-watcher

ImportError: No module named pywintypes

And going down the google hole has gotten me nowhere. Every suggestion is dated from 2017 and nothing has worked.

Any suggestions?