Docker-compose volumes, gulp watch, 100% CPU usage

Docker Desktop
1

Expected behaviour

When running gulp watch inside of a container with a volume mounted the CPU usage is minimal and updates are fast

Actual behaviour

The CPU is pegged at 100% and access to the files take a much longer than expected to edit

Information

OS X: version 10.11.4 (build: 15E65)
Docker.app: version v1.11.0-beta8.2
Running diagnostic tests:
[OK]      docker-cli
[OK]      Moby booted
[OK]      driver.amd64-linux
[OK]      vmnetd
[OK]      osxfs
[OK]      db
[OK]      slirp
[OK]      menubar
[OK]      environment
[OK]      Docker
[OK]      VT-x
Docker logs are being collected into /tmp/20160426-223203.tar.gz
Most specific failure is: No error was detected
Your unique id is: 9F2A2937-9B0B-4FCF-B426-23C6FA0F817F

Steps to reproduce the behavior

  1. Have a docker project with gulp and gulp watch watching a the directory for changes to src/**
  2. Have a volume mounted using docker-compose I.E. - .:/app/:
  3. docker-compose up
  4. Observe CPU pegged at 100%
1 Like
2

Are you able to get watch events triggered within the container from modifying files off of your local Mac FS though? I’m not even able to get that working.

3

Using the latest Docker for Mac beta and docker files below gulp is seeing the changes from my local machine but pegs my CPU to 100% while the container is running

Dockerfile:

FROM node:5

WORKDIR /app
COPY package.json package.json
COPY npm-shrinkwrap.json npm-shrinkwrap.json
RUN npm install gulp -g
RUN npm install
COPY ./ .

docker-compose.yml:

web:
  build: .
  command: gulp
  volumes:
    - .:/app/
    - /app/node_modules
4

@vinayh could you please detail what you are executing, what you expect to happen, and what actually occurs?

5

@jamesraybould Those files are very helpful, thanks. Could you share your gulpfile.js (or a part of it) and any other configuration you are using? Do you have any public projects where you observe this behavior? I am trying to replicate your issue but I don’t use npm/gulp/js so it is not straightforward for me to understand what I need to do to see the issue you are seeing.

6

Hi @dsheets

I’ve thrown together a quick example repo - https://github.com/jamesRaybould/docker-gulp

All you will need to do is run docker-compose up and it should peg your CPU at 100%, if you then change to the commented out command you will see it working as expected, but with no app reload.

7

Thanks! That’s really helpful. I can see now that gulp uses an old version of vinyl-fs which uses glob-watcher which uses an old version of gaze. I think glob-watcher expands the glob in its entirety and passes it to gaze which then uses fs.watchFile in node 5.10.1 to watch the files for changes. Unfortunately, the fs.watchFile function uses stat polling at a default interval of 5.007 seconds to track changes to the file system. This means that, with enough files, there is a nearly constant load on the shared file system and hypervisor as the file polling runs.

I think perhaps a project like gulp-watch https://github.com/floatdrop/gulp-watch could help you as it is supposed to correctly use something called chokidar which I hope actually registers inotify events.

We are actively working on improving the shared file system performance but I’m afraid that polling a large number of files in the file system at a frequent interval will always incur a significant computational overhead as at least 4 processes must handle the file system request in both directions (Linux kernel, hypervisor, file system proxy, OS X kernel).

I’m interested in how you get along with something like gulp-watch. Please keep us posted. :slight_smile:

Thanks for your help,

David

8

Thanks David that worked a treat. Also thanks for taking the time to explain the issue in such detail, it is massively appreciated!

I’ve updated the repo with gulp-watch installed in case anyone stumbles across a similar issue in the future.

1 Like
9

@dsheets sure thing - I apologize for the lack of info before. I’m using gulp-nodemon, which is a gulp wrapper for nodemon, which I believe uses inotify. I’m going to make a small repo and attach here similar to what @jamesraybould did.

10

It appears that it does work in isolation. Now I need to figure out why it’s not working with my setup.

11

Any luck on figuring out what was wrong in your setup? I’m having the same issue. The docker-nodemon app works just fine, restarts on file changes, but my actual app does not. I’ve tried many variations and can’t seem to get it to restart on file change…

12

Could either/both of you please run pinata diagnose -u and post the result here? There may be useful information about event notification failures in the logs.

Thanks,

David

13

@jrthib unfortunately have not yet. Right now we’re in a pretty intense work sprint, so I’ve fallen back to running my gulp build tasks manually and triggering server restarts by ssh-ing into the running container and saving the main server app file manually. :-\

Not ideal, but it’s working for now. Overall loving Docker Beta’s growth as a more mature platform. If I figure it out, I’ll let you guys know here right a way. Please do the same @jrthib.

@dsheets here ya go man. Feel free to let me know if you ever need me to upload my logs again. Or do something and reupload.

vinay@Vinays-MacBook-Pro ~> pinata diagnose -u
OS X: version 10.11.4 (build: 15E65)
Docker.app: version v1.11.0-beta9
Running diagnostic tests:
[OK]      docker-cli
[OK]      Moby booted
[OK]      driver.amd64-linux
[OK]      vmnetd
[OK]      osxfs
[OK]      db
[OK]      slirp
[OK]      menubar
[OK]      environment
[OK]      Docker
[OK]      VT-x
Docker logs are being collected into /tmp/20160501-151942.tar.gz
Most specific failure is: No error was detected
Your unique id is: DB968B29-02F7-450B-8B19-62201F32676A
Please quote this in all correspondence.
14

I was also having this problem using the following Docker for Mac details:

OS X: version 10.11.4 (build: 15E65)
Docker.app: version v1.11.1-beta12
Running diagnostic tests:
[OK]      Moby booted
[OK]      driver.amd64-linux
[OK]      vmnetd
[OK]      osxfs
[OK]      db
[OK]      slirp
[OK]      menubar
[OK]      environment
[OK]      Docker
[OK]      VT-x
Docker logs are being collected into /tmp/20160519-122810.tar.gz
Most specific failure is: No error was detected

I found that if I use gulp-watch (instead of gulp.watch) and set it to use a polling interval, it works. Using file system notifications, it doesn’t make the CPU crazy, but it just doesn’t work.

This is what worked for me:

var gulp = require('gulp'),
  watch = require('gulp-watch'),
  path = require('path');

gulp.task('watch', function(){
  watch(path.resolve('/source/_/scss/**/*.scss'), { usePolling: true, interval: 2000 }, function(file) {
      gulp.start('sass')
  });
});
2 Likes
16

This is not working for me. Any alternatives?