Unable to perform any 'npm install' inside a osxfs mounted directory

Docker Desktop
1

Expected behaviour

Should be able to execute ‘npm install’

Actual behaviour

Massive numbers of errors ranging from ENOENT, EEXIST, general write IO errors

Information

Pinata Diagnose:
OS X: version 10.11.5 (build: 15F34)
Docker.app: version v1.11.1-beta12
Running diagnostic tests:
[OK] Moby booted
[OK] driver.amd64-linux
[OK] vmnetd
[OK] osxfs
[OK] db
[OK] slirp
[OK] menubar
[OK] environment
[OK] Docker
[OK] VT-x

Docker run command: "docker run -it --rm -v local-folder:/usr/app -w /usr/app mhart/alpine-node:latest sh -c “apk add --no-cache --virtual .npm-deps git python make gcc linux-headers alpine-sdk && /usr/bin/npm set progress=false && /usr/bin/npm install”

Package.json in question:

{ "name": "app", "version": "0.0.1", "description": "", "main": "app.js", "author": "", "license": "ISC", "devDependencies": { "chai": "^3.0.0", "mocha": "^2.2.5", "sinon": "^1.15.3", "jsdoc":"3.4.0", "jshint":"2.9.1-rc1", "jscs":"2.6.0" }, "dependencies": { "nodemon":"1.9.1", "restify":"4.0.4", "bunyan":"1.5.1", "sigyan":"0.2.0", "trace-event":"1.3.0", "consul":"0.19.0", "fluent-logger-stream":"https://github.com/cjpark87/fluent-logger-nodejs.git", "async":"1.5.0", "uuid":"2.0.1", "restify-url-semver":"*", "validator":"4.4.0", "mongodb":"2.0.52", "kerberos":"0.0.17", "mongoose":"4.3.4", "bunyan-prettystream":"0.1.3", "redis":"2.6.0-0", "redis-connection-pool":"https://github.com/davefinster/node-redis-connection-pool.git", "assert-plus":"1.0.0", "request-scrubber":"https://github.com/docketbook/request-scrubber.git", "lodash":"4.8.2", "consul-microagent":"https://github.com/docketbook/consul-microagent.git", "bunyan-logstash-tcp":"0.3.4", "thinky":"https://github.com/davefinster/thinky.git", "bluebird-events":"2.0.0" }, "directories": { "test": "test" }, "scripts": { "test": "./node_modules/mocha/bin/mocha test/uoapi", "start": "node app.js" } }

Actual result:
npm WARN enoent ENOENT: no such file or directory, open '/usr/app/node_modules/CBuffer/package.json' [...lots more...] npm ERR! Linux 4.4.9-moby npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "install" npm ERR! node v6.2.0 npm ERR! npm v3.9.0 npm ERR! path /usr/app/node_modules/bunyan-logstash-tcp/node_modules npm ERR! code EEXIST npm ERR! errno -17 npm ERR! syscall mkdir npm ERR! EEXIST: file already exists, mkdir '/usr/app/node_modules/bunyan-logstash-tcp/node_modules' npm ERR! File exists: /usr/app/node_modules/bunyan-logstash-tcp/node_modules npm ERR! Move it away, and try again.

At this stage npm is completely unusable for me when used inside a mounted volume. If I copy the package.json to a folder within the containers normal file system, there are no issues at all and I’m able to install all packages successfully.

This issue appears to have manifested itself with the latest beta, is it possible to download an earlier beta version?

I also see the same errors if I perform an npm install in a normal non-volume folder but then attempt to copy the files manually into a volume mounted folder.

2

Thanks for the report, I’ve opened bug (#3229) on our tracker.
We’ll have a fix for this soon…

3

Thanks Dave. Any chance of links to older betas as I don’t believe I had this issue in say 11 (although happy to report if I do)

4

+1. After the latest update, our deploy script is now giving us errors which indicate that basic filesystem-related commands like cd, mv, and rm are randomly failing. (I say “randomly” because the error messages are different each time, indicating some degree of nondeterminism in whatever logic is causing this.)

5

Sorry, I missed this earlier :frowning:

Do you think this is related to: Npm install doesn't complete inside docker container

6

The test for the scenario would be to perform the exact same npm install that would otherwise fail in a non-volume folder inside the container. That helps identify whether the issue lies in osxfs or somewhere else.

7

Thanks for the advice Dave, I’m quite new to all this :slight_smile:

I’ll post my result in the other thread, as to keep it together.

8

I ran into a similar issue with npm. May want to take a look at a discussion about it on an npm github issue https://github.com/npm/npm/issues/9863#issuecomment-215953346.

The solution for that one (which may be the same) is to add the following to your Dockerfile.

RUN cd $(npm root -g)/npm \
&& npm install fs-extra \
&& sed -i -e s/graceful-fs/fs-extra/ -e s/fs.rename/fs.move/ ./lib/utils/rename.js

Fingers crossed that’ll fix it for you :slight_smile:

9

+1. Lot of ENOENT warnings, package installation fails with EEXIST because of failed mkdir command. Also it’s not possible to delete the node_modules folder (or any other within the mounted volumes) using rm -rf.

10

I’m having a similar issue with PHP Composer. The install fails because the mode of the downloaded files is broken. They appear in the filesystem but are not stat- or readable.

ls -la output:

-????????? ? ?    ?      ?            ? PatchEventTest.php
11

I had the same issue yesterday (composer) and ended up reseting docker (“Settings” > “Uninstall/Reset” > “Reset to factory defaults”) and it seem to fix it somehow.

12

Did not work for me. Still the same after reset.

13

I’m to the point where I have this in a makefile:

monkeynpm: rm -rf node_modules && \ docker run -it --rm -v $(pwd):/usr/src/app -w /usr/src/app mhart/alpine-node:latest sh -c "apk add --no-cache --virtual .npm-deps git python make gcc linux-headers alpine-sdk && mkdir ../npm-install && cp package.json ../npm-install/package.json && cd ../npm-install && npm install && tar -cf ../app/npm.tar node_modules" && \ tar -zxf npm.tar && \ rm -rf npm.tar
Essentially clears the node_modules folder, spools up a container that copies the package.json to a directory elsewhere in the container’s normal file system, does an npm install, tarballs the resulting node_modules folder, copies the tar back to the working directory (which is my source code directory) and then in OS X untars/removes the result.

Dirty hack but desperate times…

14

+1 for this - spent 8 hours debugging an app - connected to a docker-machine instance and worked like a charm! Definitely a Mac native beta issue

15

Hi Dylan,

I’m sorry for the regression introduced in Beta 12 that caused you to waste time debugging what was ultimately an issue with Docker for Mac Beta. Have you upgraded to Beta 13? I believe this issue should now be fixed in Beta 13.

Thank for participating in the Docker for Mac Beta!

Best regards,

David

1 Like
16

Hi David,

Thanks for your reply - I don’t mind as I think Docker is awesome anyway! More kicking myself over not realising for that length of time. I have installed the new versions today and will give it a test soon.

Cheers

Dylan

17

Can confirm the latest beta (13) appears to have fixed this issue - Thanks guys! I’d suggest using docker-machine for any client critical work around containers over the Mac Beta. Still going to keep testing on here though

18

Looks like this has been significantly improved, but I’m still running into a few issues; output below.

npm WARN deprecated jade@0.26.3: Jade has been renamed to pug, please install the latest version of pug instead of jade
npm WARN deprecated graceful-fs@3.0.8: graceful-fs v3.0.0 and before will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon as possible. Use 'npm ls graceful-fs' to find it in the tree.
npm ERR! Linux 4.4.11-moby
npm ERR! argv "/home/gibson/emsdk_portable/node/4.1.1_64bit/bin/node" "/home/gibson/emsdk_portable/node/4.1.1_64bit/bin/npm" "install"
npm ERR! node v4.1.1
npm ERR! npm  v2.14.4
npm ERR! code EIO
npm ERR! errno -5
npm ERR! syscall fstat

npm ERR! EIO: i/o error, fstat
npm ERR!
npm ERR! If you need help, you may report this error at:
npm ERR!     <https://github.com/npm/npm/issues>

npm ERR! Please include the following file with any support request:
npm ERR!     /cyph/shared/lib/js/github/isagalaev/highlight.js@9.4.0/npm-debug.log

npm WARN engine selenium-webdriver@2.53.2: wanted: {"node":">= 4.2.x"} (current: {"node":"4.1.1","npm":"2.14.4"})
npm WARN engine webrtc-adapter@1.3.0: wanted: {"npm":"~3.0.0"} (current: {"node":"4.1.1","npm":"2.14.4"})
npm WARN engine webrtc-adapter@1.3.0: wanted: {"npm":"~3.0.0"} (current: {"node":"4.1.1","npm":"2.14.4"})
npm WARN engine webrtc-adapter@1.3.0: wanted: {"npm":"~3.0.0"} (current: {"node":"4.1.1","npm":"2.14.4"})

> travis-multirunner@3.0.1 postinstall /cyph/shared/lib/js/github/andyet/simplewebrtc@2.1.0/node_modules/travis-multirunner
> ./bin/travis-sync


> precommit-hook@0.3.10 install /cyph/shared/lib/js/github/andyet/simplewebrtc@2.1.0/node_modules/precommit-hook
> node install.js

This project doesn't appear to be a git repository. JSHint configuration will be created anyway. To enable the pre-commit hook, run `git init` and reinstall precommit-hook.
npm WARN deprecated buffer-browserify@0.1.0: Package not maintained. Recent browserify uses https://github.com/feross/buffer
npm WARN deprecated win-spawn@2.0.0: use [cross-spawn](https://github.com/IndigoUnited/node-cross-spawn) or [cross-spawn-async](https://github.com/IndigoUnited/node-cross-spawn-async) instead.
npm WARN deprecated npmconf@2.1.2: this package has been reintegrated into npm and is now out of date with respect to npm
|
> chromedriver@2.21.2 install /cyph/shared/lib/js/github/andyet/simplewebrtc@2.1.0/node_modules/chromedriver
> node install.js

Downloading http://chromedriver.storage.googleapis.com/2.21/chromedriver_linux64.zip
Saving to /tmp/chromedriver/chromedriver_linux64.zip
Receiving...
Received 781K...
Received 1564K...
Received 2346K...
Received 2629K total.
Extracting zip contents
Copying to target path /cyph/shared/lib/js/github/andyet/simplewebrtc@2.1.0/node_modules/chromedriver/lib/chromedriver
Fixing file permissions
Done. ChromeDriver binary available at /cyph/shared/lib/js/github/andyet/simplewebrtc@2.1.0/node_modules/chromedriver/lib/chromedriver/chromedriver
npm ERR! Linux 4.4.11-moby
npm ERR! argv "/home/gibson/emsdk_portable/node/4.1.1_64bit/bin/node" "/home/gibson/emsdk_portable/node/4.1.1_64bit/bin/npm" "install"
npm ERR! node v4.1.1
npm ERR! npm  v2.14.4
npm ERR! code EIO
npm ERR! errno -5
npm ERR! syscall close

npm ERR! EIO: i/o error, close
npm ERR!
npm ERR! If you need help, you may report this error at:
npm ERR!     <https://github.com/npm/npm/issues>

npm ERR! Please include the following file with any support request:
npm ERR!     /cyph/shared/lib/js/github/andyet/simplewebrtc@2.1.0/npm-debug.log