Docker Community Forums

Share and learn in the Docker community.

File access in mounted volumes extremely slow, CPU bound


(Bernard Tai) #184

Tried unison for dealing with volumes for Magento 2 and seeing issues with syncing large volumes. Not an ideal solution we’ve been able to get acceptable load times if we pre-build images with and avoid mapping certain core library folders.

Looking forward to this issue being solved otherwise keep up the good work Docker!


(Nikolai Zujev) #185

Thank you @dsheets for detailed explanation, very helpful.

I’ve noticed an improvement in sync alpine image:

$ docker run --rm -ti -v ~/Junk:/mnt alpine sh

And running command to generate file:

time dd if=/dev/zero of=test.dat bs=1024 count=100000
  • from docker inside non-volume dir - fast (~0.3 sec)
  • from docker inside volume dir - slow (30 sec)
  • from host dir to docker mounted volume - fast (~0.3 sec)

For me it ok, as I don’t generate large files inside docker that requires sync back to host.

Writing log file inside volume is OK’ish - 10 times by 10 MB is approx 3.7 sec.


(Dave Wikoff) #186

I don’t see why host -> docker writing would ever be slow - or at least the writing to the native disk part. I may be missing something though. The slowness with reading is still there sadly so anything that reads a lot of files are still slow.


(Arthur Lutz) #187

Hi all, I think I’m hitting on the same problem. Filed an issue https://github.com/docker/docker/issues/25656 and was redirected to https://github.com/docker/docker/issues/21485 which is probably the best technical recap of these types of issues I’ve read so far.


(Taiidani) #188

Would love an update on where this is going with the Docker for Mac roadmap. We use a dozen or so containers when developing locally, 2 of them being Gulp watchers. The filesystem scanning simply tears into the host’s CPU when it runs.

Edit: I just spotted @dsheets reply from a few weeks ago. Great response and thanks for the update! I can’t give as much detail as the post requested, but know that the two biggest pain points on my stack have been:

  • gulp 3 watch processes, which are based on the standard Node 5-slim Hub image. These hurt us the most because they sit idly in the background during development but still peg the CPU due to their filesystem monitoring.
  • Validation routines on our code using PHP Code Sniffer. This is an on-demand process which tokenizes entire directories of PHP files before analyzing them.

(Den360) #189

Created a test case for Gulp.
https://github.com/den-t/docker-for-mac-gulp-test/
Currently hard to test in measurable terms. It just shows prolonged periods of CPU load, after npm install finishes.

@taiidani Could you suggest a watcher that will showcase the problem better than what is there.
PRs welcome. :slight_smile:


(David Sheets) #190

I recommend using a package like gulp-watch or gulp-nodemon or similar that takes advantage of inotify file system events rather than constantly polling the file system.


(Jamesraybould) #191

I’ve got an example repo using gulp-watch over gulp.watch here for the answer to this question


(Eugenesia) #192

I totally missed the “Opcache” workaround hidden in this reply. I’m also a Drupal developer experiencing crippling slowness when doing drush cc all.

I’ve enabled Opcache on PHP and it seems to speed things up considerably. Probably because it reduces file reads.

On a development machine I have set opcache.revalidate_freq=60, which means the opcache is only valid for 60 seconds. I can’t see the effects of code changes immediately and have to wait for 60s, but I think it is tolerable.


(Sysradium) #193

I also spotted that OSXFS process is being in the top of activity monitor. I was writing a typical python + django app where one of the containers was a buil-in debug webserver, which watches for code changes and restarts if necessary.

Turns out that’s the container which drained my battery. After I stopped it osxfs process disappeared from a top.
Is there a way to have a watcher but not kill the CPU? Or maybe Docker is going to do something about that?


(Trousered) #194

I suspect i’m seeing similar performance slowdown with Docker for Mac

Version 1.12.0-a (build: 11213)

I’m seeing differences when running pgbench_tools (https://github.com/gregs1104/pgbench-tools)

Straight postgres container with datavolume ( -v “…/postgresdatadocker:/var/lib/postgresql/data” ) :

set | scale | test | script | clients | workers | limit | tps | avg_latency | 90%< | max_latency | trans
-----±------±-----±-----------±--------±--------±------±-----±------------±------±------------±-------
1 | 1 | 1 | select.sql | 1 | 1 | | 1411 | 0.704 | 0.801 | 35 | 84691
1 | 1 | 2 | select.sql | 2 | 1 | | 2337 | 0.851 | 1.041 | 44 | 140201
1 | 1 | 3 | select.sql | 4 | 1 | | 2603 | 1.42 | 1.958 | 45 | 156191
1 | 1 | 4 | select.sql | 8 | 1 | | 2671 | 2.983 | 3.76 | 44 | 160284
1 | 1 | 5 | select.sql | 16 | 1 | | 0 | | | |
1 | 1 | 6 | select.sql | 32 | 1 | | 0 | | | |
(6 rows)

Versus Virtualbox Ubuntu VM with Docker postgres container and NFS share :

set | scale | test | script | clients | workers | limit | tps | avg_latency | 90%< | max_latency | trans
-----±------±-----±-----------±--------±--------±------±-----±------------±------±------------±-------
1 | 1 | 1 | select.sql | 1 | 1 | | 5259 | 0.187 | 0.227 | 156 | 316306
1 | 1 | 7 | select.sql | 1 | 1 | | 0 | | | |
1 | 1 | 2 | select.sql | 2 | 1 | | 5915 | 0.335 | 0.601 | 156 | 354903
1 | 1 | 3 | select.sql | 4 | 1 | | 7040 | 0.565 | 1.209 | 203 | 422412
1 | 1 | 4 | select.sql | 8 | 1 | | 8791 | 0.906 | 1.352 | 285 | 527490
1 | 1 | 5 | select.sql | 16 | 1 | | 0 | | | |
1 | 1 | 6 | select.sql | 32 | 1 | | 0 | | | |
(7 rows)


(Tobias Munk) #195

I created an issue here: https://github.com/docker/for-mac/issues/77


(Jeffery Utter) #197

@dsheets I’m wondering if there is any update to this in the past two months?

I am running 1.12.1-beta26.1 and it still seems to have this problem. I am working on a large ruby project and the time it takes to start running tests is measured in minutes, whereas my coworkers working in a VM measures it in seconds. I would love to keep using docker-for-mac, but if reasonable performance is too far out. I may have to switch back to a VM.

Thanks.


(King'ori Maina) #198

Was in the same situation (large Ruby app) … use http://docker-sync.io for now.


(Jeffery Utter) #199

Yes, I’ve been meaning to try it out. In the meantime I’ve switched to docker-machine and am getting about a 5x speedup. There are some features like user remapping which seem very handy.

What I was more hoping for though is some kind of Idea on the progress of having this solved natively. Perhaps some github issues to watch?


(Xaviermoon) #200

Any update to this issue?


(Cytopia) #201

Hi everybody,

same issue here.

Is there already a status update/estimate you could give?
When might this be fixed?


(Bradcondo) #202

Would also appreciate an update. I had to revert back to using a VirtualBox VM due to volume performance.


(Jvlmp) #203

+1
Having issues with the slow performance as well.


(Cytopia) #204

@dsheets Could somebody from the Docker staff please provide some feedback here.