File access in mounted volumes extremely slow, CPU bound

Hi @michaelperrin & @davidcrty yup, moving vendor dir out of shared dirs is an improvement (through a docker named volume it’s pretty cool for local dev if you plan tu run multiple instances of the same container).

But… even with that I can say that my behat test suite is about 4 times slower than on a virtualbox docker machine with some NFS mount.

Has anyone tried the latest update released yesterday?

https://beta.docker.com/docs/mac/release-notes/


Beta 12 Release (2016-05-17 1.11.1-beta12)

Upgrades

  • FUSE 7.23 for osxfs

Known issues

  • Docker.app sometimes uses 200% CPU after OS X wakes up from sleep mode. The issue is being investigated. The workaround is to restart Docker.app.

1 Like
Version 1.11.1-beta12 (build: 7528)
4151b0fc71e51fb8e551d0a1870893ac99a379bc
sh-4.1# time dd if=/dev/zero of=test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 25.1831 s, 4.1 MB/s

real	0m25.188s
user	0m0.140s
sys	0m2.150s
1 Like

Haven’t run specific tests but have still found beta12 to be extremely slow with mounted volumes. To the point where I’ve stopped using it, think I’ll retry with each new release and stick with alternatives until then.

1 Like
OS X: version 10.11.4 (build: 15E65)
Docker.app: version v1.11.1-beta12
Running diagnostic tests:
[OK]      Moby booted
[OK]      driver.amd64-linux
[OK]      vmnetd
[OK]      osxfs
[OK]      db
[OK]      slirp
[OK]      menubar
[OK]      environment
[OK]      Docker
[OK]      VT-x
Docker logs are being collected into /tmp/20160518-222735.tar.gz
Most specific failure is: No error was detected

time dd if=/dev/zero of=/var/www/test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
real	0m 32.85s
user	0m 0.22s
sys	0m 2.13s
1 Like

Not only the file access is slow but you can also definitely spot a change on frameworks using lots of files to render pages.

For example, a Magento run with a Apache + PHP - FPM + Percona stack is visually way lot slower on page load when the folder are mounted on the host VS when the folder is only stored as a Volume in the VM.

AB benchmark comparison:
With mounted folders:

$ ab -n10 -c10 http://magento.dev/
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking magento.dev (be patient).....done


Server Software:        
Server Hostname:        magento.dev
Server Port:            80

Document Path:          /
Document Length:        0 bytes

Concurrency Level:      10
Time taken for tests:   31.205 seconds
Complete requests:      10
Failed requests:        7
   (Connect: 0, Receive: 0, Length: 7, Exceptions: 0)
Total transferred:      235984 bytes
HTML transferred:       232603 bytes
Requests per second:    0.32 [#/sec] (mean)
Time per request:       31204.892 [ms] (mean)
Time per request:       3120.489 [ms] (mean, across all concurrent requests)
Transfer rate:          7.39 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.1      1       1
Processing:     2 17343 12491.7  23129   31203
Waiting:        0 17340 12491.0  23127   31195
Total:          3 17343 12491.8  23129   31204

Percentage of the requests served within a certain time (ms)
  50%  23129
  66%  23259
  75%  24196
  80%  30541
  90%  31204
  95%  31204
  98%  31204
  99%  31204
 100%  31204 (longest request)

With volume only on the VM:

$ ab -n10 -c10 http://magento.dev/
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking magento.dev (be patient).....done


Server Software:        Apache/2.4.20
Server Hostname:        magento.dev
Server Port:            80

Document Path:          /
Document Length:        33229 bytes

Concurrency Level:      10
Time taken for tests:   1.605 seconds
Complete requests:      10
Failed requests:        0
Total transferred:      337120 bytes
HTML transferred:       332290 bytes
Requests per second:    6.23 [#/sec] (mean)
Time per request:       1605.108 [ms] (mean)
Time per request:       160.511 [ms] (mean, across all concurrent requests)
Transfer rate:          205.11 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:   396  952 451.4    952    1604
Waiting:      394  951 451.2    951    1604
Total:        396  953 451.4    952    1605

Percentage of the requests served within a certain time (ms)
  50%    952
  66%   1057
  75%   1395
  80%   1561
  90%   1605
  95%   1605
  98%   1605
  99%   1605
 100%   1605 (longest request)

That is indeed nearly 25 times quicker without the files mounted on the host if we consider the median time :unamused:
On OS X: version 10.11.5 (build: 15F34) & Docker.app: version v1.11.1-beta12

The same applies to me with latest beta.

Running one of my projects natively runs pretty fast ~200ms per request but as soon as I add a shared volume the response time goes towards 30s+

OS X: version 10.11.4 (build: 15E65)
Docker.app: version v1.11.1-beta12
Running diagnostic tests:
[OK] Moby booted
[OK] driver.amd64-linux
[OK] vmnetd
[OK] osxfs
[OK] db
[OK] slirp
[OK] menubar
[OK] environment
[OK] Docker
[OK] VT-x

I have the same problem.

I think I can make it better by not including node_modules and tmp folders (it is an ember project). Is there a way to exclude some folders from volumes? (.dockerignore didn’t work)

Experiencing same problem running Rails apps with mounted folder.

Here is some additional info, hope it helps to debug, find the problem:

  • Macbook Pro - OS X El Capitan v10.11.5
  • Docker app version - Version 1.11.1-beta12 (build: 7528)

I also tried test posted by @tomfrost , here’s the output:

Docker Mac Beta:

/Users/andres1 # time dd if=/dev/zero of=test.dat bs=1024 count=10000
10000+0 records in
10000+0 records out
real	0m 3.44s
user	0m 0.05s
sys	0m 0.30s

Default docker-machine from Toolbox, Oracle VM, 2GB mem, nothing modified:

/Users/andres1 # time dd if=/dev/zero of=test.dat bs=1024 count=10000
10000+0 records in
10000+0 records out
real	0m 0.50s
user	0m 0.00s
sys	0m 0.31s

Default docker-machine, Oracle VM, mounted /Users using NFS:

/Users/andres1 # time dd if=/dev/zero of=test.dat bs=1024 count=10000
10000+0 records in
10000+0 records out
real	0m 0.07s
user	0m 0.00s
sys	0m 0.03s

Check now with version 1.11.1-beta13, is now fast!! I test with a Symfony project and now is 3x faster

1 Like

I can confirm this; Beta13 made things a lot faster. An empty Laravel 5.1 project used to load in 4000ms seconds with Beta12, now it’s around 1200ms.

Still not fast, but definitely getting there. Really nice to see the Docker team is making tremendous progress here!

1 Like

I can confirm the same results as @mvdstam. Running phpunit on a custom app in Beta 12 was unbearable. In Beta 13, although still slow, is now somewhat manageable. Keep up the good work guys!

1 Like
Version 1.11.1-beta13 (build: 7975)
16dbe555c7dd4304521b21e8286d83fe4864c15c
time dd if=/dev/zero of=test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 19.0928 s, 5.4 MB/s

real	0m19.097s
user	0m0.090s
sys	0m1.130s

I can definitely see a huge improvement indeed with the latest beta13 from the metrics there File access in mounted volumes extremely slow, CPU bound

Here is a new benchmark:
With mounted folder from Host:

$ ab -n10 -c10 http://magento.dev/
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking magento.dev (be patient).....done


Server Software:        
Server Hostname:        magento.dev
Server Port:            80

Document Path:          /
Document Length:        0 bytes

Concurrency Level:      10
Time taken for tests:   8.492 seconds
Complete requests:      10
Failed requests:        7
   (Connect: 0, Receive: 0, Length: 7, Exceptions: 0)
Non-2xx responses:      1
Total transferred:      202759 bytes
HTML transferred:       199673 bytes
Requests per second:    1.18 [#/sec] (mean)
Time per request:       8491.525 [ms] (mean)
Time per request:       849.152 [ms] (mean, across all concurrent requests)
Transfer rate:          23.32 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.1      1       1
Processing:     2 4023 3614.3   5173    8490
Waiting:        0 4021 3613.7   5172    8482
Total:          3 4024 3614.3   5174    8491

Percentage of the requests served within a certain time (ms)
  50%   5174
  66%   6435
  75%   7491
  80%   8074
  90%   8491
  95%   8491
  98%   8491
  99%   8491
 100%   8491 (longest request)

With volume only on the VM:

$ ab -n10 -c10 http://magento.dev/
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking magento.dev (be patient).....done


Server Software:        Apache/2.4.20
Server Hostname:        magento.dev
Server Port:            80

Document Path:          /
Document Length:        33229 bytes

Concurrency Level:      10
Time taken for tests:   1.161 seconds
Complete requests:      10
Failed requests:        0
Total transferred:      337120 bytes
HTML transferred:       332290 bytes
Requests per second:    8.61 [#/sec] (mean)
Time per request:       1161.144 [ms] (mean)
Time per request:       116.114 [ms] (mean, across all concurrent requests)
Transfer rate:          283.53 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.3      1       1
Processing:   395  764 278.1    882    1160
Waiting:      395  764 278.1    881    1159
Total:        396  765 278.2    883    1161

Percentage of the requests served within a certain time (ms)
  50%    883
  66%    905
  75%    965
  80%   1075
  90%   1161
  95%   1161
  98%   1161
  99%   1161
 100%   1161 (longest request)

So I can confirm the general feeling, the beta13 makes it usable now and the change are on the right path we are just not as the near OS speed yet, though.

Beta13 is much more performant on my very simple Symfony example (a default Symfony install) too. Running this:

ab -n 100 -r 127.0.0.1:8080

takes 26.9 seconds instead of 210 seconds! (I’m using a late-2013 Macbook Pro)

I published more detailed results here: https://github.com/michaelperrin/docker-symfony-test/blob/master/README.md#results

It’s still quite slow but it’s already a huge step forward.

The http call (read) is maybe fast but write speed of the mounted volumes are still slow.

Tried same test again with beta13, it’s faster, but still slow compared to VirtualBox:

  • Macbook Pro - OS X El Capitan v10.11.5

  • Docker app version - Version 1.11.1-beta12 (build: 7528)

Docker Mac Beta:

/Users/andres1 #  time dd if=/dev/zero of=test.dat bs=1024 count=10000
10000+0 records in
10000+0 records out
real	0m 2.40s
user	0m 0.03s
sys	0m 0.15s

Default docker-machine, Oracle VM, mounted /Users using NFS:

/Users/andres1 # time dd if=/dev/zero of=test.dat bs=1024 count=10000
10000+0 records in
10000+0 records out
real	0m 0.07s
user	0m 0.00s
sys	0m 0.03s

~1s improvement compared to beta12

(I don´t know if that really belongs here)

We have major issues with the latest version.

We are using NodeJS/Gulp aund gulp-watch to watch the filesystem for changes.

Up until yesterday it was just very slow but since 1.11.1-beta13 (build: 7975) the watch mechanism is completely broken. It does not recognize any file changes in a long time and when it does the docker container breaks itself completely.

gulp-watch uses internally chokidar which uses “a native extension exposing the Darwin
FSEvents API” (https://github.com/paulmillr/chokidar).

Anyone else experiencing similar issues?

@alexlawrence

Using the search, I was able to find topics more related to the gulp-watch problem. One of them is this one:

HTH,
Alexandre

Macbook Pro – OS X El Capitan v10.11.5 (15F34)
Docker Mac – Version 1.11.1-beta13.1 (build: 8193)

# time dd if=/dev/zero of=test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
real	0m 26.45s
user	0m 0.18s
sys	0m 1.19s

How to enable sharing with NFS?
Currently docker for mac is not useable :frowning: