File access in mounted volumes extremely slow, CPU bound

Hi from Paraguay,

I’m running a Rails app in one container linked to a Postgres DB in another, and yes, with a volume for my source code the app runs really slow averaging almost 2 mins for simple HTTP requests. Without the volume it seems at least as fast as when using docker-machine. By the way, instead of using a volume for my sources, I enable the RemoteSync plugin in my Atom editor.

Following I provide some info about my environment

OS X El Capitan (version 10.11.4)

docker version

Client:
Version: 1.11.1
API version: 1.23
Go version: go1.5.4
Git commit: 5604cbe
Built: Wed Apr 27 00:34:20 2016
OS/Arch: darwin/amd64

Server:
Version: 1.11.1
API version: 1.23
Go version: go1.5.4
Git commit: 8b63c77
Built: Fri Apr 29 14:13:26 2016
OS/Arch: linux/amd64

Version of my Docker app

Version 1.11.1-beta10 (build: 6662)
b4d942b304b7d7d0ee48d96e3b424ccfbcceca89

docker-compose.yml

web:
  image: cnatural_web:latest
  volumes:
    - .:/home/app/webapp
  links:
    - db
  ports:
    - "3001:80"
    - "2221:22"
  environment:
    RAILS_ENV: development
    PASSENGER_APP_ENV: development
    BUNDLE_PATH: /home/app/bundle
    FB_APP_ID: XXXXXX
  volumes_from:
    - bundle

db:
  image: postgres:9.4.4
  ports:
    - "55431:5432"

bundle:
  image: cnatural_web:latest
  command: ls /home/app/bundle
  volumes:
    - /home/app/bundle

My cnatural_web image is FROM phusion/passenger-customizable:0.9.17

WITH VOLUME docker stats

CONTAINER           CPU %               MEM USAGE / LIMIT     MEM %               NET I/O               BLOCK I/O           PIDS
dc9cf20b0f73        0.58%               207.4 MB / 4.143 GB   5.01%               554.1 kB / 215 kB     0 B / 618.5 kB      56
e14d47647570        0.00%               19.78 MB / 4.143 GB   0.48%               107.7 kB / 361.1 kB   0 B / 8.872 MB      7

It takes really long to even start processing the request and strangely - perhaps the symptom - BLOCK I/O both report 0B/xxx.

WITHOUT VOLUME docker stats

CONTAINER           CPU %               MEM USAGE / LIMIT     MEM %               NET I/O               BLOCK I/O             PIDS
04ea0ab1185d        0.58%               435.1 MB / 4.143 GB   10.50%              1.555 MB / 1.237 MB   38.28 MB / 10.76 MB   58
e14d47647570        0.00%               21.54 MB / 4.143 GB   0.52%               125.4 kB / 460.2 kB   266.2 kB / 17.9 MB    7

I hope that this volume slow performance issue can be fixed soon because although the RemoteSync plugin does a good job copying filies via rsync, it’s somehow burdensome when it comes to deal with changing directory structure with git because you have to manually scp the files. Also, you always have to take care of copying some files like Gemfile.lock from container to host.

Using Docker for Mac vs. Docker Machine (VMware Fusion 7 with NFS shared /Users directory on OS X 10.11.4) I get 20 sec vs 1.5 sec when dd’ing into shared directory:

Docker for Mac 1.11.1-beta10:

$ docker run --rm -it -v `pwd`:`pwd` -w `pwd` alpine /bin/sh
/Users/fuerst/Sites # time dd if=/dev/zero of=speedtest bs=1024 count=100000
100000+0 records in
100000+0 records out
real	0m 20.24s
user	0m 0.41s
sys	0m 1.21s

Docker Machine 1.11.0:

$ docker run --rm -it -v `pwd`:`pwd` -w `pwd` alpine /bin/sh
/Users/fuerst/Sites # time dd if=/dev/zero of=speedtest bs=1024 count=100000
100000+0 records in
100000+0 records out
real	0m 1.46s
user	0m 0.01s
sys	0m 0.13s

The great news: Docker for Mac behaves perfectly for every application I work on, great job, guys!
The bad news: The filesystem performance means I have to switch back to my VirtualBox docker-machine for now.

Hello, same thing here, to slow for now. 2 to 4 times slower than with my vmware docker machine that mounts my mac through NFS.
I am waiting the next update!

You rock guys you will figure out something!

Just confirming this behavior. For our app, we moved away from virtualbox/fusion shared folder to rsync based fusion once. This reduced the execution of a general from about 4 minutes to 2.2 seconds.

Using docker for mac, things are either as worse as with virtualbox/fusion shared folders, or rather even worse.

This said, still docker for mac is nice to have services bound to localhost to ease up every development / debugging configuration ( remote debugging needs some tender care with docker-machine, no matter with PHP, ruby or Java). Also the xhyve based stack seems to run a lot more efficient, then the vmware-fusion variant, which is faster the vbox.

But having not solved the shared-folder issue is still blocking a lot of mac users and the “spreading” of docker as a development tool in general for mac users and windows users, i guess.

Thank you for the hard work anyway!

I can confirm the slow FS performance.

Here’s my stats:

user@Native-Mac-FS/app$ time time dd if=/dev/zero of=speedtest bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes transferred in 0.360199 secs (284287317 bytes/sec)
real    0m0.373s
user    0m0.022s
sys     0m0.288s

vs performance in the container:

root@660d6823b06f:/app$ time dd if=/dev/zero of=speedtest bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 39.5285 s, 2.6 MB/s
real    0m39.538s
user    0m0.200s
sys     0m1.990s

I can also +1 slow FS performance is preventing us from deploying Docker in a bigger way for our development environments.

Unfortunately, having the same problem, application is unusable cause it runs all commands very slow. When i used docker-machine on mac, i manually mounted shared volume via nfsd and it fixed the problem. I use docker-compose to build everything like this:

data:
      ...
      volumes:
        - .:/var/www

And run it like: docker-compose up -d.
I hope this problem will be fixed in the future releases. for now have to switch back to docker-machine.

Why are most of you all switching back to docker-machine / dockertoolbox due to this issue?
The issue with docker-mac-dev and sorts are, they make to many assumptions (like forcing to user VirtualBox). But this is not needed

For this issue to be fixed no matter you use docker-machine (virtualbox or fusion or paralells) or docker for mac, simply:

If you want to resync on changes automatically, use Filewatcher ( shell or ruby gem, what suits you best ) and thats basically it. This way, you are independent of the solution like VirtualBox, Fusion or docker for mac.

This solution is 2 times faster as NFS is.

Rsync with docker for mac is pretty nice, at least for my purpose. Though, i hope they fix this natively to remove rsync as an extra-dependency to the stack

5 Likes

With the new beta still a problem:

Version 1.11.1-beta11 (build: 6974)
37559e5f6acd56a4810963acc7001e88f2d88017
sh-4.1# time dd if=/dev/zero of=test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 26.5737 s, 3.9 MB/s

real	0m26.579s
user	0m0.190s
sys	0m1.630s

With Docker for mac:

docker version

Client:
Version: 1.11.1
API version: 1.23
Go version: go1.5.4
Git commit: 5604cbe
Built: Wed Apr 27 00:34:20 2016
OS/Arch: darwin/amd64

Server:
Version: 1.11.1
API version: 1.23
Go version: go1.5.4
Git commit: 8b63c77
Built: Tue May 10 10:39:20 2016
OS/Arch: linux/amd64

time dd if=/dev/zero of=test.dat bs=1024 count=100000

100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 38.0505 s, 2.7 MB/s
real 0m38.057s
user 0m0.170s
sys 0m2.030s

With Docker machine

time dd if=/dev/zero of=test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 1.71642 s, 59.7 MB/s

real 0m1.721s
user 0m0.000s
sys 0m0.100s

I am experiencing the same issues with 1.11.1, happy to provide specifics if useful, but they somewhat mirror those above

Looks like virtio-9p (VirtFS) is being added to xhyve. That could be the way to go.

Was excited to join the beta today because of this:

“Volume mounting for your code and data: volume mounting and data access just works, including file change notifications (on Mac inotify now works seamlessly inside containers for volume mounted directories). This enables edit/test cycles for “in container” development.”

Not so excited after trying the beta because of the slow mounted volumes.

Hello from Paris,

I’m having the same issue with my Symfony 3 application.

The page take more than 30sec to be generated. With my MAMP server it took less than 1sec.
Here is my config :
OS X El Capitan v10.11.4

docker version
Client:
Version: 1.11.1
API version: 1.23
Go version: go1.5.4
Git commit: 5604cbe
Built: Wed Apr 27 00:34:20 2016
OS/Arch: darwin/amd64

Server:
Version: 1.11.1
API version: 1.23
Go version: go1.5.4
Git commit: 8b63c77
Built: Tue May 10 10:39:20 2016
OS/Arch: linux/amd64

And my docker-compose.yml :
version: ‘2’
services:
db:
image: mysql
ports:
- “3306:3306”
environment:
MYSQL_ROOT_PASSWORD: root
MYSQL_DATABASE: my_db
MYSQL_USER: root
MYSQL_PASSWORD: root
php:
build: ./php-fpm
expose:
- “9000”
volumes:
- ./symfony:/var/www/symfony
- ./logs/symfony:/var/www/symfony/app/logs
links:
- db
nginx:
build: ./nginx
ports:
- “8080:80”
links:
- php
volumes_from:
- php
volumes:
- ./logs/nginx/:/var/log/nginx

1 Like

Hi @davidcrty,

There is an other thread concerning very slow Symfony apps: How to speed up shared folders

As shown in my last posts this is due to the vendors (much more than shared cache and logs folders to my surprise). So for now I move the vendor directory out of the shared directories.

I wish I could develop Symfony apps normally using Docker too!

Michaël (from Paris too)

2 Likes

Hi @michaelperrin & @davidcrty yup, moving vendor dir out of shared dirs is an improvement (through a docker named volume it’s pretty cool for local dev if you plan tu run multiple instances of the same container).

But… even with that I can say that my behat test suite is about 4 times slower than on a virtualbox docker machine with some NFS mount.

Has anyone tried the latest update released yesterday?

https://beta.docker.com/docs/mac/release-notes/


Beta 12 Release (2016-05-17 1.11.1-beta12)

Upgrades

  • FUSE 7.23 for osxfs

Known issues

  • Docker.app sometimes uses 200% CPU after OS X wakes up from sleep mode. The issue is being investigated. The workaround is to restart Docker.app.

1 Like
Version 1.11.1-beta12 (build: 7528)
4151b0fc71e51fb8e551d0a1870893ac99a379bc
sh-4.1# time dd if=/dev/zero of=test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 25.1831 s, 4.1 MB/s

real	0m25.188s
user	0m0.140s
sys	0m2.150s
1 Like

Haven’t run specific tests but have still found beta12 to be extremely slow with mounted volumes. To the point where I’ve stopped using it, think I’ll retry with each new release and stick with alternatives until then.

1 Like
OS X: version 10.11.4 (build: 15E65)
Docker.app: version v1.11.1-beta12
Running diagnostic tests:
[OK]      Moby booted
[OK]      driver.amd64-linux
[OK]      vmnetd
[OK]      osxfs
[OK]      db
[OK]      slirp
[OK]      menubar
[OK]      environment
[OK]      Docker
[OK]      VT-x
Docker logs are being collected into /tmp/20160518-222735.tar.gz
Most specific failure is: No error was detected

time dd if=/dev/zero of=/var/www/test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
real	0m 32.85s
user	0m 0.22s
sys	0m 2.13s
1 Like