File access in mounted volumes extremely slow, CPU bound

Docker Desktop for Mac
21

@ddunbar: @cliffom is correct; I apologize for the misunderstanding. osxfs is quite operational, and appears to be the source of the performance problems.

Two common fixes to this in the docker-machine + VM solution are to use rsync (docker-osx-dev) to one-way rsync a folder from host to VM, or NFS (dinghy for virtualbox, dlite for xhyve) to share the /Users folder with the VM over a private network. I’ve switched to using dlite with a solution to forward inotify events. This gives ultra-fast file access, bi-directional file syncing, and full inotify support.

NFS being a kernel-integrated solution that operates on request is largely what prevents file operations from going CPU-bound. Perhaps a better approach to osxfs would be to implement a solution similar to NFS, but allowing for native fswatch/inotify forwarding over the same pipe. That would eliminate the need for touch commands fired over ssh or similar out-of-band solutions to trigger inotify in the VM.

1 Like
22

I started using Docker for Mac beta today and immediately found the same issue: access to the shared volumes are slower than they were in VirtualBox. A change on a file took about 12 seconds to be propagated into the container, rather than the usual 2 seconds.

I’m enjoying the fact that I can move away from my previous solution to get filesystem events propagated. But the performance, if not improved, is going to become an issue and might be a show-stopper.

23

Another data point. For me it actually takes 1m26s for a mounted volume in OSX vs 1s for non mounted. And 12s mounted vs 0.65s non-mounted with docker-machine:

Docker for Mac:

docker run --rm -it -v `pwd`:`pwd` -w `pwd` alpine /bin/sh
/Users/stefanfoulis/tmp/speedtest # time dd if=/dev/zero of=test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
real	1m 26.05s
user	0m 0.17s
sys	0m 2.44s
/Users/stefanfoulis/tmp/speedtest # cd /
/ # time dd if=/dev/zero of=test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
real	0m 1.06s
user	0m 0.06s
sys	0m 0.91s


pinata diagnose
OS X: version 10.11.4 (build: 15E65)
Docker.app: version v1.11.0-beta6
Running diagnostic tests:
[OK]      docker-cli
[OK]      Moby booted
[OK]      driver.amd64-linux
[OK]      vmnetd
[OK]      osxfs
[OK]      db
[OK]      slirp
[OK]      menubar
[OK]      environment
[OK]      Docker
[OK]      VT-x
Docker logs are being collected into /tmp/20160413-150849.tar.gz.

docker-toolbox / docker-machine

docker-1.10.3 run --rm -it -v `pwd`:`pwd` -w `pwd` alpine /bin/sh
/Users/stefanfoulis/tmp # time dd if=/dev/zero of=test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
real	0m 12.25s
user	0m 0.08s
sys	0m 6.50s
/Users/stefanfoulis/tmp # cd /
/ # time dd if=/dev/zero of=test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
real	0m 0.65s
user	0m 0.04s
sys	0m 0.61s
1 Like
24

Switched back to the latest Docker Beta to keep performance tests current – as of 1.11.0-beta8, this is still a pretty severe problem. Running the same test suite in the beta as dlite, the beta runs in 72 seconds, dlite 22.

CPU usage difference between the two is extremely considerable. The beta pegs my CPU while my tests run, while dlite is barely a blip in top.

Hoping this can be overcome!

25

@tomfrost I’m curious, how your shared folder is mounted in the xhyve VM? Could you check that for me running the mount command inside the xhyve VM? Is it also using a fuse/osxfs mount? 9p was brievly suggested till beta 7 of docker for mac. BTW xhyve#97 is maybe good news if 9p has better performances thant fuse/osxfs.

26

I’ve not tried using 9p in conjunction with Docker for Mac, but I did trial the dlite 2.0 beta (9p on xhyve-- same idea). Performance was similar to what I’ve recorded with osxfs – many times worse than what I got using docker-machine on virtualbox. Dlite’s author is moving 2.0 back to NFS as a result. It’s a shame; as a technology 9p is quite impressive. I think its failure is that it’s not optimized for small file reads/writes like docker’s core use case for volumes: source code and logs.

27

Thx for your feedback, it’s a pity to admit that NFS is still the less worse solution, let’s see if osxfs can eventually be a challenger at some time.

28

I’ve just installed Docker for Mac beta 9 for the first time and immediately noticed this performance issue which is a real shame as everything else I’ve seen so far has been excellent.

29

same issue, symfony is unusable (15 seconds vs 60ms)

1 Like
30

I’ve seen at least a 10x slowdown in my very simple 2 container setup after switching from docker-machine/virtualbox.

  • Apache / PHP / Laravel Framework
  • Mysql

I’ve tried using a remote MySQL host instead of connecting to the other container, thinking it was something with MySQL, but indeed the remote host was 10X+ faster than the docker mysql.

Any suggestions are greatly appreciated

1 Like
31

Maybe it’s silly, but in doubt I’ll ask: is there any cleanup step that I need to perform (being an ex-docker-machine user) in order to use Docker for Mac Beta? Maybe uninstalling boot2docker or stuff like that? I just find very strange that only some people is experiencing this issue.

32

Still a problem, even with the latest Beta released last night.

For reference, I’ve been using docker in a Vagrant VM with host-mounted files for a while. My appserver with our entire EAR file mounted to the host file system can start up in about 70 seconds in this fashion, but with the Mac Beta it takes over 19 minutes to start.

I’m using vagrant rsync in the first case, not vboxsf, because using the this style of mount through Vagrant is much too slow. Is it a similar problem with the Mac Beta?

33

Just an update here from the Docker camp…
We’re still working on it! We’ve added some small performance boosts over the last few Beta’s and performance is heading in the right direction. There are a few other fixes that should be heading in over the next few betas that will also provide a decent boost. The target is to be much faster than VirtualBox and we’re confident we can do this before release.

In the meantime the best thing you can do to help is to give us samples of workloads you feel are slow ideally in a a form that we can replicate e.g Dockerfiles, Compose Files etc… We can then analyse these and try to optimise as much as possible for each!

Thanks for bearing with us and for trying the Docker for Mac beta

1 Like
34

Another developer joining into the conversation i’m seeing the same poor performance as everybody else. Posting so I can updates as this progresses…

35

Hi from Paraguay,

I’m running a Rails app in one container linked to a Postgres DB in another, and yes, with a volume for my source code the app runs really slow averaging almost 2 mins for simple HTTP requests. Without the volume it seems at least as fast as when using docker-machine. By the way, instead of using a volume for my sources, I enable the RemoteSync plugin in my Atom editor.

Following I provide some info about my environment

OS X El Capitan (version 10.11.4)

docker version

Client:
Version: 1.11.1
API version: 1.23
Go version: go1.5.4
Git commit: 5604cbe
Built: Wed Apr 27 00:34:20 2016
OS/Arch: darwin/amd64

Server:
Version: 1.11.1
API version: 1.23
Go version: go1.5.4
Git commit: 8b63c77
Built: Fri Apr 29 14:13:26 2016
OS/Arch: linux/amd64

Version of my Docker app

Version 1.11.1-beta10 (build: 6662)
b4d942b304b7d7d0ee48d96e3b424ccfbcceca89

docker-compose.yml

web:
  image: cnatural_web:latest
  volumes:
    - .:/home/app/webapp
  links:
    - db
  ports:
    - "3001:80"
    - "2221:22"
  environment:
    RAILS_ENV: development
    PASSENGER_APP_ENV: development
    BUNDLE_PATH: /home/app/bundle
    FB_APP_ID: XXXXXX
  volumes_from:
    - bundle

db:
  image: postgres:9.4.4
  ports:
    - "55431:5432"

bundle:
  image: cnatural_web:latest
  command: ls /home/app/bundle
  volumes:
    - /home/app/bundle

My cnatural_web image is FROM phusion/passenger-customizable:0.9.17

WITH VOLUME docker stats

CONTAINER           CPU %               MEM USAGE / LIMIT     MEM %               NET I/O               BLOCK I/O           PIDS
dc9cf20b0f73        0.58%               207.4 MB / 4.143 GB   5.01%               554.1 kB / 215 kB     0 B / 618.5 kB      56
e14d47647570        0.00%               19.78 MB / 4.143 GB   0.48%               107.7 kB / 361.1 kB   0 B / 8.872 MB      7

It takes really long to even start processing the request and strangely - perhaps the symptom - BLOCK I/O both report 0B/xxx.

WITHOUT VOLUME docker stats

CONTAINER           CPU %               MEM USAGE / LIMIT     MEM %               NET I/O               BLOCK I/O             PIDS
04ea0ab1185d        0.58%               435.1 MB / 4.143 GB   10.50%              1.555 MB / 1.237 MB   38.28 MB / 10.76 MB   58
e14d47647570        0.00%               21.54 MB / 4.143 GB   0.52%               125.4 kB / 460.2 kB   266.2 kB / 17.9 MB    7

I hope that this volume slow performance issue can be fixed soon because although the RemoteSync plugin does a good job copying filies via rsync, it’s somehow burdensome when it comes to deal with changing directory structure with git because you have to manually scp the files. Also, you always have to take care of copying some files like Gemfile.lock from container to host.

36

Using Docker for Mac vs. Docker Machine (VMware Fusion 7 with NFS shared /Users directory on OS X 10.11.4) I get 20 sec vs 1.5 sec when dd’ing into shared directory:

Docker for Mac 1.11.1-beta10:

$ docker run --rm -it -v `pwd`:`pwd` -w `pwd` alpine /bin/sh
/Users/fuerst/Sites # time dd if=/dev/zero of=speedtest bs=1024 count=100000
100000+0 records in
100000+0 records out
real	0m 20.24s
user	0m 0.41s
sys	0m 1.21s

Docker Machine 1.11.0:

$ docker run --rm -it -v `pwd`:`pwd` -w `pwd` alpine /bin/sh
/Users/fuerst/Sites # time dd if=/dev/zero of=speedtest bs=1024 count=100000
100000+0 records in
100000+0 records out
real	0m 1.46s
user	0m 0.01s
sys	0m 0.13s
37

The great news: Docker for Mac behaves perfectly for every application I work on, great job, guys!
The bad news: The filesystem performance means I have to switch back to my VirtualBox docker-machine for now.

38

Hello, same thing here, to slow for now. 2 to 4 times slower than with my vmware docker machine that mounts my mac through NFS.
I am waiting the next update!

You rock guys you will figure out something!

39

Just confirming this behavior. For our app, we moved away from virtualbox/fusion shared folder to rsync based fusion once. This reduced the execution of a general from about 4 minutes to 2.2 seconds.

Using docker for mac, things are either as worse as with virtualbox/fusion shared folders, or rather even worse.

This said, still docker for mac is nice to have services bound to localhost to ease up every development / debugging configuration ( remote debugging needs some tender care with docker-machine, no matter with PHP, ruby or Java). Also the xhyve based stack seems to run a lot more efficient, then the vmware-fusion variant, which is faster the vbox.

But having not solved the shared-folder issue is still blocking a lot of mac users and the “spreading” of docker as a development tool in general for mac users and windows users, i guess.

Thank you for the hard work anyway!

40

I can confirm the slow FS performance.

Here’s my stats:

user@Native-Mac-FS/app$ time time dd if=/dev/zero of=speedtest bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes transferred in 0.360199 secs (284287317 bytes/sec)
real    0m0.373s
user    0m0.022s
sys     0m0.288s

vs performance in the container:

root@660d6823b06f:/app$ time dd if=/dev/zero of=speedtest bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 39.5285 s, 2.6 MB/s
real    0m39.538s
user    0m0.200s
sys     0m1.990s

I can also +1 slow FS performance is preventing us from deploying Docker in a bigger way for our development environments.