Docker Community Forums

Share and learn in the Docker community.

File access in mounted volumes extremely slow, CPU bound


(Eugenesia) #192

I totally missed the “Opcache” workaround hidden in this reply. I’m also a Drupal developer experiencing crippling slowness when doing drush cc all.

I’ve enabled Opcache on PHP and it seems to speed things up considerably. Probably because it reduces file reads.

On a development machine I have set opcache.revalidate_freq=60, which means the opcache is only valid for 60 seconds. I can’t see the effects of code changes immediately and have to wait for 60s, but I think it is tolerable.


(Sysradium) #193

I also spotted that OSXFS process is being in the top of activity monitor. I was writing a typical python + django app where one of the containers was a buil-in debug webserver, which watches for code changes and restarts if necessary.

Turns out that’s the container which drained my battery. After I stopped it osxfs process disappeared from a top.
Is there a way to have a watcher but not kill the CPU? Or maybe Docker is going to do something about that?


(Trousered) #194

I suspect i’m seeing similar performance slowdown with Docker for Mac

Version 1.12.0-a (build: 11213)

I’m seeing differences when running pgbench_tools (https://github.com/gregs1104/pgbench-tools)

Straight postgres container with datavolume ( -v “…/postgresdatadocker:/var/lib/postgresql/data” ) :

set | scale | test | script | clients | workers | limit | tps | avg_latency | 90%< | max_latency | trans
-----±------±-----±-----------±--------±--------±------±-----±------------±------±------------±-------
1 | 1 | 1 | select.sql | 1 | 1 | | 1411 | 0.704 | 0.801 | 35 | 84691
1 | 1 | 2 | select.sql | 2 | 1 | | 2337 | 0.851 | 1.041 | 44 | 140201
1 | 1 | 3 | select.sql | 4 | 1 | | 2603 | 1.42 | 1.958 | 45 | 156191
1 | 1 | 4 | select.sql | 8 | 1 | | 2671 | 2.983 | 3.76 | 44 | 160284
1 | 1 | 5 | select.sql | 16 | 1 | | 0 | | | |
1 | 1 | 6 | select.sql | 32 | 1 | | 0 | | | |
(6 rows)

Versus Virtualbox Ubuntu VM with Docker postgres container and NFS share :

set | scale | test | script | clients | workers | limit | tps | avg_latency | 90%< | max_latency | trans
-----±------±-----±-----------±--------±--------±------±-----±------------±------±------------±-------
1 | 1 | 1 | select.sql | 1 | 1 | | 5259 | 0.187 | 0.227 | 156 | 316306
1 | 1 | 7 | select.sql | 1 | 1 | | 0 | | | |
1 | 1 | 2 | select.sql | 2 | 1 | | 5915 | 0.335 | 0.601 | 156 | 354903
1 | 1 | 3 | select.sql | 4 | 1 | | 7040 | 0.565 | 1.209 | 203 | 422412
1 | 1 | 4 | select.sql | 8 | 1 | | 8791 | 0.906 | 1.352 | 285 | 527490
1 | 1 | 5 | select.sql | 16 | 1 | | 0 | | | |
1 | 1 | 6 | select.sql | 32 | 1 | | 0 | | | |
(7 rows)


(Tobias Munk) #195

I created an issue here: https://github.com/docker/for-mac/issues/77


(Jeffery Utter) #197

@dsheets I’m wondering if there is any update to this in the past two months?

I am running 1.12.1-beta26.1 and it still seems to have this problem. I am working on a large ruby project and the time it takes to start running tests is measured in minutes, whereas my coworkers working in a VM measures it in seconds. I would love to keep using docker-for-mac, but if reasonable performance is too far out. I may have to switch back to a VM.

Thanks.


(King'ori Maina) #198

Was in the same situation (large Ruby app) … use http://docker-sync.io for now.


(Jeffery Utter) #199

Yes, I’ve been meaning to try it out. In the meantime I’ve switched to docker-machine and am getting about a 5x speedup. There are some features like user remapping which seem very handy.

What I was more hoping for though is some kind of Idea on the progress of having this solved natively. Perhaps some github issues to watch?


(Xaviermoon) #200

Any update to this issue?


(Cytopia) #201

Hi everybody,

same issue here.

Is there already a status update/estimate you could give?
When might this be fixed?


(Bradcondo) #202

Would also appreciate an update. I had to revert back to using a VirtualBox VM due to volume performance.


(Jvlmp) #203

+1
Having issues with the slow performance as well.


(Cytopia) #204

@dsheets Could somebody from the Docker staff please provide some feedback here.


(Tomfrost) #205

Earlier in this thread, I suggested making it an option to mount volumes via NFS, and expose a service that could receive file-change notifications out-of-band.

As this has continued to be a problem for my team, I wrote exactly that. FS-EventBridge is a tiny, statically compiled Rust app that opens a TCP server to receive streaming file change notifications. On the host mac, one can run fsbridge to listen for and stream fsevents from any folder into the eventbridge server. We’ve also integrated it into DevLab (our little docker-compose alternative to support a different kind of workflow) so project folders automatically stream their events. For anyone interested in testing this setup, I put together a bootstrapper called DevBox that will configure it all for you. Just uninstall docker-machine first if you already have it installed from somewhere other than brew.

fsbridge is perfectly capable of monitoring large folders (I had it running for my entire user folder at one point – definitely not ideal because it’ll pick up ~/Library and such, but everything still ran fine). So, with a few more lines of code to mitigate any connection issues and exclude unnecessary folders, this could easily be run on startup though a LaunchAgent or otherwise, maybe even pointed to a user-specified projects/workspace folder. That way anyone’s tooling, including docker-compose, would continue working fine without the need to manually run the fsbridge client first.

This setup is screaming fast, giving us near-native performance with two-way volumes and instant inotify events, with any root-created files on the VM’s mounted volume getting the UID and GID of the mac user, all with barely a blip on the CPU usage graph during filesystem-intensive operations, and no additional containers running. It’s the best pieces of all the different solutions discussed here. Please consider this solution for the official app!


(Bgross27) #206

Significant problem for me as well. Processes are all disk-access bound, when they should be CPU-bound… I’ll understand in beta, but I’ll defect if this is still there in production


(Cytopia) #207

@tomfrost

Thanks for the effort putting this all together.

However, this is still just a work-around which shouldn’t even be needed.
It will actually add much more complexity to the machines.

The whole thing needs to be fixed on docker/osx side.


(Tomfrost) #208

@cytopia Apologies if I misrepresented my stance-- I wasn’t suggesting the Docker team make a standard of so many moving parts. I was just trying to make two points:

  • Filesharing over NFS + streaming file change notifications out-of-band is incredibly fast and reliable with little-to-no resource overhead compared to osxfs.

  • That can easily be implemented in the official Docker for Mac without as many moving parts, due to the ability to widely monitor large folders with the same low overhead.

Dinghy, for example, uses a localized NFS implementation that doesn’t rely on the underlying operating system’s NFS daemon or configuration. Docker for Mac could embed the same, along with the file monitor, and it would be entirely self-contained.


(Fulcrum) #209

So I have benchmarked running a full optimized Drupal stack (mariadb/redis/php-fpm/nginx/varnish/haproxy) on both Docker for Mac and DockerRoot/Xhyve (https://github.com/ailispaw/docker-root-xhyve) which is what we have been using for a year+ since the earlier solution of Docker on the Mac using VirtualBox suffered from the same exact file performance problem because of the terrible VirtualBox vboxsf driver.

Loading a Drupal home page here are our results:

DockerRoot/Xhyve ~ 170ms
Docker for Mac ~ 5000ms

That is a slowdown of 2950%!!! That is pretty much unusable and not very compelling compared to technology over a year old. I was hoping we would be able to simplify our dev environment with Docker for Mac, and if it was only a 100% slowdown I might consider it, but this really just has my engineers spinning their heels.

I would really like to see Docker for Mac offer a NFS volume mount option.


(Mikeball) #210

That’s an understatement, I’m seeing 150-200 times slower times attempting to generate a jar. Docker for mac is simply unusable for development work, even 6 months after this was know and 2 months after it was released.


(Fulcrum) #211

I have successfully gotten Docker for Mac to hit the speeds we were seeing with DockerRoot/Xhyve without the use of any 3rd party syncing. It installs the Alpine Linux apk nfs-utils package on the Docker for Mac Moby vm. You can find the solution here:


(Tomfrost) #212

@fulcrum Your link is 404’ing – is that a public repo?

I think we were sharing a brain on this one-- I’ve been working with that solution as well, but hit a wall when I discovered Moby was triggering a restart after unmounting /Users. I’d love to see how you handled this, and perhaps submit a PR to integrate FS-EventBridge to get the inotify events back.