Docker Community Forums

Share and learn in the Docker community.

File access in mounted volumes extremely slow, CPU bound


(Tomfrost) #205

Earlier in this thread, I suggested making it an option to mount volumes via NFS, and expose a service that could receive file-change notifications out-of-band.

As this has continued to be a problem for my team, I wrote exactly that. FS-EventBridge is a tiny, statically compiled Rust app that opens a TCP server to receive streaming file change notifications. On the host mac, one can run fsbridge to listen for and stream fsevents from any folder into the eventbridge server. We’ve also integrated it into DevLab (our little docker-compose alternative to support a different kind of workflow) so project folders automatically stream their events. For anyone interested in testing this setup, I put together a bootstrapper called DevBox that will configure it all for you. Just uninstall docker-machine first if you already have it installed from somewhere other than brew.

fsbridge is perfectly capable of monitoring large folders (I had it running for my entire user folder at one point – definitely not ideal because it’ll pick up ~/Library and such, but everything still ran fine). So, with a few more lines of code to mitigate any connection issues and exclude unnecessary folders, this could easily be run on startup though a LaunchAgent or otherwise, maybe even pointed to a user-specified projects/workspace folder. That way anyone’s tooling, including docker-compose, would continue working fine without the need to manually run the fsbridge client first.

This setup is screaming fast, giving us near-native performance with two-way volumes and instant inotify events, with any root-created files on the VM’s mounted volume getting the UID and GID of the mac user, all with barely a blip on the CPU usage graph during filesystem-intensive operations, and no additional containers running. It’s the best pieces of all the different solutions discussed here. Please consider this solution for the official app!


(Bgross27) #206

Significant problem for me as well. Processes are all disk-access bound, when they should be CPU-bound… I’ll understand in beta, but I’ll defect if this is still there in production


(Cytopia) #207

@tomfrost

Thanks for the effort putting this all together.

However, this is still just a work-around which shouldn’t even be needed.
It will actually add much more complexity to the machines.

The whole thing needs to be fixed on docker/osx side.


(Tomfrost) #208

@cytopia Apologies if I misrepresented my stance-- I wasn’t suggesting the Docker team make a standard of so many moving parts. I was just trying to make two points:

  • Filesharing over NFS + streaming file change notifications out-of-band is incredibly fast and reliable with little-to-no resource overhead compared to osxfs.

  • That can easily be implemented in the official Docker for Mac without as many moving parts, due to the ability to widely monitor large folders with the same low overhead.

Dinghy, for example, uses a localized NFS implementation that doesn’t rely on the underlying operating system’s NFS daemon or configuration. Docker for Mac could embed the same, along with the file monitor, and it would be entirely self-contained.


(Fulcrum) #209

So I have benchmarked running a full optimized Drupal stack (mariadb/redis/php-fpm/nginx/varnish/haproxy) on both Docker for Mac and DockerRoot/Xhyve (https://github.com/ailispaw/docker-root-xhyve) which is what we have been using for a year+ since the earlier solution of Docker on the Mac using VirtualBox suffered from the same exact file performance problem because of the terrible VirtualBox vboxsf driver.

Loading a Drupal home page here are our results:

DockerRoot/Xhyve ~ 170ms
Docker for Mac ~ 5000ms

That is a slowdown of 2950%!!! That is pretty much unusable and not very compelling compared to technology over a year old. I was hoping we would be able to simplify our dev environment with Docker for Mac, and if it was only a 100% slowdown I might consider it, but this really just has my engineers spinning their heels.

I would really like to see Docker for Mac offer a NFS volume mount option.


(Mikeball) #210

That’s an understatement, I’m seeing 150-200 times slower times attempting to generate a jar. Docker for mac is simply unusable for development work, even 6 months after this was know and 2 months after it was released.


(Fulcrum) #211

I have successfully gotten Docker for Mac to hit the speeds we were seeing with DockerRoot/Xhyve without the use of any 3rd party syncing. It installs the Alpine Linux apk nfs-utils package on the Docker for Mac Moby vm. You can find the solution here:


(Tomfrost) #212

@fulcrum Your link is 404’ing – is that a public repo?

I think we were sharing a brain on this one-- I’ve been working with that solution as well, but hit a wall when I discovered Moby was triggering a restart after unmounting /Users. I’d love to see how you handled this, and perhaps submit a PR to integrate FS-EventBridge to get the inotify events back.


(Fulcrum) #213

@tomfrost, made the repo public, sorry about that. I actually decided to not unmount /Users, but just mount it under a different mount point.


(Donnykurnia) #214

@fulcrum do you know how to make the nfs mount permanent (automatically mounted when moby booting)?


(Fulcrum) #215

@donnykurnia, I am using a wrapper script to bring up D4M when I want it.

I have looked into trying to make it permanent, but unless there is a boot script like @tomfrost asked in another thread, I think it would have to be accomplished with a bootstrap container that is flagged with restart and calls out to the Mac perhaps with curl when it starts up, and then the Mac would have to mount the NFS again in the moby VM. It feels a bit convoluted, but I think it could satisfy the use case. My concern would be more moving parts to break.


(Yannisc) #216

Seems interesting. Could you provide more information on how this could be used along with docker-compose?


(Fulcrum) #217

@yannisc please see the git repo: https://github.com/IFSight/d4m-nfs


(Yannisc) #218

The examples you added are very helpful! thanks!


(Cytopia) #219

@fulcrum is this also going to work with docker-compose with multiple containers and multiple mounts per container?

Anyone willing to help me get this set up for: https://github.com/cytopia/devilbox


(Eugenmayer) #220

@fulcrum your findings with the Drupal “slowness” was the main reason to create http://docker-sync.io - as far as i can say and tested, als NFS will lead to enormous bottlenecks when developing with Drupal - due to the read performance. Try and compare some drush commands with NFS shared to no shares at all, in general, that should be about 3,5 times faster.

This is a huge impact when your cc all takes about 21 seconds instead of 7, if you need to repeat it for css/theme development what so ever.

NFS will most probably not suite your “Drupal needs”, that why unison/rsync works a lot better here, since it is the native performance.


(Fulcrum) #221

@cytopia this does work with multiple containers, we use 7 with in our docker-compose.yml and they each have anywhere between 3 and 9 mount a piece.


(Cytopia) #222

@fulcrum Thanks. I will try it out.

Meanwhile, I am still waiting for some status update from mods/devs… anyone else has some insights about this issue?


(Fulcrum) #223

@eugenmayer I think that sync solutions are great for a good number of use cases. For my engineers, since they have to be able to work on quite a number of fully different code bases, this can start putting the files to sync into 250k range.

As far as drush, we have to bootstrap it with a wrapper script to work with some of our performance enhancements. The uli command takes about 5 seconds, cc takes about 10 secs.


(Rbayliss) #224

We’ve been using Vagrant (Virtualbox) + NFS for years for Drupal development, and performance has been acceptable, so I imagine it would be fine here as well. It’s important to us that the volume doesn’t double the disk space required (which I believe docker-sync currently does).

For the Docker folks, here’s a sample Dockerfile illustrating the problem. These numbers are actual output I got on 1.12.2-beta28.

Finished stat x 1000.  Local: 2ms, Mounted: 245ms (11156% difference)
Finished open x 1000.  Local: 3ms, Mounted: 778ms (19496% difference)
Finished write x 1000.  Local: 1ms, Mounted: 249ms (14827% difference)