Since especially this thread was dragged off topic while searching and discussion alternatives, i created this topic to separate this discussions, so people actually interested in getting osxfs fixed do not get bothered by the people actually just looking for a solution, no matter it is osxfs or something else solving this under docker for mac.
An alternative for shares under docker-for-mac is http://docker-sync.io - it does support:
Its performance matches the performance having no shares at all - its as fast as docker is, see this chart
Different strategies to sync and watch for changes: rsync/unison/unison+unox (transparent sync) and will even support NFS in the near future. See a brief description of the possible strategies here
You can chose a strategy to sync with, depending on each project or even for each container you have.
Since, like in the linked topic above, use cases do differ a lot, you cannot just ask all users to use one single strategy, like most of the other ādocker-syncā alternatives do. Docker-sync supports you choosing the right strategy depending on your project size, your project kind and your needs on transperency/speed of sync
Different advanced options to help you justifying it to your use case
sync_args and more help you introducing flags for your specific use cas
It does support uid/guid mapping to your container, to fix permission issues
It supports being used as a library for scaffolding and implementing your own project based logic, see scripting
It supports per-prroject configuration and also does keep your docker-compose.yml portable by not applying changes to it, all changes go to docker-compose-dev.yml
It supports full-stack bootstrap by starting sync and compose for you by running docker-sync-stack start - fire and forget
It does support notifications on sync so you are aware what happened when
As mentions above, there are already enough alternatives you might want to try also - most of them do not support docker for mac at all (except docker-sync-magic, which is part of docker-sync now) or do only implement one strategy to rule them all, which is not working, as a lot of people have different needs. Some of them have still performance issues or do only support one-way syncing and so on
Itās entirely possible to support NFS like Vagrant OOTB. You can already connect to the Docker for Mac VM via the tty socket and the NFS package can be installed through apk. The only thing thatās needed is to set up the export on your host in a way thatās not horrendously insecure and to have some automagic way of configuring the mount inside the DfM VM.
I wish we had legit SSH access to the DfM VM, because in that case, you could add the DfM VM to docker-machine and use docker-machine-nfs.
I understand that NFS has itās shortcomings, but for some use cases, itās still a step up from osxfs for the time being.
Well, OSXFS is explicitly not the topic in here, rather all alternatives. I tried NFS, its just way to slow and thats why i am currently no longer consider adding it to docker-sync anymore ( right now ). The new unison strategy in docker-sync, which combines it with unox, does a full transparent sync like NFS, but without any performance degrading at all.
On the bottom line, i cannot see a single advantage in using NFS over unison+unox - if you give me any good arguments, i might consider it in docker-sync in the future.
For now, we have an ultra-fast sync with rsync ( onsided ), and a very robust and fast / CPU low-profile 2 way sync with unison-unox
NFS might be too slow for you, but itās fast enough for many people. I tried Docker Sync and the project Iām on just has way too many files for it to work properly. There were also sync conflicts from Unison because our web container ships with a specific release of the project include (so that we can just pull/run the container in production). The workflow right now is that weāre mounting files over the top of that directory, but Unison complains loudly when thatās the case.
In addition, the entire point of containerizing our environment was so that our developers didnāt have to install extra things on their host (only Docker), so Docker Sync is not really an acceptable solution for us at present.
To be very clear, when I say that NFS would work well for us, Iām not suggesting that happen through Docker Sync. Iām suggesting the DfM should have an option to fall back to NFS if we donāt need filesystem notifications. Also, Docker for Windows uses SMB mounts. Itās not ideal, but even that works better for our current project than osxfs (compared Docker for Mac to Docker for Windows and DfW was blazingly fast in comparison). We have no need for inotify type things at present. We just need decent read performance, and with cachefilesd (even with a 10 second timeout), itās decent enough.
Fair enaugh, if you handle > 2GB of files, unison is slow at the start, thats right.
I fully get the point with zero-dev deps on the OSX host, but be serious, thats not possible right now. Neither docker works well enough under OSX (under linux, docker-sync is useless) but in most cases you end up installing some of the interpreters, be it ruby/php others to have a decent IDE integration, since not anything can be handled using remote-interpreters.
I really like the idea of the āclean OSX hostā, but for now, there is a lot to happen that this actually works out ( not only a good solution without something like docker-sync ). That said, i at least expected, that a lot of people will still have a proper non EOL ruby installed ( not 1.9, like in OSX ), and thats basically all you need.
Still, running zero deps, NFS most probably is a good tool and also for huge codebases ( we tracked down huge means rather size and amount, not just amount. 40k files are not issue)
mickael is very actively contributing to docker-sync and the ābestā out of docker-magic-sync is in docker-sync āunisonā. He is not used magic-sync right now also ( too unreliable )
The issue why we did not use āmagic-syncā as a strategy is, its to faulty. A lot of OSXFS fsevents are lost, so sync does not happen. So until this issue Inotify events are not / stop working from host to container is not solved, its just not usable. I know that you already had some conversation about this already with him, AFAIR, so you probably already know this.
As soon as it gets fixed, it will be part of docker-sync. When we tackle to go-rewrite even the ruby dependency would be removed, but well, thats about to discussed anyway
My point is, this is doable without local dependencies or configuration of any sort. The ideal solution for my team is a few extra lines in our docker compose file and thatās it. Magic sync - as a concept - satisfies this requirement, barring the fs events being lost (which is something that needs to be fixed/maybe has been fixed in osxfs).
Even building docker-sync in Go is a nonstarter in my book because it still requires something extra running on the host (though losing the Ruby dependency is nice).
Just following up here. https://github.com/cweagans/docker-bg-sync is a working alternative to both docker-magic-sync and docker-sync that has no host dependencies. Itās not quite as automatic as docker-magic-sync, but you only have to set it up once and it works. Iām seeing a 10x-11x speedup on my Drupal 8 project (measured via uncached page load time).
I tested cweagans/bg-sync over the weekend with Drupal 8 (including loads of additional dependencies managed by Composer), and got the exact same results (> 10x performance improvement measured by ApacheBench). The reason Iām interested in this approach above the others is the same as whatās listed in the why not use x section of the README. Namely, my entire reason for wanting to use Docker is to contain as many dependencies as possible, leaving my host machine as clean as possible. In this case, all the syncing happens from within the bg-sync container. Of course Iād prefer the filesystem latency issues fixed in OSXFS, but until then, this solution is perfect for my use.
Sure it has, unison and unox. Therefor easy-install of python and xcode + gcc for compiling unison or homebrew.
That is for sure not dependency free, that is just cherry picking. Anyway, as long as it works for you, its a go.
@cweagans oh i see, you are still using OSXFS to sync and then just syncing further to the app-container. Are you aware that this will have issues due to OSXFS not passing all file events, leading to syncs not triggered for unison?
The solution is elegant, but its not reliable, thats the reason why we yet did not implement it in docker-sync.
We believe osxfs delivers all mutating file system events (those that unison would care about) to the Linux VFS. You can read about the events that osxfs supports and how at https://docs.docker.com/docker-for-mac/osxfs/#/file-system-events. There are no outstanding partial event delivery bugs that we are aware of. If you know of existing issues with osxfs's inotify support, please report those issues to https://github.com/docker/for-mac and we will work to fix them.
@eugenmayer I canāt do anything with that report of 100% CPU use until I get a lot more details about the use case/reproduction that causes it to occur. There are known performance issues with osxfs and, recently, with the virtual block device. It would be best to supply a reproduction of the excessive CPU use or slow performance in order to help us understand if this is the same as previously reported issues or a new problem.
