File access in mounted volumes extremely slow, CPU bound

Please use the github-issue queue so we do not carry this topic offtopic.

I confused myself with docker_sync and docker-sync. I removed the old gem (docker_sync) and added a new one docker-sync

Install it
gem install docker-sync

I fixed the task issue, it was a special issue when running docker-sync from not a thor directory. Sorry.
Be aware, the start syntax has been simplified

docker-sync start

I also created a simple boilerplate you can start from https://github.com/EugenMayer/docker-sync-boilerplate
This should explain how simple it is in a real case i guess

I added Unison support with 0.0.5, so you can try out both implementation named in this thread ( rsync and unison ).
You can chose, per sync point, which strategy to use - so you can also combine rsync and unison depending on the folder you want to sync for your project, which can make a lot of sense

Just gave docker-sync a go. Incredible work and Symfony 3 is now loading in 300ms while in debug mode. A small note for anyone giving it a go, ensure you name the sync something new. I named mine the same as a container name and ran into some conflicts.

@eugenmayer You are right, i’ll keep the issues to github, sorry.
Just tried it out the new version tho. An excellent work around for now, many thanks!

Hi! Can you show please your docker-sync file? I’m in troubles with permissions for user. I get ‘Permission denied’ because the user is xfs, not www-data.

My absolute last comment on this in this thread, since people will get annoyed, out of good reasons, listing to arbitrary “third part tool issue” in this thread. This thread is for performance issues with docker for mac shared http://docker-sync.io is just a tool to solve that.

If you want any help on your issue, use github issues and please, read the docs.

I will not answer on any “how do i do / i have this issue with docker-sync” in here anymore, i will gladly assist you on github though.

My apologies to all listing to this “offtopic stuff” - wont happen again.

3 Likes

Curious why do you need to run a command on the host - Couldn’t you mount a directory using osxfs and then rsync/unison the files to the volume?

That’s a great idea. I wonder if the initial sync will be a little slower, but it certainly simplifies the setup. Can anyone think of any downsides?

Part of documentation says: “Adjust your docker-compose.yml as explained here”. For me this kind of requirement disqualifies that solution. It’s not portable (what if part of the team uses linux?) and feels like dirty hack.

1 Like

please guys STOP to discuss about Docker-unison or other alternative
solutions, stay on topic!

6 Likes

Alternative workarounds are always valuable discussion, but if a moderator could split those posts into a separate topic, that would be ideal. Even before that discussion picked up, this topic had multiple times the views and comments of any other thread in this forum, so there’s definitely a large audience here interested in hearing about progress against osxfs itself.

To that end, and hopefully to bring us back on topic: if the Docker for Mac team could afford some time to give us an update on progress being made here, the target performance metrics to hit public release, etc., that would be wonderful to hear! There’s a bit of negativity in this thread, but my team is incredibly floored to use this as long as the performance versus our current solution doesn’t prohibit their workflow :slight_smile:

3 Likes

Version 1.12.0-rc4-beta19 (build: 10258)

with volumes:

time dd if=/dev/zero of=test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 26.133 s, 3.9 MB/s
real	0m26.141s
user	0m0.130s
sys	0m1.980s

wihout volumes

time dd if=/dev/zero of=test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 0.270396 s, 379 MB/s
real 0m0.273s
user 0m0.010s
sys 0m0.260s

When people test, can you specify which image you’re using. I’m interested to know if some images are affected more than others. It’d be interesting to track.

I also suggest adding read-performance, since this matters a lot and actually is one of the main reasons the applications are running that slow ( as some mentioned above )

1 Like

Hey everyone,

Here’s what I’m doing locally to get past this problem. It’s a variation of the above but using the unison binary inside my Docker image / container without the need to run another executable on the host machine.

I have a shared volume mount pointing to /app and use Unison to sync files to /running, which is where the app server is looking. Since /running is not volume mounted, it doesn’t suffer from the performance issues that inspired this thread.

First, my app’s Docker image also installs a newer version of the unison and unison-fsmonitor libraries from a repo I found (the Ubuntu 14.04 version is a bit stale and had some issues):

RUN wget -q https://github.com/TentativeConvert/Syndicator/raw/master/unison-binaries/unison-2.48.3 -O /bin/unison \
    && chmod +x /bin/unison
RUN wget -q https://github.com/TentativeConvert/Syndicator/raw/master/unison-binaries/unison-fsmonitor -O /bin/unison-fsmonitor
    && chmod +x /bin/unison-fsmonitor

The Dockerfile also adds my local files and does an initial sync:

ADD . /app
RUN unison -auto -batch -ignore 'Path .git' -silent /app /running

My docker-compose.yml references a custom shell script:

command: ./docker/start

And my start script does this:

unison -repeat watch -auto -batch -ignore 'Path .git' /app /running &
cd /running && bundle exec rails start

Notice the unison executable is being launched as a background process.

The reason I like unison is because of its bi-directional sync. If my app generates a file that I need to look at in my code editor or Web browser, it’s automatically located where I need it to be. I haven’t run into any permissions issues or other significant problems, but curious if the above technique works consistently for others.

@philm I tried this but it has one severe problem, at least on my machines. Try renaming a file on your host machine. That rename ultimately does not propagate. I don’t know if it is osxfuse or unison falling down in this case, but that made this approach a non-starter for me and I switched back to running unison locally and connecting over a socket.

Interesting - seems to work for me when doing a ‘mv’ via a docker-compose exec session. Would you mind posting your Dockerfile, docker-compose.yml, and runtime command/script (and anything else you think is helpful) either here or in a private gist? Would love to help you debug it.

@philm yes, that works fine, but can you try from OSX? The issue is when going through osxfuse. If you do a mv from within the container itself via exec it works as you’d expect.

Gotcha - I’ve confirmed the issue. Let me do some digging.

No significant find yet - if I cp the file and rm the original on the host, or if I mv the file and then touch it, it works fine of course.