Docker Community Forums

Share and learn in the Docker community.

File access in mounted volumes extremely slow, CPU bound


(Aaron Jensen) #121

Ugh, nevermind me. Somehow the one container i was testing still had a shared volume mounted to the same place: .:/myapp. and it was winning out config wise. Once I got rid of that it was fast again. Sorry for the rabbit hole.


(Eugenmayer) #122

I have created a new project called docker-sync: https://github.com/EugenMayer/docker_sync

Install it by : gem install docker-sync

It supports docker-for-mac and syncing files will not influence the container speed at all, see https://github.com/EugenMayer/docker_sync#performance

It also supports docker-machine with any hypervisor, be it Fusion/VBbox or Paralells, or others - all it needs is a running docker.

After looking at all the alternative i could find: https://github.com/EugenMayer/docker_sync#other-projects-with-similar-purpose-i-know-of
i could not fine either a solution supporting docker for mac, but nor anything which was speedy.

Hope this helps some of you. Feedback would be great, use the github issues if you like


(Yakobe) #123

@eugenmayer That sounds very interesting! I am trying to test it out but hitting a few problems:

first problem is installation:

gem install docker-sync  
ERROR:  Could not find a valid gem 'docker-sync' (>= 0) in any repository
ERROR:  Possible alternatives: docker_sync 

I installed docker_sync instead. But when i try to start i get:

docker-sync sync:start
Could not find command "sync:start".

Any help would be much appreciated. Really looking for a simple work around until the performance is nolonger an issue


(Eugenmayer) #124

Please use the github-issue queue so we do not carry this topic offtopic.

I confused myself with docker_sync and docker-sync. I removed the old gem (docker_sync) and added a new one docker-sync

Install it
gem install docker-sync

I fixed the task issue, it was a special issue when running docker-sync from not a thor directory. Sorry.
Be aware, the start syntax has been simplified

docker-sync start

I also created a simple boilerplate you can start from https://github.com/EugenMayer/docker-sync-boilerplate
This should explain how simple it is in a real case i guess


(Eugenmayer) #125

I added Unison support with 0.0.5, so you can try out both implementation named in this thread ( rsync and unison ).
You can chose, per sync point, which strategy to use - so you can also combine rsync and unison depending on the folder you want to sync for your project, which can make a lot of sense


(Thepixeldeveloper) #126

Just gave docker-sync a go. Incredible work and Symfony 3 is now loading in 300ms while in debug mode. A small note for anyone giving it a go, ensure you name the sync something new. I named mine the same as a container name and ran into some conflicts.


(Yakobe) #127

@eugenmayer You are right, i’ll keep the issues to github, sorry.
Just tried it out the new version tho. An excellent work around for now, many thanks!


(Vlad Kopaygorodsky) #128

Hi! Can you show please your docker-sync file? I’m in troubles with permissions for user. I get ‘Permission denied’ because the user is xfs, not www-data.


(Eugenmayer) #129

My absolute last comment on this in this thread, since people will get annoyed, out of good reasons, listing to arbitrary “third part tool issue” in this thread. This thread is for performance issues with docker for mac shared http://docker-sync.io is just a tool to solve that.

If you want any help on your issue, use github issues and please, read the docs.

I will not answer on any “how do i do / i have this issue with docker-sync” in here anymore, i will gladly assist you on github though.

My apologies to all listing to this “offtopic stuff” - wont happen again.


(Dave Wikoff) #130

Curious why do you need to run a command on the host - Couldn’t you mount a directory using osxfs and then rsync/unison the files to the volume?


(Aaron Jensen) #131

That’s a great idea. I wonder if the initial sync will be a little slower, but it certainly simplifies the setup. Can anyone think of any downsides?


(Jakubsacha) #132

Part of documentation says: “Adjust your docker-compose.yml as explained here”. For me this kind of requirement disqualifies that solution. It’s not portable (what if part of the team uses linux?) and feels like dirty hack.


(Paolo Mainardi) #133

please guys STOP to discuss about Docker-unison or other alternative
solutions, stay on topic!


(Tomfrost) #134

Alternative workarounds are always valuable discussion, but if a moderator could split those posts into a separate topic, that would be ideal. Even before that discussion picked up, this topic had multiple times the views and comments of any other thread in this forum, so there’s definitely a large audience here interested in hearing about progress against osxfs itself.

To that end, and hopefully to bring us back on topic: if the Docker for Mac team could afford some time to give us an update on progress being made here, the target performance metrics to hit public release, etc., that would be wonderful to hear! There’s a bit of negativity in this thread, but my team is incredibly floored to use this as long as the performance versus our current solution doesn’t prohibit their workflow :slight_smile:


(Tapajos) #135

Version 1.12.0-rc4-beta19 (build: 10258)

with volumes:

time dd if=/dev/zero of=test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 26.133 s, 3.9 MB/s
real	0m26.141s
user	0m0.130s
sys	0m1.980s

wihout volumes

time dd if=/dev/zero of=test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 0.270396 s, 379 MB/s
real 0m0.273s
user 0m0.010s
sys 0m0.260s


(Thepixeldeveloper) #136

When people test, can you specify which image you’re using. I’m interested to know if some images are affected more than others. It’d be interesting to track.


(Eugenmayer) #137

I also suggest adding read-performance, since this matters a lot and actually is one of the main reasons the applications are running that slow ( as some mentioned above )


(Phil Misiowiec) #138

Hey everyone,

Here’s what I’m doing locally to get past this problem. It’s a variation of the above but using the unison binary inside my Docker image / container without the need to run another executable on the host machine.

I have a shared volume mount pointing to /app and use Unison to sync files to /running, which is where the app server is looking. Since /running is not volume mounted, it doesn’t suffer from the performance issues that inspired this thread.

First, my app’s Docker image also installs a newer version of the unison and unison-fsmonitor libraries from a repo I found (the Ubuntu 14.04 version is a bit stale and had some issues):

RUN wget -q https://github.com/TentativeConvert/Syndicator/raw/master/unison-binaries/unison-2.48.3 -O /bin/unison \
    && chmod +x /bin/unison
RUN wget -q https://github.com/TentativeConvert/Syndicator/raw/master/unison-binaries/unison-fsmonitor -O /bin/unison-fsmonitor
    && chmod +x /bin/unison-fsmonitor

The Dockerfile also adds my local files and does an initial sync:

ADD . /app
RUN unison -auto -batch -ignore 'Path .git' -silent /app /running

My docker-compose.yml references a custom shell script:

command: ./docker/start

And my start script does this:

unison -repeat watch -auto -batch -ignore 'Path .git' /app /running &
cd /running && bundle exec rails start

Notice the unison executable is being launched as a background process.

The reason I like unison is because of its bi-directional sync. If my app generates a file that I need to look at in my code editor or Web browser, it’s automatically located where I need it to be. I haven’t run into any permissions issues or other significant problems, but curious if the above technique works consistently for others.


(Aaron Jensen) #139

@philm I tried this but it has one severe problem, at least on my machines. Try renaming a file on your host machine. That rename ultimately does not propagate. I don’t know if it is osxfuse or unison falling down in this case, but that made this approach a non-starter for me and I switched back to running unison locally and connecting over a socket.


(Phil Misiowiec) #140

Interesting - seems to work for me when doing a ‘mv’ via a docker-compose exec session. Would you mind posting your Dockerfile, docker-compose.yml, and runtime command/script (and anything else you think is helpful) either here or in a private gist? Would love to help you debug it.