File access in mounted volumes extremely slow, CPU bound

Docker Desktop
104

Same here, obviously.

The container was started using: docker run -it --rm -v /tmp:/tmp ubuntu bash.

Here’s the output of a sample dd:

$ pwd
/tmp

$ time dd if=/dev/zero of=test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB, 98 MiB) copied, 30.7729 s, 3.3 MB/s

real	0m30.791s
user	0m0.160s
sys	0m1.150s

Meanwhile, on the host I was caputring system calls with dtruss: sudo dtruss -c -d -e -f -o -p 25329 2> /tmp/dtruss.log

25329/0x150c7a:     16678     143     20 write(0x6, "\r\0", 0x1)		 = 1 0
25329/0x150c7b:     11956     163     12 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     11974     191     13 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     11999      44      6 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12005      36      4 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12013      44      6 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12026      49      8 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12036      47      7 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12046      43      7 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12053      42      6 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12061      45      6 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12071      41      6 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12079      36      6 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12087      44      6 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12096      36      7 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12104      42      6 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12113      41      5 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12121      49      5 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12129      46      6 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12139     173      7 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12145      47      4 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12153      38      5 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12165      39      6 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12170      32      4 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12178      43      6 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12187      73      6 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12199     118      7 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c80:      2080    1945     16 kevent(0x5, 0x0, 0x0)		 = 1 0
25329/0x150c80:      2102      13      6 read(0x6, "\r\n\0", 0x1000)		 = 2 0
25329/0x150c80:      2128      21     16 write(0x1, "\r\n\0", 0x2)		 = 2 0
25329/0x150c80:      2137       5      1 read(0x6, "\0", 0x1000)		 = -1 Err#35
25329/0x150c80:      2145       6      1 kevent(0x5, 0x0, 0x0)		 = 0 0
25329/0x150c7b:     12207     181      5 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12220      61     11 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12230      41      7 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12247      50      5 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12254      38      5 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12262      40      5 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12268      39      5 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12274      35      3 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12281      37      5 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12287      37      4 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12294      39      6 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12299      34      3 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12304      37      4 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12310      38      5 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12317      38      5 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12322      36      3 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12330      38      7 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12337      40      5 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12344      38      4 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12352      44      6 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12360      41      6 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12368      39      5 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12375      35      5 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12384      41      7 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12392      63      5 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12400     129      6 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12407     195      5 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12416     416      6 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12423     806      5 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12431    1614      5 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12438    2621      4 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12448    6437      8 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c7b:     12459      35      2 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c80:      2158 30790240      7 kevent(0x5, 0x0, 0x0)		 = 1 0
25329/0x150c80:      2184      96      9 read(0x6, "100000+0 records in\r\n100000+0 records out\r\n102400000 bytes (102 MB, 98 MiB) copied, 30.7729 s, 3.3 MB/s\r\n\0", 0x1000)		 = 105 0
25329/0x150c80:      2209      20     15 write(0x1, "100000+0 records in\r\n100000+0 records out\r\n102400000 bytes (102 MB, 98 MiB) copied, 30.7729 s, 3.3 MB/s\r\n\0", 0x69)		 = 105 0
25329/0x150c80:      2217       5      1 read(0x6, "\0", 0x1000)		 = -1 Err#35
25329/0x150c80:      2223       5      1 kevent(0x5, 0x0, 0x0)		 = 0 0
25329/0x150c7b:     12477      55      7 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0
25329/0x150c80:      2234    1188      7 kevent(0x5, 0x0, 0x0)		 = 1 0
25329/0x150c80:      2253       9      4 read(0x6, "\r\nreal\t0m30.791s\r\nuser\t0m0.160s\r\nsys\t0m1.150s\r\n\033]0;root@93aefffad6f2: /tmp\aroot@93aefffad6f2:/tmp# \033[K\0", 0x1000)		 = 102 0
25329/0x150c80:      2289     109     14 write(0x1, "\r\nreal\t0m30.791s\r\nuser\t0m0.160s\r\nsys\t0m1.150s\r\n\033]0;root@93aefffad6f2: /tmp\aroot@93aefffad6f2:/tmp# \033[K\0", 0x66)		 = 102 0
25329/0x150c80:      2309      90      8 read(0x6, "\0", 0x1000)		 = -1 Err#35
25329/0x150c80:      2323       9      5 kevent(0x5, 0x0, 0x0)		 = 0 0
25329/0x150c7b:     12494      38      4 select(0x0, 0x0, 0x0, 0x0, 0x700000080DE8)		 = 0 0


CALL                                        COUNT
write                                           4
kevent                                          7
read                                            7
select                                         61

As you can see, there’s really one blocking steps if I understand this correctly. At the moment, I am unable to dig deeper, but I’ll see if I can find a way.

1 Like
105

There are also two popular projects for setting up this kind of environment:

106

Version 1.12.0-rc2-beta17 (build: 9779)

Not a volume:

root@8c23cfe51d65:/tmp# time dd if=/dev/zero of=test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 0.18435 s, 555 MB/s

real	0m0.188s
user	0m0.020s
sys	0m0.160s

Mounted volume:

root@8c23cfe51d65:/tmp# cd /mounted/
root@8c23cfe51d65:/mounted# time dd if=/dev/zero of=test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 23.5597 s, 4.3 MB/s

real	0m23.564s
user	0m0.410s
sys	0m1.080s
107

Hi

Pretty much the same here, I tested a Nginx setup + PHP app with a mounted volume and serving a basic page takes ~3 seconds whereas the same page loaded without the volume (as in the container) takes 216ms.

Client:
 Version:      1.12.0-rc2
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   906eacd
 Built:        Fri Jun 17 20:35:33 2016
 OS/Arch:      darwin/amd64
 Experimental: true

Server:
 Version:      1.12.0-rc2
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   a7119de
 Built:        Wed Jun 29 10:03:33 2016
 OS/Arch:      linux/amd64
 Experimental: true
108

Docker for Mac: version: mac-v1.12.0-beta18
OS X: version 10.11.5 (build: 15F34)

With volume

root@07f6e31194b7:/tmp# pwd
/tmp
root@07f6e31194b7:/tmp# time dd if=/dev/zero of=test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB, 98 MiB) copied, 23.6226 s, 4.3 MB/s

real	0m23.626s
user	0m0.140s
sys	0m1.440s
root@07f6e31194b7:/tmp# exit
exit

without

docker run -it --rm ubuntu bash
root@bdacb4cc0ebc:/# cd tmp
root@bdacb4cc0ebc:/tmp# time dd if=/dev/zero of=test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB, 98 MiB) copied, 0.273326 s, 375 MB/s

real	0m0.276s
user	0m0.020s
sys	0m0.250s
root@bdacb4cc0ebc:/tmp#
109

I’m using unison data container for now as a hack to distribute files to containers without using mounted shared volumes. This isn’t as fast as shared volumes in ubuntu but it’s considerably more faster than using docker for mac.

Otherwise docker for mac feels really good :). I hope that the performance issues can be fixed soon.

3 Likes
110

@onnimonni

I’ve never used unison. I understand that it does bi-directional syncing, but what happens when you first start it up. I didn’t see a way to specify a source and target in your source code so I’m just not sure if the docker image files will win out or the host’s files, or even how to properly set up the volumes for it.
I did see it defaults to a /data folder, but is that for source or target?
Could you provide documentation on how to setup docker-unison to sync host directory with a docker volume?
Thanks for what you’ve done so far! This could be far easier to use than the lsyncd setup I’ve been using!

111

Oh, looks like you’ve updated the readme since I last looked at it, or my phone hid it :/. Thanks!

112

Looks cool @onnimonni, could you provide some numbers here? Thanks for sharing!

113

@cirocosta I’m non sure how to provide those in this scenario which would be as scientific as the numbers before in this conversation. Because writing is super fast on both sides but the sync is just only eventually consistent on other side.

I’m using this for running webpack with watch options inside container on project which has 13444 files. After I make changes to 1 file which triggers webpack it takes ~20-25s to run it using docker volume. Using this hack the same scenario takes ~5s.

I’m using this also for big wordpress projects and the webserver is blazingly fast using the unison sync. Setting this up is quite easy so just try it for your scenario.

The first sync just takes some time.

These are from my example project and the first sync takes about 10 seconds:

$ du -sh .                          
1.2G
$ find . -type f | wc -l            
20467
2 Likes
114

Thanks man you saved my life :smiley:

This is obviously not a solution to the problem highlighted in this topic but a awesome workaround for me until they fix it!

115

EDIT: I was doing something wrong here, I still had a osxfs mount as well (my config spanned two files, and the first declared mount won out)

@onnimonni unless I’m missing something, mounted volumes, even if synced with unison, are just as slow. I’ve got unison set up in the way you detail for docker-compose and:

# time dd if=/dev/zero of=test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 27.5858 s, 3.7 MB/s

real    0m27.589s
user    0m0.300s
sys     0m2.370s

Things are definitely fast on non-volumes, but I don’t know how to share a non-volume with another container. Your example here doesn’t even actually create/expose a volume: https://github.com/onnimonni/docker-unison#docker

Am I missing something?

116

@aaronjensen What directory are you trying to sync? I haven’t used that docker container yet, but make sure you DON’T use the default osxfs volumes at all. It really looks like you’re stilling using them. Unison does not use mounts or volumes - it’s essentially (I’m over-simplifying) a glorified bi-directional rsync.

117

EDIT: I was doing something wrong here, I still had a osxfs mount as well (my config spanned two files, and the first declared mount won out)

@derimagia Yeah, it looks like the container (which is a standalone container running unison) would only be able to handle the syncing and share that with other containers via volumes.

So… is there a way to do that w/o volumes? (I remember docker of old had data containers… pre volumes… is that still a thing?)

Or is there a way to create a non-osx-fuse volume to share between them?

118

Thats my docker-compose.yml
It’s for Drupal 8 and using unison it’s about 10 times faster than using a shared folder.
After docker compose up I start unison like this

unison code socket://127.0.0.1:5000/ -ignore 'Name {node_modules}' -ignore 'Regex .*\.idea|git|sass-cache.*' -ignore 'Path {/files/php, /files/styles}' -auto -batch

I can’t actually make unison watch all of the code because there’re too many files so I then use this command to actually watch and sync changes

unison code/sites/mysite.com/ socket://127.0.0.1:5000/sites/mysite.com/ -repeat watch -ignore 'Name {node_modules}' -ignore 'Regex .*\.idea|git|sass-cache.*' -ignore 'Path {/files/php, /files/styles}' -auto -batch

# dd if=/dev/zero of=test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 2.98254 s, 34.3 MB/s
# pwd
/var/www/web
#

` version: ‘2’

services:
  mysql:
    image: mysql:5.5
    environment:
      MYSQL_ROOT_PASSWORD: rootpasswd
      MYSQL_DATABASE: drupal
    ports:
      - "33306:3306"
    volumes:
      - mysqldata:/var/lib/mysql
      - ./conf/mysql/conf.d:/etc/mysql/conf.d
    networks:
      dev_net:

  mail:
    # https://hub.docker.com/r/mailhog/mailhog/
    image: mailhog/mailhog
    ports:
     - "1025"
     - "8025"
  unison:
    image: onnimonni/unison:2.48.4
    environment:
      - UNISON_DIR=/var/www/web
    ports:
      - "5000:5000"
    volumes:
      - webdata:/var/www/web
  web:
    build: ./build/drupal-nginx-php56x
    depends_on:
     - unison
    ports:
     - "32080:80"
     - "32443:443"
     - "32022:22"
    volumes_from:
     - unison
#    volumes:
#     - ./code/:/var/www/web

    links:
     - mysql
     - mail
    networks:
      dev_net:

# config globals

volumes:
  mysqldata:
  webdata:

networks:
  dev_net:
    driver: bridge`
119

Right here you’re mounting your local directory to docker using osxfs. Change this back to what the readme says which would just be /var/www/web

120

Actually I’m not. It’s not a folder it’s a named volume (see below # config globals) and it’s stored inside the VM or whatever that xhyve instance is called.

I think I could also use volume webdata:/var/www/web instead of volumes_from for the web service.

$ docker volume ls DRIVER VOLUME NAME ... local mysite_mysqldata local mysite_webdata

$ docker volume inspect mysite_webdata [ { "Name": "mysite_webdata", "Driver": "local", "Mountpoint": "/var/lib/docker/volumes/mysite_webdata/_data", "Labels": null, "Scope": "local" } ]

121

Ugh, nevermind me. Somehow the one container i was testing still had a shared volume mounted to the same place: .:/myapp. and it was winning out config wise. Once I got rid of that it was fast again. Sorry for the rabbit hole.

122

I have created a new project called docker-sync: https://github.com/EugenMayer/docker_sync

Install it by : gem install docker-sync

It supports docker-for-mac and syncing files will not influence the container speed at all, see https://github.com/EugenMayer/docker_sync#performance

It also supports docker-machine with any hypervisor, be it Fusion/VBbox or Paralells, or others - all it needs is a running docker.

After looking at all the alternative i could find: https://github.com/EugenMayer/docker_sync#other-projects-with-similar-purpose-i-know-of
i could not fine either a solution supporting docker for mac, but nor anything which was speedy.

Hope this helps some of you. Feedback would be great, use the github issues if you like

123

@eugenmayer That sounds very interesting! I am trying to test it out but hitting a few problems:

first problem is installation:

gem install docker-sync  
ERROR:  Could not find a valid gem 'docker-sync' (>= 0) in any repository
ERROR:  Possible alternatives: docker_sync

I installed docker_sync instead. But when i try to start i get:

docker-sync sync:start
Could not find command "sync:start".

Any help would be much appreciated. Really looking for a simple work around until the performance is nolonger an issue