File access in mounted volumes extremely slow, CPU bound

Docker Desktop
225

@fulcrum That is brilliant! I did not know we could reach the moby vm like that!

Did you make some performance tests yourself? I am pretty close to get rid of my docker-machine with your approach.

Here are my results, docker-machine is still twice better but we are more than 10times faster than the native mount.

On the MAC: (obviously FAST)

$ time dd if=/dev/zero of=speedtest bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes transferred in 0.238384 secs (429559016 bytes/sec)
dd if=/dev/zero of=speedtest bs=1024 count=100000 0.02s user 0.22s system 94% cpu 0.251 total

On a simple container (obviously FAST too)

$ docker run -it --rm ubuntu /bin/bash
root@4dfac39eb311:/# time dd if=/dev/zero of=speedtest bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB, 98 MiB) copied, 0.291937 s, 351 MB/s

real 0m0.297s
user 0m0.000s
sys 0m0.000s

On the native docker for mac Mount: (REALLY SLOW)

$ docker run -it --rm -v /Users/Plopix/DOCKFILES/:/plop ubuntu /bin/bash
root@a16f23368c5d:/# cd /plop/
root@a16f23368c5d:/plop# time dd if=/dev/zero of=speedtest bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB, 98 MiB) copied, 25.389 s, 4.0 MB/s

real 0m25.393s
user 0m0.000s
sys 0m0.000s

With my previous Docker machine using the same NFS approach: (GOOD)

$ docker run -it --rm -v /data/DOCKER_SOURCES:/plop ubuntu /bin/bash
root@a8f7d34119fd:/# cd /plop/
root@a8f7d34119fd:/plop# time dd if=/dev/zero of=speedtest bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 0.685166 s, 149 MB/s

real 0m0.689s
user 0m0.020s
sys 0m0.110s

With the @fulcrum NFS approach: (a mount from Moby to the mac) => (LESS GOOD)

$ docker run -it --rm -v /data/DOCKER_SOURCES:/plop ubuntu /bin/bash
root@23ebcb36f0b6:/# cd /plop/
root@23ebcb36f0b6:/plop# time dd if=/dev/zero of=speedtest bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB, 98 MiB) copied, 1.56219 s, 65.5 MB/s

real 0m1.570s
user 0m0.000s
sys 0m0.170s

I am going to work couple of days with that on big Symfony and eZ Publish/Platform projects to see if it is convenient enough.

@fulcrum, others? do you have the same results?

Thank you!

EDIT: to me docker-sync is great but… it’s too much configuration and also too many duplication of files.

226

Thats a good point - of course docker sync will double the size your data uses.

I guess, thats a tradeof, either you have this “VERY FAST” performance with docker-sync, since it is like native, but with double the size of data

or you have the “GOOD” performance but with no data duplication.

It really depends on the case i guess, what will make sense, but the point is pretty valid.

227

I had different results

MBP - Mid 2014
MacOS 10.12
Docker 1.12.1 (12133)

FileIO (sysbench)

Simple docker
$ docker run -d --name file_0 ubuntu tail -f /var/log/lastlog
$ time docker exec -it file_0 bash -c "cd /sysbench; sysbench --test=fileio --file-num=1024 --file-total-size=1G prepare"
bash: line 0: cd: /sysbench: No such file or directory
sysbench 0.4.12: multi-threaded system evaluation benchmark

1024 files, 1024Kb each, 1024Mb total
Creating files for the test...

real    0m34.149s
user    0m0.010s
sys    0m0.010s

$ time docker exec -it file_0 bash -c "cd /sysbench; sysbench --test=fileio --file-num=1024 --file-total-size=1G --file-test-mode=rndrw --init-rng=on --max-time=300 --max-requests=0 run"
bash: line 0: cd: /sysbench: No such file or directory
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1
Initializing random number generator from timer.


Extra file open flags: 0
1024 files, 1Mb each
1Gb total file size
Block size 16Kb
Number of random requests for random IO: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Threads started!
Time limit exceeded, exiting...
Done.

Operations performed:  44760 Read, 29840 Write, 763091 Other = 837691 Total
Read 699.38Mb  Written 466.25Mb  Total transferred 1.1383Gb  (3.8854Mb/sec)
  248.67 Requests/sec executed

Test execution summary:
    total time:                          300.0017s
    total number of events:              74600
    total time taken by event execution: 7.5445
    per-request statistics:
         min:                                  0.00ms
         avg:                                  0.10ms
         max:                                  6.49ms
         approx.  95 percentile:               0.73ms

Threads fairness:
    events (avg/stddev):           74600.0000/0.00
    execution time (avg/stddev):   7.5445/0.00

real    5m0.150s
user    0m0.012s
sys    0m0.013s

Docker Volume
$ docker run -d --name file_1 -v sysbench:/sysbench ubuntu tail -f /var/log/lastlog
$ time docker exec -it file_1 bash -c "cd /sysbench; sysbench --test=fileio --file-num=1024 --file-total-size=1G prepare"
sysbench 0.4.12: multi-threaded system evaluation benchmark

1024 files, 1024Kb each, 1024Mb total
Creating files for the test...

real    0m39.605s
user    0m0.011s
sys    0m0.011s
$ time docker exec -it file_1 bash -c "cd /sysbench; sysbench --test=fileio --file-num=1024 --file-total-size=1G --file-test-mode=rndrw --init-rng=on --max-time=300 --max-requests=0 run"
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1
Initializing random number generator from timer.


Extra file open flags: 0
1024 files, 1Mb each
1Gb total file size
Block size 16Kb
Number of random requests for random IO: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Threads started!
Time limit exceeded, exiting...
Done.

Operations performed:  45960 Read, 30640 Write, 784070 Other = 860670 Total
Read 718.12Mb  Written 478.75Mb  Total transferred 1.1688Gb  (3.9896Mb/sec)
  255.33 Requests/sec executed

Test execution summary:
    total time:                          300.0024s
    total number of events:              76600
    total time taken by event execution: 11.8245
    per-request statistics:
         min:                                  0.00ms
         avg:                                  0.15ms
         max:                                  8.75ms
         approx.  95 percentile:               0.84ms

Threads fairness:
    events (avg/stddev):           76600.0000/0.00
    execution time (avg/stddev):   11.8245/0.00


real    5m0.143s
user    0m0.011s
sys    0m0.011s

Host directory
$ docker run -d --name file_2 -v $(pwd)/sysbench:/sysbench ubuntu tail -f /var/log/lastlog
$ time docker exec -it file_2 bash -c "cd /sysbench; sysbench --test=fileio --file-num=1024 --file-total-size=1G prepare"
sysbench 0.4.12: multi-threaded system evaluation benchmark

1024 files, 1024Kb each, 1024Mb total
Creating files for the test...

real    0m24.376s
user    0m0.009s
sys    0m0.009s
$ time docker exec -it file_2 bash -c "cd /sysbench; sysbench --test=fileio --file-num=1024 --file-total-size=1G --file-test-mode=rndrw --init-rng=on --max-time=300 --max-requests=0 run"
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1
Initializing random number generator from timer.


Extra file open flags: 0
1024 files, 1Mb each
1Gb total file size
Block size 16Kb
Number of random requests for random IO: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Threads started!
Time limit exceeded, exiting...
Done.

Operations performed:  1193797 Read, 795865 Write, 20373504 Other = 22363166 Total
Read 18.216Gb  Written 12.144Gb  Total transferred 30.36Gb  (103.63Mb/sec)
 6632.20 Requests/sec executed

Test execution summary:
    total time:                          300.0002s
    total number of events:              1989662
    total time taken by event execution: 284.0402
    per-request statistics:
         min:                                  0.00ms
         avg:                                  0.14ms
         max:                                 94.55ms
         approx.  95 percentile:               0.39ms

Threads fairness:
    events (avg/stddev):           1989662.0000/0.00
    execution time (avg/stddev):   284.0402/0.00


real    5m1.179s
user    0m0.011s
sys    0m0.011s
228

I made the same test with mysql and sysbench
Simple
$ docker run -d --name mysql_0 -p 3306:3306 -it -e MYSQL_ROOT_PASSWORD=password mysql:5.6
$ time docker exec -it mysql_0 sysbench --test=oltp --oltp-table-size=1000000 --mysql-db=test --mysql-user=root --mysql-password=password prepare
sysbench 0.4.12: multi-threaded system evaluation benchmark

No DB drivers specified, using mysql
Creating table 'sbtest'...
Creating 1000000 records in table 'sbtest'...

real    0m22.419s
user    0m0.009s
sys    0m0.010s

$ time docker exec -it mysql_0 sysbench --test=oltp --oltp-table-size=1000000 --mysql-db=test --mysql-user=root --mysql-password=password run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

No DB drivers specified, using mysql
Running the test with following options:
Number of threads: 1

Doing OLTP test.
Running mixed OLTP test
Using Special distribution (12 iterations,  1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Maximum number of requests for OLTP test is limited to 10000
Threads started!
Done.

OLTP test statistics:
    queries performed:
        read:                            140000
        write:                           50000
        other:                           20000
        total:                           210000
    transactions:                        10000  (59.24 per sec.)
    deadlocks:                           0      (0.00 per sec.)
    read/write requests:                 190000 (1125.50 per sec.)
    other operations:                    20000  (118.47 per sec.)

Test execution summary:
    total time:                          168.8131s
    total number of events:              10000
    total time taken by event execution: 168.6915
    per-request statistics:
         min:                                  6.80ms
         avg:                                 16.87ms
         max:                                321.43ms
         approx.  95 percentile:              31.28ms

Threads fairness:
    events (avg/stddev):           10000.0000/0.00
    execution time (avg/stddev):   168.6915/0.00


real    2m49.079s
user    0m0.010s
sys    0m0.011s

Docker Volume
$ docker run -d --name mysql_1 -v mysql-test:/var/lib/mysql -p 3316:3306 -it -e MYSQL_ROOT_PASSWORD=password mysql:5.6
$ time docker exec -it mysql_1 sysbench --test=oltp --oltp-table-size=1000000 --mysql-db=test --mysql-user=root --mysql-password=password prepare
sysbench 0.4.12: multi-threaded system evaluation benchmark

No DB drivers specified, using mysql
Creating table 'sbtest'...
Creating 1000000 records in table 'sbtest'...

real    0m28.369s
user    0m0.011s
sys    0m0.012s


$ time docker exec -it mysql_1 sysbench --test=oltp --oltp-table-size=1000000 --mysql-db=test --mysql-user=root --mysql-password=password run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

No DB drivers specified, using mysql
Running the test with following options:
Number of threads: 1

Doing OLTP test.
Running mixed OLTP test
Using Special distribution (12 iterations,  1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Maximum number of requests for OLTP test is limited to 10000
Threads started!
Done.

OLTP test statistics:
    queries performed:
        read:                            140000
        write:                           50000
        other:                           20000
        total:                           210000
    transactions:                        10000  (65.36 per sec.)
    deadlocks:                           0      (0.00 per sec.)
    read/write requests:                 190000 (1241.91 per sec.)
    other operations:                    20000  (130.73 per sec.)

Test execution summary:
    total time:                          152.9906s
    total number of events:              10000
    total time taken by event execution: 152.8872
    per-request statistics:
         min:                                  6.84ms
         avg:                                 15.29ms
         max:                                172.75ms
         approx.  95 percentile:              23.37ms

Threads fairness:
    events (avg/stddev):           10000.0000/0.00
    execution time (avg/stddev):   152.8872/0.00


real    2m33.233s
user    0m0.012s
sys    0m0.013s

Host dir
$ docker run -d --name mysql_2 -v /Users/docker/mysql:/var/lib/mysql -p 3326:3306 -it -e MYSQL_ROOT_PASSWORD=password mysql:5.6
$ time docker exec -it mysql_2 sysbench --test=oltp --oltp-table-size=1000000 --mysql-db=test --mysql-user=root --mysql-password=password prepare
sysbench 0.4.12: multi-threaded system evaluation benchmark

No DB drivers specified, using mysql
Creating table 'sbtest'...
Creating 1000000 records in table 'sbtest'...

real    0m15.752s
user    0m0.010s
sys    0m0.011s


$ time docker exec -it mysql_2 sysbench --test=oltp --oltp-table-size=1000000 --mysql-db=test --mysql-user=root --mysql-password=password run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

No DB drivers specified, using mysql
Running the test with following options:
Number of threads: 1

Doing OLTP test.
Running mixed OLTP test
Using Special distribution (12 iterations,  1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Maximum number of requests for OLTP test is limited to 10000
Threads started!
Done.

OLTP test statistics:
    queries performed:
        read:                            140000
        write:                           50000
        other:                           20000
        total:                           210000
    transactions:                        10000  (258.21 per sec.)
    deadlocks:                           0      (0.00 per sec.)
    read/write requests:                 190000 (4905.92 per sec.)
    other operations:                    20000  (516.41 per sec.)

Test execution summary:
    total time:                          38.7287s
    total number of events:              10000
    total time taken by event execution: 38.6561
    per-request statistics:
         min:                                  2.36ms
         avg:                                  3.87ms
         max:                               1047.34ms
         approx.  95 percentile:               5.26ms

Threads fairness:
    events (avg/stddev):           10000.0000/0.00
    execution time (avg/stddev):   38.6561/0.00


real    0m38.949s
user    0m0.011s
sys    0m0.011s

The host directory mounted on the docker had be best performance

229

Really confused now

$ docker run —rm ubuntu bash -c "mkdir /speedtest; time dd if=/dev/zero of=/speedtest/out bs=1024 count=100000"
100000+0 records in
100000+0 records out
102400000 bytes (102 MB, 98 MiB) copied, 0.185211 s, 553 MB/s
real    0m0.186s
user    0m0.000s
sys     0m0.180s

$ docker run —rm -v $(pwd):/speedtest ubuntu bash -c "mkdir /speedtest; time dd if=/dev/zero of=/speedtest/out bs=1024 count=100000"
mkdir: cannot create directory '/speedtest': File exists
100000+0 records in
100000+0 records out
102400000 bytes (102 MB, 98 MiB) copied, 27.2135 s, 3.8 MB/s
real    0m27.218s
user    0m0.130s
sys     0m1.740s
230

github link is broken :frowning:

231

@gianni I can view the github link https://github.com/IFSight/d4m-nfs in an incognito window just fine and can see that people have been successfully pulling from it. Can you specify exactly what link you are trying?

232

What I’ve found so far is that large files are ok, it’s small files that suffer. I’m currently working on an image that allows users to download their Xbox Live gamedvr and screenshots. If I fopen the download URL and write the data as it is received it’s incredibly slow. If I read the file into memory then write out the file (around 10-15MB each) then it’s fine.

I also did an image for MySQL with innobackupex. When running a backup or restore of the large innodb files it’s fine even if moving files between two mounted/mapped directories.

233

Just spent the whole weekend trying to move all my development to Docker for Mac. As my goal was to keep a clean host machine I feel rather disappointed as it feels like that we are out of one hole (permission issues while using shared volumes) and straight into new one. I understand this is open source tool and I was not expecting everything to be fine and dandy. But seeing official replies saying that we should’t expect any serious improvements with the stable release makes me feel scammed by those who were advertising how this is a new, great way to develop software on OSX. I’d wish they would put a large, red text at the download page saying “PLEASE BE ADVISED, UNUSABLE PERFORMANCE FOR DEVELOPMENT”.

2 Likes
234

@ejekabsons, using docker-sync (by @eugenmayer) or nfs sync (by @fulcrum) makes it pretty fast and certainly usable.

235

Has anyone tested this out?

In the new beta release:

osxfs: switch to libev/kqueue to improve latency

236

Performance for host:container shared volumes is just as bad is it’s ever been.

1 Like
237

I did some research myself;
One of my projects (Large PHP project with a lot of files):

d4m-nfs	1.60s
osxfs	4.93s

I loaded the homepage 10 times, and calculated the average loading time.

The problem is that I did not benched before the latest beta update, it feels alot faster then before.

Conclusion for now: I’m sticking with d4m-nfs.

238

Hasn’t improved with the latest beta.

1 Like
239

Confirmed also on latest Docker Beta v1.12.3-beta30 (13798)

still testing using the original posted command:

time dd if=/dev/zero of=test.dat bs=1024 count=100000

direct on MacOS Sierra host :
real 0m1.756s
user 0m0.023s
sys 0m0.243s

Inside Docker Container /tmp dir :
real 0m0.522s
user 0m0.010s
sys 0m0.170s

Inside Docker Container Volume mount :
real 0m28.041s
user 0m0.170s
sys 0m2.030s

Basically the same as ever for this particular read/write issue I think.

240

I am experiencing the same issue with docker mac 1.12.3-beta30.1 - brutally slow compared with docker machine on linux which we are using as the testing host. Its strange as it wasn’t too bad, and it does seem to vary. I have not been able to put a pattern to it as yet. Prior version seemed better than this one does though. I am in this case using neo4j database with database on volume.

241

I have been using Docker for Mac for development purpose for last 6 months with few hacks to avoid slow write to the mounted volume, but it was usable. I just upgraded to 1.12.3 stable and surprise, surprise, it is not possible to run any app, all the logging is slowing things down.

A.

242

Same experience, terribly slow.

For me Docker for Mac was by far the slowest option.

I’ve wrapped up the numbers at http://tekkie.flashbit.net/devops/docker-performance-on-mac

243

Also experiencing crippling slowness. I run a fairly large Magento site and its so slow its practically unusable for even development purposes.

I have tried using Unison and although I see some improvement its not enough to workaround these performance issues. I can’t see how docker for mac can be considered production ready with such a big issue that affects so many users.

Is there any official benchmarks on docker performance between the major host OS’s?

Very disappointing as I’ve no choice but to use an alternative approach, especially as there is no feedback from Docker on when these issues will be resolved.

244

Please give https://github.com/IFSight/d4m-nfs a try. It is a wrapper to have volume mounted over NFS instead of osxfs.

1 Like