MySQL Slow Performance In Docker

General
1

I’m running Docker on Ubuntu Xenial with the following command. For certain reason it’s 15 times slower than Virtual Box on the same machine with ext4. It’s about 5 times slower on XFS. I have tried different configuration but couldn’t figure out why its such slow. Is there anyway we can optimize MySQL performance in docker?

docker run -d --env=“MYSQL_ROOT_PASSWORD=drupal” --name=mysql -p 3306:3306 mysql

MySQL restore test in VirtualBox
weikai@drupal:~$ time mysql vbox < drupal3.sql
real 2m12.081s
user 0m4.868s
sys 0m1.208s

MySQL restore test in Docker
root@e3488a990fde:~# time mysql -u root -p dockerdb < drupal3.sql
Enter password:

real 11m47.783s
user 0m4.784s
sys 0m13.316s

1 Like
3

You can try and “mount” a local folder, instead of using the container as storage.
try adding: -v $HOME/datadir:/var/lib/mysql

4

It makes no differences. It’s about 15 times slower than the VirtualBox on the same machine since $HOME is on ext4 partition.

root@3ff6a442cbb2:~# time mysql dockerdb < drupal3.sql

real 29m41.965s
user 0m6.456s
sys 0m0.472s

weikai@cloud:~$ mount|grep ‘^/dev’
/dev/mapper/cloud–vg-root on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/sda1 on /boot type ext2 (rw,relatime,block_validity,barrier,user_xattr,acl)
/dev/sde on /mnt/disk1 type ext4 (rw,relatime,data=ordered)
/dev/mapper/data–vg-docker on /var/lib/docker type xfs (rw,relatime,attr2,inode64,noquota)
/dev/mapper/data–vg-data on /srv type ext4 (rw,relatime,data=ordered)
/dev/mapper/data–vg-data on /home type ext4 (rw,relatime,data=ordered)
/dev/mapper/data–vg-vbox on /home/vbox type ext4 (rw,relatime,data=ordered)
/dev/mapper/data–vg-vbox on /srv/home/vbox type ext4 (rw,relatime,data=ordered)
/dev/mapper/data–vg-docker on /var/lib/docker/plugins type xfs (rw,relatime,attr2,inode64,noquota)
/dev/mapper/data–vg-docker on /var/lib/docker/aufs type xfs (rw,relatime,attr2,inode64,noquota)

weikai@cloud:~$ docker info
Containers: 3
Running: 3
Paused: 0
Stopped: 0
Images: 89
Server Version: 17.06.1-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: xfs
Dirs: 106
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-92-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 19.5GiB
Name: cloud
ID: Q4IZ:YS4B:QDBR:5W6J:W4EN:ZT2T:72RE:X3MT:SUWI:BZGD:L34Q:KGI7
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

5

I’m experiencing the same issue. MySQL is very slow. I’m running mysql:5.7 on Docker version 17.09.0-ce, build afdb6d4 with a local volume for the data directory, which is ext4. The volume driver is local and on my system the mounts show up as nsfs.

Importing a gigabyte of data from an external server takes half an hour on localhost, but close to 24 hours on this container.

Server Version: 17.09.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Kernel Version: 4.4.0-93-generic
Operating System: Linux Mint 18
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.64GiB
Docker Root Dir: /home/local/docker
Debug Mode (client): false
Debug Mode (server): false
Experimental: false
Live Restore Enabled: false

WARNING: No swap limit support
6

Same there :frowning: I’ve tested with volumes as well and same behaviour. Perfs divided by 10.

7

Not many people experience the issue. I’m wondering if we can find something in common. The following is my server information. The slow less problem is gone if I mount the volume on the small SSD. I’m wondering why it’s this slow on the magnetic drive (Internal drive Seagate 8TB BarraCuda Pro). VirtualBox instances running on the same machine using magnetic partition can run 10 - 20 times faster on the same MySQL operations.

OS: Ubuntu 16.04
CPU: Intel® Xeon® CPU E31235 @ 3.20GHz
Hard Drives: sda SSD lvm 128MB
sdb magnetic lvm 8TB

8

OS: Ubuntu 16.04.3 (Linux Mint 18.3)
Kernel: 4.10.0-40-generic x86_64
Volume: Crucial m4 SSD 256GB
CPU: Intel Core i7-3770 @ 3.40 GHz

9

Hey @Everyone experiencing this issue. I did an unexpected discovery.

In order to circumvent this problem, I connected my docker container to a fast MySQL setup running on the localhost, outside of docker.

However, I was surprised to see there was no speed improvement at all. Importing 1GB of data still took 24 hours.

This leads me to believe there is a serious and unexpected bottleneck in the docker system when using a mysql-client, and the culprit is not the mysql-server.

2 Likes
10

Hi @redsandro, everyone,

I had the same problem, running mysql anormally slow in docker, and discovered that it had nothing to do with docker at all (i have pretty much the same configuration as you, but with ext4 partition); it was my file system extremly slow with mysql (and maybe with other packages but i didn’t notice).
I resolved the problem following this tutorial:
http://phpforus.com/how-to-make-mysql-run-fast-with-ext4-on-ubuntu/

Not the best I presume, but after that, I had the exactly same performance as my colleagues who were on another system.

1 Like
11

The performance problem on my system is actually inside Docker. Mysql running on both Ubuntu physical Host and VirtualBox instances are 10X+ time faster than MySQL server inside Docker.

12

Thanks @renomarx for sharing this information. I’m glad you’ve found a way around this. I have to agree with @weikai though; this problem is more complicated than it looks.

In my case, I have moved the MySQL server from a Docker container to my localhost, which is also ext4, and it was still slow. But hosting the application locally in stead of in docker - while using that same local MySQL server - improves the speed more than drastically.

Unintuitively, It seems that this is not Docker-specific volume/mount/link/share/volume related, but Docker related nontheless.

13

Docker performance issue on my server is resolved. It’s quicker than VirtualBox now. Thank you @renomarx for the article link. I mounted an ext4 partition at /var/lib/docker with barrier=0. It also explains why the problem only happens on magnetic disk not SSD on the same server because SSD doesn’t have barrier on by default.

14

I am still facing this problem :frowning: Also happens on Docker for Mac… mysql imports take a very long time. If I run the client from a different machine it takes half as long, but still 20min to import aprox 300mb of files.

1 Like
15

Same issue with mysql:5.7 on Azure

16

Almost same here. I realized queries to the database were about 5 times slower compared to a non-docker MySQL installation.

After I ran “optimize table” on every table in the mysql docker, the query ran 22 times faster than before and 4-5 times faster compared to the non-docker MySQL installation.
However, after a complete drop database and fresh import of all tables, the performance of the query was just as fast as after the optimization of the tables.

My conclusion was, that during the (development) usage of the mysql docker the performance might have become worse and worse. Maybe the index files are getting corrupted by stopping a stack with ctrl-c rather than docker-compose down.

I’ll keep an eye on it, to confirm or falsify the above.

17

We have experienced a similar issue where MySQL is extremely slow. But not only in a container, also on a Ubuntu system directly if docker is installed.
We were unable to solve this issue in many hours, so we have created a ticket on github:

18

I too am experiencing the same issue with docker for mac. Mysql in a virtualbox with vagrant is exponentially faster

19

Hi. I think I am having a similar problem:

  • hosting the server with or without docker does not change performance.

  • When the application runs outside of docker I have normal/expected performance. No matter on which machine in my network.

  • When I run the application inside a container with --network host on the same machine as the mysql database + using 127.0.0.1 as host address: I have normal performance.

  • When I am running the application inside docker on a different machine than the database performance goes down. Here I get a net_read_timeout while running a LOAD DATA LOCAL INFILE- however this seems to kick in when the file becomes ‘large’ (> 1000 lines, 3 cols) and works otherwise. The read speed is so low that raising the timeout is not an option.

Did you find any solution for yourself?

20

That sounds about right. Unfortunately we did not find a solution. We’re not running the main app on docker anymore because of this.

21

Hello guys,

I had same issue and the instruction @renomarx linked helped to fix it. Mysql performance under Docker is dramatically improved.

Here is my setup:

  • Macbook pro retina late 2013, 8GB RAM, i5, SSD with dual boot Ubuntu 18.10.

Thank you!