Apps/services running inside Docker are !significantly! slower than on host system

General
,
1

Dear community,

:sos:
I would like to ask often but still didn’t solved question, why apps/services is running much much slover inside docker, then directly running on host system.

I spend a lot of time to solve this situation but without success. I tested it on multiple systems with multiple processors and multiple architectures together with multiple images inside docker, but situation was always same: Processes and apps running inside docker is running slower (not signifficantly, but multiple times slower!). For example, building same nodejs app with same conditions is running multiple times slower inside docker than outside docker (caching is not the reason, I’m sure).

Building on host system takes 3 minutes, build same app takes inside docker takes about 20 minutes :warning: Tested with multiple host machines, with multiple docker images, result was the same.

Probably, you will be asking me for docker info, but I think, this is not relevant (some issue in configuration), because as I written, I tested it on multiple machines, with multiple architectures and etc. and situation was the same.

By the another questions in this forum, I’m not alone with this experience.
Therefore my question is, if this slowdown is feature and normal behavior of Docker or if it is something wrong. I cannot trust, that this slowdown is OK, I was expected, that docker performance will be similar than on host machine.

Also I don’t know, if this slowdown is comming from CPU performance, memory operations or disk operations. If zou can recommend me some simple tools to measure it, please share with me to deeper investigate the reason.

Thanks a lot for an each help.

2

Which Docker version are you using on which host operating system? Using Docker Desktop, which runs its own VM?

3

Hi @bluepuma77, I tested almost all combinations and the result was the same.

Local machine:

Architecture x86_64
Ubuntu: 24.04.1 LTS
Docker version 27.3.1, build ce12230
Docker Desktop 4.34.2

Tested with docker images: alpine:3, ubuntu:latest, amazonlinux… etc.

Builder server machine:

Architecture x86_64
Ubuntu: 22.04 LTS
Docker Desktop: none
...next information I will add in a minutes

Client: Docker Engine - Community
 Version:    27.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.15.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.28.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 30
 Server Version: 27.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: *******
 runc version: v1.1.13-0
 init version: *****
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-117-generic
 Operating System: Ubuntu 22.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 245.7GiB
 Name: *****
 ID: *******
 Docker Root Dir: /******/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

…but the problem is probably not relevant to docker versions, because I had the problem also with lot of previous versions.

4

@bluepuma77 I added more detailed info from Builder machine.

5

Docker Desktop launches a Linux VM which runs Docker Engine on it
Therefore, it has limited allocated memory and resources

This may not be all there is to it, but if you’re using a Linux host, avoid Docker Desktop

6

Hi @deanayalon I know, on building machine, there is not installed docker-desktop. Also I know, that the resources can be limited, but this is not my case.

I spend a lot of time on investigation of this performance problem. If you are running some services onside docker and you will try run it also on host, probably you will be the same results/experiences like me.

7

I see you masked your docker data root dir. Is it on a network filesystem for example? Did you have the same docker data root on each machine?

Since a Docker container is an isolated environment for processes where the process can’t see everything on the host, but the host can see everything that runs in the container, there is not much that can make things much slower. The overlay filesystem used for containers based on multiple layers merged into one from the container’s point of view. I can imagine that in your use case it doesn’t work properly, that would mean there is something special about the docker data root.

When you tested the speed, did you test it on the container’s filesystem or did you mount a folder in which the process accessed files?

1 Like