Hello,
this is my first post on this forums, hopefully this is the right place for this topic.
We are running several VMs with Docker in production and for development, testing and QA. For the QA systems we use docker compose to start a subset of microservices (around 100) to create ephemeral environments of the application. This happens currently on several machines, each being used to deploy different versions or feature branches etc.
Recently it happend that the system load inceased drastically after a system update. See details below.
VM configuration for all QA systems:
- 2 CPU cores, 10GB RAM
- Ubuntu 16.04 LTS
- docker-ce=18.06.3~ce~3-0~ubuntu (deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable) (pinned)
- libseccomp2=2.3.1-2.1ubuntu2~16.04.1 (until dist-upgrade)
- docker-compose=1.24.0 (via pip)
System load before ubuntu dist-upgrade:
Figure 1: qa1_before
As can be seen on the graph the baseline for cpu utilization was about 30 percent. On 5th of June there was the dist-upgrade in a scope of our usually monthly happening update procedures.
Checking the apt logs there were a few packages upgraded but nothing really essential besides libseccomp2.
From the logs: “Upgrade: libseccomp2:amd64 (2.3.1-2.1ubuntu2~16.04.1, 2.4.1-0ubuntu0.16.04.2)”
This is the load profile of the same application after the dist-upgrade or more to say after the update of libseccomp2.
Figure 2: qa1_after
After some testing I found out that the problem can be mitigated in three ways but all of them are not satisfying or a good idea at all.
Downgrade docker-ce and libseccomp2
Install docker-ce=17.12.0~ce-0~ubuntu and libseccomp2=2.2.3-3ubuntu3 which leads to an even lower overall utilization (about 20%). The reason is as docker-ce > 17.12.0~ce-0~ubuntu has libseccomp2 >= 2.3.0 as a dependency and there are only installation candidates for libseccomp2 version 2.2.3 and 2.4.1 it is nescesary to go back that far with the docker-ce version. Remerber that libseccomp2 was version 2.3.1 before the dist-upgade which is not available anymore from the available apt repositories - it was superseded by 2.4.1.
Install libseccomp2 in version 2.3.1 from ppa:Lxd
Downgrade to libseccomp2 version 2.3.1 (2.3.1-2.1ubuntu3~ubuntu16.04.1~ppa1)
https://www.ubuntuupdates.org/package/lxd/xenial/main/base/libseccomp2
This results in an overall system utilization like before the update, making libseccomp2 the suspect to cause the increase of cpu load.
Figure 3: qa1_current
Disable Seccomp security
During multiple tests with different recent versions of Ubuntu, docker-ce and combinations of versions it was all of the time the same result: with libseccomp2 version 2.4.1 the performance was bad like shown in figure 2. Adding “–security-opt seccomp=unconfined” to each container mitigated the effects. But obviousliy this ist not a real solution.
Figure 4: qa2_noseccomp
So after all there seems to be a relation to libseccomp2 affecting performance heavily.
Did anyone notice this too and found a valid solution to this or can someone tell me what I miss here?
If you have any further questions to better understand or if there is anything totally unclear just ask.
Best regards,
Imo