Docker + Kubernetes: hyperkit CPU

Docker Desktop
1

I have installed docker Version 17.12.0-ce-rc2-mac41 (21193) on the edge channel, which comes with kubernetes

I am using MacOS 10.13.3 beta 1 (understand support, this could be a factor). I’m running on a 2016 macbook pro 15

The first thing I noticed is that with only docker+kubernetes running, there is a noticeable ongoing background cpu load. This is without having any user containers/pods active. hyperkit is taking around 25-30% cpu - this seems to continue long-term even once things are stable

09:59 $ docker images
REPOSITORY                                               TAG                 IMAGE ID            CREATED             SIZE
<none>                                                   <none>              3a253f0d8c18        4 weeks ago         587MB
rangerdocker_mariadb                                     latest              7d1b12e78fbe        4 weeks ago         448MB
mariadb                                                  latest              abcee1d29aac        4 weeks ago         396MB
kamoulox/compose-controller                              v0.1.2              6480f70f7ac6        4 weeks ago         37.9MB
ubuntu                                                   latest              dd6f76d9cc90        5 weeks ago         122MB
gcr.io/google_containers/kube-apiserver-amd64            v1.8.2              6278a1092d08        7 weeks ago         194MB
gcr.io/google_containers/kube-controller-manager-amd64   v1.8.2              5eabb0eae58b        7 weeks ago         129MB
gcr.io/google_containers/kube-scheduler-amd64            v1.8.2              b48970f8473e        7 weeks ago         54.9MB
gcr.io/google_containers/kube-proxy-amd64                v1.8.2              88e2c85d3d02        7 weeks ago         93.1MB
gcr.io/google_containers/k8s-dns-sidecar-amd64           1.14.5              fed89e8b4248        2 months ago        41.8MB
gcr.io/google_containers/k8s-dns-kube-dns-amd64          1.14.5              512cd7425a73        2 months ago        49.4MB
gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64     1.14.5              459944ce8cc4        2 months ago        41.4MB
anaconda_with_spark                                      v3                  d1f0bc2d574b        2 months ago        8.64GB
rstudio-server                                           v2                  e9f71f2ef29a        2 months ago        2.3GB
sandbox-hdp                                              latest              c3cef4760133        4 months ago        12.2GB
gcr.io/google_containers/etcd-amd64                      3.0.17              243830dae7dd        9 months ago        169MB
gcr.io/google_containers/pause-amd64                     3.0                 99e59f495ffa        19 months ago       747kB
✔ /Library/Logs/DiagnosticReports 
10:50 $ docker ps
CONTAINER ID        IMAGE                                                    COMMAND                  CREATED             STATUS              PORTS               NAMES
c5595f837a7b        kamoulox/compose-controller                              "/compose-controller…"   2 minutes ago       Up 2 minutes                            k8s_compose_compose-64c876947c-qwwnr_docker_f9976364-e05b-11e7-ac10-025000000001_0
bdaee77a95ab        gcr.io/google_containers/k8s-dns-sidecar-amd64           "/sidecar --v=2 --lo…"   2 minutes ago       Up 2 minutes                            k8s_sidecar_kube-dns-545bc4bfd4-mvjrs_kube-system_dcfac4a6-e05b-11e7-ac10-025000000001_0
3d32c2fef41a        gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64     "/dnsmasq-nanny -v=2…"   2 minutes ago       Up 2 minutes                            k8s_dnsmasq_kube-dns-545bc4bfd4-mvjrs_kube-system_dcfac4a6-e05b-11e7-ac10-025000000001_0
34f09b89fe34        gcr.io/google_containers/kube-proxy-amd64                "/usr/local/bin/kube…"   2 minutes ago       Up 2 minutes                            k8s_kube-proxy_kube-proxy-hsqkm_kube-system_dcf932ad-e05b-11e7-ac10-025000000001_0
6de5accbdefe        gcr.io/google_containers/k8s-dns-kube-dns-amd64          "/kube-dns --domain=…"   2 minutes ago       Up 2 minutes                            k8s_kubedns_kube-dns-545bc4bfd4-mvjrs_kube-system_dcfac4a6-e05b-11e7-ac10-025000000001_0
5d43522808ab        gcr.io/google_containers/pause-amd64:3.0                 "/pause"                 2 minutes ago       Up 2 minutes                            k8s_POD_kube-proxy-hsqkm_kube-system_dcf932ad-e05b-11e7-ac10-025000000001_0
ef8c82278ed5        gcr.io/google_containers/pause-amd64:3.0                 "/pause"                 2 minutes ago       Up 2 minutes                            k8s_POD_compose-64c876947c-qwwnr_docker_f9976364-e05b-11e7-ac10-025000000001_0
ea46c6ab8a53        gcr.io/google_containers/pause-amd64:3.0                 "/pause"                 2 minutes ago       Up 2 minutes                            k8s_POD_kube-dns-545bc4bfd4-mvjrs_kube-system_dcfac4a6-e05b-11e7-ac10-025000000001_0
e75c5e2f06ca        gcr.io/google_containers/kube-scheduler-amd64            "kube-scheduler --ad…"   2 minutes ago       Up 2 minutes                            k8s_kube-scheduler_kube-scheduler-docker-for-desktop_kube-system_972d74c9fc2f4ebd8ab673058e386a65_0
fc166ae3fc11        gcr.io/google_containers/kube-controller-manager-amd64   "kube-controller-man…"   2 minutes ago       Up 2 minutes                            k8s_kube-controller-manager_kube-controller-manager-docker-for-desktop_kube-system_8fb2a9d14fc1cdac084f95f8513bd907_0
9a52c118b85b        gcr.io/google_containers/kube-apiserver-amd64            "kube-apiserver --ad…"   2 minutes ago       Up 2 minutes                            k8s_kube-apiserver_kube-apiserver-docker-for-desktop_kube-system_cface09d1b5d77b2f65921df84325ddc_0
9140eda8734d        gcr.io/google_containers/etcd-amd64                      "etcd --listen-clien…"   2 minutes ago       Up 2 minutes                            k8s_etcd_etcd-docker-for-desktop_kube-system_8cf2571a4c3f64cf3429d6f601861eec_0
25896a5e1e4f        gcr.io/google_containers/pause-amd64:3.0                 "/pause"                 2 minutes ago       Up 2 minutes                            k8s_POD_kube-scheduler-docker-for-desktop_kube-system_972d74c9fc2f4ebd8ab673058e386a65_0
0568e3739cca        gcr.io/google_containers/pause-amd64:3.0                 "/pause"                 2 minutes ago       Up 2 minutes                            k8s_POD_kube-controller-manager-docker-for-desktop_kube-system_8fb2a9d14fc1cdac084f95f8513bd907_0
e7e8ccecb934        gcr.io/google_containers/pause-amd64:3.0                 "/pause"                 2 minutes ago       Up 2 minutes                            k8s_POD_etcd-docker-for-desktop_kube-system_8cf2571a4c3f64cf3429d6f601861eec_0
7f6d1cc916d9        gcr.io/google_containers/pause-amd64:3.0                 "/pause"                 2 minutes ago       Up 2 minutes                            k8s_POD_kube-apiserver-docker-for-desktop_kube-system_cface09d1b5d77b2f65921df84325ddc_0
✔ /Library/Logs/DiagnosticReports
2

Still seeing this with yesterday’s update to 17.12.0-ce-rc3

3

I also logged on to the hyperkit vm docker created having tried

screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty

And that was running at 1% cpu - nothing out of the ordinary though I did notice this large memory consumer
2349 2336 root S 10296m 256% 0 0% etcd --listen-client-urls=http://1
2349 root 2:18 etcd --listen-client-urls=http://127.0.0.1:2379 --advertise-client-urls=http://127.0.0.1:2379 --data-dir=/var/lib/etcd

This is with nothing being used, though the actual hyperkit proces is taking cpu ie

14:51 $ ps -ef | grep hyperkit
  501 57895 57891   0 11:10am ??        42:34.97 com.docker.hyperkit -A -u -F /Users/jonesn/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/hyperkit.pid -c 2 -m 4096M -s 0:0,hostbridge -s 31,lpc -s 1:0,virtio-vpnkit,path=/Users/jonesn/Library/Containers/com.docker.docker/Data/s50,uuid=08ba537d-d24a-477f-a183-b9dc5c4aa311 -s 2:0,ahci-hd,file:///Users/jonesn/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2?sync=os&buffered=1,format=qcow,qcow-config=discard=true;compact_after_unmaps=262144;keep_erased=262144;runtime_asserts=false -s 3,virtio-sock,guest_cid=3,path=/Users/jonesn/Library/Containers/com.docker.docker/Data,guest_forwards=2376;1525 -s 4,ahci-cd,/Applications/Docker.app/Contents/Resources/linuxkit/docker-for-mac.iso -s 5,ahci-cd,/Users/jonesn/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/database.iso -s 6,virtio-rnd -s 7,virtio-9p,path=/Users/jonesn/Library/Containers/com.docker.docker/Data/s51,tag=port -l com1,autopty=/Users/jonesn/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty,log=/Users/jonesn/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/console-ring -f bootrom,/Applications/Docker.app/Contents/Resources/uefi/UEFI.fd,,
4

we are aware of this issue and track it internally. The amount of CPU taken varies from system to system and is only present when Kube is enabled. With Kube enabled a couple of fairly chunky containers are running and it is yet unclear if it is these containers taking up the CPU or interactions with HyperKit

5

Any updates on this? I am seeing the exact same thing as @planetf1. Around 30% constant CPU load in MacOS, 1% in the hyperkit vm. controller_manager, apiserver followed by etcd being the the ones that stick out with a docker stats.
Really annoying, but it seems to be the same problem with minikube on VirtualBox and HyperKit.

6

I’m also seeing this on docker for mac version 18.06.1-ce

    Mem: 5593708K used, 506240K free, 1352K shrd, 406164K buff, 3758040K cached
CPU:   7% usr   7% sys   0% nic  84% idle   0% io   0% irq   0% sirq
Load average: 1.53 1.51 1.47 3/1082 73381
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
 1369  1270 root     S     903m  15%   1   2% /usr/local/bin/dockerd -H unix:///
 3274  3255 root     S     150m   3%   4   1% kube-controller-manager --leader-e
 3221  3200 root     S     649m  11%   4   1% kube-apiserver --admission-control
 1289  1271 root     S     159m   3%   5   1% kubelet --kubeconfig=/etc/kubernet
 3191  3173 root     S    10.0g 171%   1   0% etcd --advertise-client-urls=https
 3247  3229 root     S    48040   1%   3   0% kube-scheduler --address=127.0.0.1
77039 76268 999      S    4714m  78%   0   0% {beam.smp} /usr/lib/erlang/erts-9.
  442     1 root     S    13240   0%   0   0% /usr/bin/memlogd -fd-log 3 -fd-que
 1443  1369 root     S    1652m  27%   5   0% docker-containerd --config /var/ru
 1001     1 root     S    47452   1%   3   0% /usr/bin/containerd
76243  1443 root     S     9960   0%   0   0% docker-containerd-shim -namespace
 3761  3742 root     S    47196   1%   4   0% /usr/local/bin/kube-proxy --config
70690   448 root     R     1552   0%   0   0% top
 4239  4220 root     S    50000   1%   4   0% /api-server --kubeconfig  --authen
 4069  4032 65534    S    38152   1%   1   0% /sidecar --v=2 --logtostderr --pro
 1276  1275 root     S    12712   0%   3   0% /usr/bin/transfused
 1668  1640 root     S    10704   0%   2   0% /usr/bin/logwrite -max-log-files 8
83794     2 root     SW       0   0%   5   0% [kworker/5:2]
18557     2 root     SW       0   0%   0   0% [kworker/0:1]
 1214  1120 root     S     114m   2%   1   0% rpc.statd
linuxkit-025000000001:~# ps -ef|grep 3191
 3191 root      1h08 etcd --advertise-client-urls=https://127.0.0.1:2379 --key-file=/run/config/pki/etcd/server.key --peer-cert-file=/run/config/pki/etcd/peer.crt --peer-trusted-ca-file=/run/config/pki/etcd/ca.crt --listen-client-urls=https://127.0.0.1:2379 --client-cert-auth=true --peer-client-cert-auth=true --data-dir=/var/lib/etcd --cert-file=/run/config/pki/etcd/server.crt --trusted-ca-file=/run/config/pki/etcd/ca.crt --peer-key-file=/run/config/pki/etcd/peer.key
74358 root      0:00 grep 3191

Most of the load coming from etcd, which causes the battery on macbook to drain fairly quick. Is there an open issue for this on the repos somewhere?

7

nvm, I see theres an issue now https://github.com/docker/for-mac/issues/2601