Docker Community Forums

Share and learn in the Docker community.

Docker + Kubernetes: hyperkit CPU


(Planetf1) #1

I have installed docker Version 17.12.0-ce-rc2-mac41 (21193) on the edge channel, which comes with kubernetes

I am using MacOS 10.13.3 beta 1 (understand support, this could be a factor). I’m running on a 2016 macbook pro 15

The first thing I noticed is that with only docker+kubernetes running, there is a noticeable ongoing background cpu load. This is without having any user containers/pods active. hyperkit is taking around 25-30% cpu - this seems to continue long-term even once things are stable

09:59 $ docker images
REPOSITORY                                               TAG                 IMAGE ID            CREATED             SIZE
<none>                                                   <none>              3a253f0d8c18        4 weeks ago         587MB
rangerdocker_mariadb                                     latest              7d1b12e78fbe        4 weeks ago         448MB
mariadb                                                  latest              abcee1d29aac        4 weeks ago         396MB
kamoulox/compose-controller                              v0.1.2              6480f70f7ac6        4 weeks ago         37.9MB
ubuntu                                                   latest              dd6f76d9cc90        5 weeks ago         122MB
gcr.io/google_containers/kube-apiserver-amd64            v1.8.2              6278a1092d08        7 weeks ago         194MB
gcr.io/google_containers/kube-controller-manager-amd64   v1.8.2              5eabb0eae58b        7 weeks ago         129MB
gcr.io/google_containers/kube-scheduler-amd64            v1.8.2              b48970f8473e        7 weeks ago         54.9MB
gcr.io/google_containers/kube-proxy-amd64                v1.8.2              88e2c85d3d02        7 weeks ago         93.1MB
gcr.io/google_containers/k8s-dns-sidecar-amd64           1.14.5              fed89e8b4248        2 months ago        41.8MB
gcr.io/google_containers/k8s-dns-kube-dns-amd64          1.14.5              512cd7425a73        2 months ago        49.4MB
gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64     1.14.5              459944ce8cc4        2 months ago        41.4MB
anaconda_with_spark                                      v3                  d1f0bc2d574b        2 months ago        8.64GB
rstudio-server                                           v2                  e9f71f2ef29a        2 months ago        2.3GB
sandbox-hdp                                              latest              c3cef4760133        4 months ago        12.2GB
gcr.io/google_containers/etcd-amd64                      3.0.17              243830dae7dd        9 months ago        169MB
gcr.io/google_containers/pause-amd64                     3.0                 99e59f495ffa        19 months ago       747kB
✔ /Library/Logs/DiagnosticReports 
10:50 $ docker ps
CONTAINER ID        IMAGE                                                    COMMAND                  CREATED             STATUS              PORTS               NAMES
c5595f837a7b        kamoulox/compose-controller                              "/compose-controller…"   2 minutes ago       Up 2 minutes                            k8s_compose_compose-64c876947c-qwwnr_docker_f9976364-e05b-11e7-ac10-025000000001_0
bdaee77a95ab        gcr.io/google_containers/k8s-dns-sidecar-amd64           "/sidecar --v=2 --lo…"   2 minutes ago       Up 2 minutes                            k8s_sidecar_kube-dns-545bc4bfd4-mvjrs_kube-system_dcfac4a6-e05b-11e7-ac10-025000000001_0
3d32c2fef41a        gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64     "/dnsmasq-nanny -v=2…"   2 minutes ago       Up 2 minutes                            k8s_dnsmasq_kube-dns-545bc4bfd4-mvjrs_kube-system_dcfac4a6-e05b-11e7-ac10-025000000001_0
34f09b89fe34        gcr.io/google_containers/kube-proxy-amd64                "/usr/local/bin/kube…"   2 minutes ago       Up 2 minutes                            k8s_kube-proxy_kube-proxy-hsqkm_kube-system_dcf932ad-e05b-11e7-ac10-025000000001_0
6de5accbdefe        gcr.io/google_containers/k8s-dns-kube-dns-amd64          "/kube-dns --domain=…"   2 minutes ago       Up 2 minutes                            k8s_kubedns_kube-dns-545bc4bfd4-mvjrs_kube-system_dcfac4a6-e05b-11e7-ac10-025000000001_0
5d43522808ab        gcr.io/google_containers/pause-amd64:3.0                 "/pause"                 2 minutes ago       Up 2 minutes                            k8s_POD_kube-proxy-hsqkm_kube-system_dcf932ad-e05b-11e7-ac10-025000000001_0
ef8c82278ed5        gcr.io/google_containers/pause-amd64:3.0                 "/pause"                 2 minutes ago       Up 2 minutes                            k8s_POD_compose-64c876947c-qwwnr_docker_f9976364-e05b-11e7-ac10-025000000001_0
ea46c6ab8a53        gcr.io/google_containers/pause-amd64:3.0                 "/pause"                 2 minutes ago       Up 2 minutes                            k8s_POD_kube-dns-545bc4bfd4-mvjrs_kube-system_dcfac4a6-e05b-11e7-ac10-025000000001_0
e75c5e2f06ca        gcr.io/google_containers/kube-scheduler-amd64            "kube-scheduler --ad…"   2 minutes ago       Up 2 minutes                            k8s_kube-scheduler_kube-scheduler-docker-for-desktop_kube-system_972d74c9fc2f4ebd8ab673058e386a65_0
fc166ae3fc11        gcr.io/google_containers/kube-controller-manager-amd64   "kube-controller-man…"   2 minutes ago       Up 2 minutes                            k8s_kube-controller-manager_kube-controller-manager-docker-for-desktop_kube-system_8fb2a9d14fc1cdac084f95f8513bd907_0
9a52c118b85b        gcr.io/google_containers/kube-apiserver-amd64            "kube-apiserver --ad…"   2 minutes ago       Up 2 minutes                            k8s_kube-apiserver_kube-apiserver-docker-for-desktop_kube-system_cface09d1b5d77b2f65921df84325ddc_0
9140eda8734d        gcr.io/google_containers/etcd-amd64                      "etcd --listen-clien…"   2 minutes ago       Up 2 minutes                            k8s_etcd_etcd-docker-for-desktop_kube-system_8cf2571a4c3f64cf3429d6f601861eec_0
25896a5e1e4f        gcr.io/google_containers/pause-amd64:3.0                 "/pause"                 2 minutes ago       Up 2 minutes                            k8s_POD_kube-scheduler-docker-for-desktop_kube-system_972d74c9fc2f4ebd8ab673058e386a65_0
0568e3739cca        gcr.io/google_containers/pause-amd64:3.0                 "/pause"                 2 minutes ago       Up 2 minutes                            k8s_POD_kube-controller-manager-docker-for-desktop_kube-system_8fb2a9d14fc1cdac084f95f8513bd907_0
e7e8ccecb934        gcr.io/google_containers/pause-amd64:3.0                 "/pause"                 2 minutes ago       Up 2 minutes                            k8s_POD_etcd-docker-for-desktop_kube-system_8cf2571a4c3f64cf3429d6f601861eec_0
7f6d1cc916d9        gcr.io/google_containers/pause-amd64:3.0                 "/pause"                 2 minutes ago       Up 2 minutes                            k8s_POD_kube-apiserver-docker-for-desktop_kube-system_cface09d1b5d77b2f65921df84325ddc_0
✔ /Library/Logs/DiagnosticReports

(Planetf1) #2

Still seeing this with yesterday’s update to 17.12.0-ce-rc3


(Planetf1) #3

I also logged on to the hyperkit vm docker created having tried

screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty

And that was running at 1% cpu - nothing out of the ordinary though I did notice this large memory consumer
2349 2336 root S 10296m 256% 0 0% etcd --listen-client-urls=http://1
2349 root 2:18 etcd --listen-client-urls=http://127.0.0.1:2379 --advertise-client-urls=http://127.0.0.1:2379 --data-dir=/var/lib/etcd

This is with nothing being used, though the actual hyperkit proces is taking cpu ie

14:51 $ ps -ef | grep hyperkit
  501 57895 57891   0 11:10am ??        42:34.97 com.docker.hyperkit -A -u -F /Users/jonesn/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/hyperkit.pid -c 2 -m 4096M -s 0:0,hostbridge -s 31,lpc -s 1:0,virtio-vpnkit,path=/Users/jonesn/Library/Containers/com.docker.docker/Data/s50,uuid=08ba537d-d24a-477f-a183-b9dc5c4aa311 -s 2:0,ahci-hd,file:///Users/jonesn/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2?sync=os&buffered=1,format=qcow,qcow-config=discard=true;compact_after_unmaps=262144;keep_erased=262144;runtime_asserts=false -s 3,virtio-sock,guest_cid=3,path=/Users/jonesn/Library/Containers/com.docker.docker/Data,guest_forwards=2376;1525 -s 4,ahci-cd,/Applications/Docker.app/Contents/Resources/linuxkit/docker-for-mac.iso -s 5,ahci-cd,/Users/jonesn/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/database.iso -s 6,virtio-rnd -s 7,virtio-9p,path=/Users/jonesn/Library/Containers/com.docker.docker/Data/s51,tag=port -l com1,autopty=/Users/jonesn/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty,log=/Users/jonesn/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/console-ring -f bootrom,/Applications/Docker.app/Contents/Resources/uefi/UEFI.fd,,

(Rolf Neugebauer) #4

we are aware of this issue and track it internally. The amount of CPU taken varies from system to system and is only present when Kube is enabled. With Kube enabled a couple of fairly chunky containers are running and it is yet unclear if it is these containers taking up the CPU or interactions with HyperKit


(Linblo) #5

Any updates on this? I am seeing the exact same thing as @planetf1. Around 30% constant CPU load in MacOS, 1% in the hyperkit vm. controller_manager, apiserver followed by etcd being the the ones that stick out with a docker stats.
Really annoying, but it seems to be the same problem with minikube on VirtualBox and HyperKit.


(Edward Raigosa) #6

I’m also seeing this on docker for mac version 18.06.1-ce

    Mem: 5593708K used, 506240K free, 1352K shrd, 406164K buff, 3758040K cached
CPU:   7% usr   7% sys   0% nic  84% idle   0% io   0% irq   0% sirq
Load average: 1.53 1.51 1.47 3/1082 73381
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
 1369  1270 root     S     903m  15%   1   2% /usr/local/bin/dockerd -H unix:///
 3274  3255 root     S     150m   3%   4   1% kube-controller-manager --leader-e
 3221  3200 root     S     649m  11%   4   1% kube-apiserver --admission-control
 1289  1271 root     S     159m   3%   5   1% kubelet --kubeconfig=/etc/kubernet
 3191  3173 root     S    10.0g 171%   1   0% etcd --advertise-client-urls=https
 3247  3229 root     S    48040   1%   3   0% kube-scheduler --address=127.0.0.1
77039 76268 999      S    4714m  78%   0   0% {beam.smp} /usr/lib/erlang/erts-9.
  442     1 root     S    13240   0%   0   0% /usr/bin/memlogd -fd-log 3 -fd-que
 1443  1369 root     S    1652m  27%   5   0% docker-containerd --config /var/ru
 1001     1 root     S    47452   1%   3   0% /usr/bin/containerd
76243  1443 root     S     9960   0%   0   0% docker-containerd-shim -namespace
 3761  3742 root     S    47196   1%   4   0% /usr/local/bin/kube-proxy --config
70690   448 root     R     1552   0%   0   0% top
 4239  4220 root     S    50000   1%   4   0% /api-server --kubeconfig  --authen
 4069  4032 65534    S    38152   1%   1   0% /sidecar --v=2 --logtostderr --pro
 1276  1275 root     S    12712   0%   3   0% /usr/bin/transfused
 1668  1640 root     S    10704   0%   2   0% /usr/bin/logwrite -max-log-files 8
83794     2 root     SW       0   0%   5   0% [kworker/5:2]
18557     2 root     SW       0   0%   0   0% [kworker/0:1]
 1214  1120 root     S     114m   2%   1   0% rpc.statd
linuxkit-025000000001:~# ps -ef|grep 3191
 3191 root      1h08 etcd --advertise-client-urls=https://127.0.0.1:2379 --key-file=/run/config/pki/etcd/server.key --peer-cert-file=/run/config/pki/etcd/peer.crt --peer-trusted-ca-file=/run/config/pki/etcd/ca.crt --listen-client-urls=https://127.0.0.1:2379 --client-cert-auth=true --peer-client-cert-auth=true --data-dir=/var/lib/etcd --cert-file=/run/config/pki/etcd/server.crt --trusted-ca-file=/run/config/pki/etcd/ca.crt --peer-key-file=/run/config/pki/etcd/peer.key
74358 root      0:00 grep 3191

Most of the load coming from etcd, which causes the battery on macbook to drain fairly quick. Is there an open issue for this on the repos somewhere?


(Edward Raigosa) #7

nvm, I see theres an issue now https://github.com/docker/for-mac/issues/2601