Docker Community Forums

Share and learn in the Docker community.

Docker EE (17.06.2-ee-16) kube-controller High CPU Usage


(Billgercken) #1

Hi!

I am working on a basic setup of Docker EE and I am seeing kube-controller using a large portion of the CPU. This is on a fresh install with only the manager node registered. (CentOS 7)

Any ideas as to where to look to resolve this issue?

Thanks!
-bill

Process Info

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 5861 root      20   0   42348  17300  10448 R 300.0  0.1 545:35.32 kube-controller
10024 root      20   0  164616   2796   1556 R  14.3  0.0   0:00.07 top

Docker Info

[root@docker-manager2 containers]# docker info
Containers: 49
 Running: 33
 Paused: 0
 Stopped: 16
Images: 35
Server Version: 17.06.2-ee-16
Storage Driver: devicemapper
 Pool Name: docker-thinpool
 Pool Blocksize: 524.3kB
 Base Device Size: 10.74GB
 Backing Filesystem: xfs
 Data file:
 Metadata file:
 Data Space Used: 5.833GB
 Data Space Total: 102GB
 Data Space Available: 96.17GB
 Metadata Space Used: 2.671MB
 Metadata Space Total: 1.07GB
 Metadata Space Available: 1.067GB
 Thin Pool Minimum Free Space: 10.2GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.146-RHEL7 (2018-01-22)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: l6rpstjnobj7f1i41qs49cvri
 Is Manager: true
 ClusterID: pxu79dkh8s5l9aj02izz67opl
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 10
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
  External CAs:
    cfssl: https://192.168.0.191:12381/api/v1/cfssl/sign
 Root Rotation In Progress: false
 Node Address: 192.168.0.191
 Manager Addresses:
  192.168.0.191:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 462c82662200a17ee39e74692f536067a3576a50
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-862.9.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.51GiB
Name: docker-manager2
ID: FS7T:JBYK:YB5B:WNUN:EHIL:EOHE:YCWA:Q3BZ:2CC2:TNHX:CQKT:B7HN
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: bridge-nf-call-ip6tables is disabled

Container list

 docker container ls
CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS                            PORTS                                                                             NAMES
f38be62853e9        docker/ucp-kube-dns-dnsmasq-nanny    "/dnsmasq-nanny -v..."   18 seconds ago      Up 17 seconds                                                                                                       k8s_ucp-dnsmasq-nanny_kube-dns-57b9f4b7f7-6cnx2_kube-system_d5e8a232-9355-11e8-87d1-0242ac110012_55
a8d9b22747af        docker/ucp-agent:3.0.3               "/bin/ucp-agent agent"   3 hours ago         Up 3 hours                        2376/tcp                                                                          ucp-agent.l6rpstjnobj7f1i41qs49cvri.sqq6euz60xa0mjmmnollqnt9s
4ca86fe2df3c        docker/dtr-postgres:2.5.4            "/bin/postgreswrapper"   3 hours ago         Up 3 hours                        5432/tcp                                                                          dtr-scanningstore-f990a196e639
8bfc132d15d4        docker/dtr-notary-signer:2.5.4       "/bin/notarywrappe..."   3 hours ago         Up 3 hours                                                                                                          dtr-notary-signer-f990a196e639
e7df883da4a3        docker/dtr-jobrunner:2.5.4           "/bin/jobrunner wo..."   3 hours ago         Up 3 hours                                                                                                          dtr-jobrunner-f990a196e639
44ee1a03ece5        docker/dtr-nginx:2.5.4               "/bin/nginxwrapper"      3 hours ago         Up 3 hours                        0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp                                          dtr-nginx-f990a196e639
530e329cb237        docker/dtr-notary-server:2.5.4       "/bin/notarywrappe..."   3 hours ago         Up 3 hours                                                                                                          dtr-notary-server-f990a196e639
7cd25b9490f7        docker/dtr-api:2.5.4                 "/bin/api"               3 hours ago         Up 3 hours                                                                                                          dtr-api-f990a196e639
78605f8b584e        docker/dtr-garant:2.5.4              "/bin/garant"            3 hours ago         Up 3 hours                                                                                                          dtr-garant-f990a196e639
786fe02f7110        docker/dtr-registry:2.5.4            "/bin/registry"          3 hours ago         Up 3 hours                                                                                                          dtr-registry-f990a196e639
783ad27915b1        docker/dtr-rethink:2.5.4             "/bin/rethinkwrapper"    3 hours ago         Up 3 hours                                                                                                          dtr-rethinkdb-f990a196e639
055a986bbd97        docker/ucp-kube-dns-sidecar          "/sidecar --v=2 --..."   4 hours ago         Up 4 hours                                                                                                          k8s_ucp-kubedns-sidecar_kube-dns-57b9f4b7f7-6cnx2_kube-system_d5e8a232-9355-11e8-87d1-0242ac110012_5
31e28b76a908        docker/ucp-calico-cni                "/install-cni.sh"        4 hours ago         Up 4 hours                                                                                                          k8s_install-cni_calico-node-n7r6h_kube-system_d0244db1-9355-11e8-87d1-0242ac110012_5
6e0ccb29d58c        docker/ucp-calico-node               "start_runit"            4 hours ago         Up 4 hours                                                                                                          k8s_calico-node_calico-node-n7r6h_kube-system_d0244db1-9355-11e8-87d1-0242ac110012_5
332b6fa14141        docker/ucp-calico-kube-controllers   "/usr/bin/kube-con..."   4 hours ago         Up 4 hours                                                                                                          k8s_calico-kube-controllers_calico-kube-controllers-96b8846c-dn87q_kube-system_cff1ead8-9355-11e8-87d1-0242ac110012_5
c7ceb8beadc1        docker/ucp-pause:3.0.3               "/pause"                 4 hours ago         Up 4 hours                                                                                                          k8s_POD_compose-67d4578cf4-zmphr_kube-system_d78b8460-9355-11e8-87d1-0242ac110012_5
b73d1119370e        docker/ucp-pause:3.0.3               "/pause"                 4 hours ago         Up 4 hours                                                                                                          k8s_POD_calico-node-n7r6h_kube-system_d0244db1-9355-11e8-87d1-0242ac110012_5
3b95984ee382        docker/ucp-pause:3.0.3               "/pause"                 4 hours ago         Up 4 hours                                                                                                          k8s_POD_kube-dns-57b9f4b7f7-6cnx2_kube-system_d5e8a232-9355-11e8-87d1-0242ac110012_5
84b1fc258290        docker/ucp-pause:3.0.3               "/pause"                 4 hours ago         Up 4 hours                                                                                                          k8s_POD_calico-kube-controllers-96b8846c-dn87q_kube-system_cff1ead8-9355-11e8-87d1-0242ac110012_5
b377a0b1646d        docker/ucp-hyperkube:3.0.3           "/bin/apiserver_en..."   5 hours ago         Up 4 hours                        0.0.0.0:6443->6443/tcp                                                            ucp-kube-apiserver
6765df7586d8        docker/ucp-controller:3.0.3          "/bin/controller s..."   5 hours ago         Up Less than a second (healthy)   0.0.0.0:8443->8080/tcp                                                            ucp-controller
297e378306c0        docker/ucp-auth:3.0.3                "/usr/local/bin/en..."   2 days ago          Up 4 hours (healthy)              0.0.0.0:12385->4443/tcp                                                           ucp-auth-api
9f403b48dc61        docker/ucp-hyperkube:3.0.3           "kube-controller-m..."   2 days ago          Up Less than a second (healthy)                                                                                     ucp-kube-controller-manager
4cbca512e7f9        docker/ucp-swarm:3.0.3               "/bin/swarm manage..."   2 days ago          Up Less than a second (healthy)   0.0.0.0:2376->2375/tcp                                                            ucp-swarm-manager
0f454568ef0c        docker/ucp-auth:3.0.3                "/usr/local/bin/en..."   2 days ago          Up 4 hours (healthy)              0.0.0.0:12386->4443/tcp                                                           ucp-auth-worker
392040ea3a1e        docker/ucp-hyperkube:3.0.3           "kube-scheduler --..."   2 days ago          Up Less than a second (healthy)                                                                                     ucp-kube-scheduler
fcbdf4b0f286        docker/ucp-auth-store:3.0.3          "rethinkdb --bind ..."   2 days ago          Up Less than a second (healthy)   0.0.0.0:12383-12384->12383-12384/tcp                                              ucp-auth-store
0f17ebe5f2ed        docker/ucp-metrics:3.0.3             "/bin/entrypoint.s..."   2 days ago          Up Less than a second (healthy)   0.0.0.0:12387->12387/tcp                                                          ucp-metrics
37b8a6a6c5da        docker/ucp-etcd:3.0.3                "/bin/entrypoint.s..."   2 days ago          Up Less than a second (healthy)   2380/tcp, 4001/tcp, 7001/tcp, 0.0.0.0:12380->12380/tcp, 0.0.0.0:12379->2379/tcp   ucp-kv
c37e9879afe8        docker/ucp-cfssl:3.0.3               "/bin/ucp-ca serve..."   2 days ago          Up Less than a second (healthy)   0.0.0.0:12382->12382/tcp                                                          ucp-client-root-ca
6ce917f7767c        docker/ucp-cfssl:3.0.3               "/bin/ucp-ca serve..."   2 days ago          Up 4 hours (healthy)              0.0.0.0:12381->12381/tcp                                                          ucp-cluster-root-ca
46b78465fb2a        docker/ucp-hyperkube:3.0.3           "kube-proxy --clus..."   2 days ago          Up Less than a second                                                                                               ucp-kube-proxy
7d65c2143f28        docker/ucp-hyperkube:3.0.3           "kubelet --allow-p..."   2 days ago          Up Less than a second                                                                                               ucp-kubelet
82794f34aafc        docker/ucp-agent:3.0.3               "/bin/ucp-agent pr..."   2 days ago          Up Less than a second (healthy)   0.0.0.0:6444->6444/tcp, 0.0.0.0:12378->12378/tcp, 0.0.0.0:12376->2376/tcp         ucp-proxy

Process List

[root@docker-manager2 log]# docker ps
CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS                            PORTS                                                                             NAMES
ad4378795db3        docker/ucp-kube-dns-dnsmasq-nanny    "/dnsmasq-nanny -v..."   49 seconds ago      Up 48 seconds                                                                                                       k8s_ucp-dnsmasq-nanny_kube-dns-57b9f4b7f7-6cnx2_kube-system_d5e8a232-9355-11e8-87d1-0242ac110012_54
a8d9b22747af        docker/ucp-agent:3.0.3               "/bin/ucp-agent agent"   3 hours ago         Up 3 hours                        2376/tcp                                                                          ucp-agent.l6rpstjnobj7f1i41qs49cvri.sqq6euz60xa0mjmmnollqnt9s
4ca86fe2df3c        docker/dtr-postgres:2.5.4            "/bin/postgreswrapper"   3 hours ago         Up 3 hours                        5432/tcp                                                                          dtr-scanningstore-f990a196e639
8bfc132d15d4        docker/dtr-notary-signer:2.5.4       "/bin/notarywrappe..."   3 hours ago         Up 3 hours                                                                                                          dtr-notary-signer-f990a196e639
e7df883da4a3        docker/dtr-jobrunner:2.5.4           "/bin/jobrunner wo..."   3 hours ago         Up 3 hours                                                                                                          dtr-jobrunner-f990a196e639
44ee1a03ece5        docker/dtr-nginx:2.5.4               "/bin/nginxwrapper"      3 hours ago         Up 3 hours                        0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp                                          dtr-nginx-f990a196e639
530e329cb237        docker/dtr-notary-server:2.5.4       "/bin/notarywrappe..."   3 hours ago         Up 3 hours                                                                                                          dtr-notary-server-f990a196e639
7cd25b9490f7        docker/dtr-api:2.5.4                 "/bin/api"               3 hours ago         Up 3 hours                                                                                                          dtr-api-f990a196e639
78605f8b584e        docker/dtr-garant:2.5.4              "/bin/garant"            3 hours ago         Up 3 hours                                                                                                          dtr-garant-f990a196e639
786fe02f7110        docker/dtr-registry:2.5.4            "/bin/registry"          3 hours ago         Up 3 hours                                                                                                          dtr-registry-f990a196e639
783ad27915b1        docker/dtr-rethink:2.5.4             "/bin/rethinkwrapper"    3 hours ago         Up 3 hours                                                                                                          dtr-rethinkdb-f990a196e639
055a986bbd97        docker/ucp-kube-dns-sidecar          "/sidecar --v=2 --..."   4 hours ago         Up 4 hours                                                                                                          k8s_ucp-kubedns-sidecar_kube-dns-57b9f4b7f7-6cnx2_kube-system_d5e8a232-9355-11e8-87d1-0242ac110012_5
31e28b76a908        docker/ucp-calico-cni                "/install-cni.sh"        4 hours ago         Up 4 hours                                                                                                          k8s_install-cni_calico-node-n7r6h_kube-system_d0244db1-9355-11e8-87d1-0242ac110012_5
6e0ccb29d58c        docker/ucp-calico-node               "start_runit"            4 hours ago         Up 4 hours                                                                                                          k8s_calico-node_calico-node-n7r6h_kube-system_d0244db1-9355-11e8-87d1-0242ac110012_5
332b6fa14141        docker/ucp-calico-kube-controllers   "/usr/bin/kube-con..."   4 hours ago         Up 4 hours                                                                                                          k8s_calico-kube-controllers_calico-kube-controllers-96b8846c-dn87q_kube-system_cff1ead8-9355-11e8-87d1-0242ac110012_5
c7ceb8beadc1        docker/ucp-pause:3.0.3               "/pause"                 4 hours ago         Up 4 hours                                                                                                          k8s_POD_compose-67d4578cf4-zmphr_kube-system_d78b8460-9355-11e8-87d1-0242ac110012_5
b73d1119370e        docker/ucp-pause:3.0.3               "/pause"                 4 hours ago         Up 4 hours                                                                                                          k8s_POD_calico-node-n7r6h_kube-system_d0244db1-9355-11e8-87d1-0242ac110012_5
3b95984ee382        docker/ucp-pause:3.0.3               "/pause"                 4 hours ago         Up 4 hours                                                                                                          k8s_POD_kube-dns-57b9f4b7f7-6cnx2_kube-system_d5e8a232-9355-11e8-87d1-0242ac110012_5
84b1fc258290        docker/ucp-pause:3.0.3               "/pause"                 4 hours ago         Up 4 hours                                                                                                          k8s_POD_calico-kube-controllers-96b8846c-dn87q_kube-system_cff1ead8-9355-11e8-87d1-0242ac110012_5
b377a0b1646d        docker/ucp-hyperkube:3.0.3           "/bin/apiserver_en..."   5 hours ago         Up 4 hours                        0.0.0.0:6443->6443/tcp                                                            ucp-kube-apiserver
6765df7586d8        docker/ucp-controller:3.0.3          "/bin/controller s..."   5 hours ago         Up Less than a second (healthy)   0.0.0.0:8443->8080/tcp                                                            ucp-controller
297e378306c0        docker/ucp-auth:3.0.3                "/usr/local/bin/en..."   2 days ago          Up 4 hours (healthy)              0.0.0.0:12385->4443/tcp                                                           ucp-auth-api
9f403b48dc61        docker/ucp-hyperkube:3.0.3           "kube-controller-m..."   2 days ago          Up Less than a second (healthy)                                                                                     ucp-kube-controller-manager
4cbca512e7f9        docker/ucp-swarm:3.0.3               "/bin/swarm manage..."   2 days ago          Up Less than a second (healthy)   0.0.0.0:2376->2375/tcp                                                            ucp-swarm-manager
0f454568ef0c        docker/ucp-auth:3.0.3                "/usr/local/bin/en..."   2 days ago          Up 4 hours (healthy)              0.0.0.0:12386->4443/tcp                                                           ucp-auth-worker
392040ea3a1e        docker/ucp-hyperkube:3.0.3           "kube-scheduler --..."   2 days ago          Up Less than a second (healthy)                                                                                     ucp-kube-scheduler
fcbdf4b0f286        docker/ucp-auth-store:3.0.3          "rethinkdb --bind ..."   2 days ago          Up Less than a second (healthy)   0.0.0.0:12383-12384->12383-12384/tcp                                              ucp-auth-store
0f17ebe5f2ed        docker/ucp-metrics:3.0.3             "/bin/entrypoint.s..."   2 days ago          Up Less than a second (healthy)   0.0.0.0:12387->12387/tcp                                                          ucp-metrics
37b8a6a6c5da        docker/ucp-etcd:3.0.3                "/bin/entrypoint.s..."   2 days ago          Up Less than a second (healthy)   2380/tcp, 4001/tcp, 7001/tcp, 0.0.0.0:12380->12380/tcp, 0.0.0.0:12379->2379/tcp   ucp-kv
c37e9879afe8        docker/ucp-cfssl:3.0.3               "/bin/ucp-ca serve..."   2 days ago          Up Less than a second (healthy)   0.0.0.0:12382->12382/tcp                                                          ucp-client-root-ca
6ce917f7767c        docker/ucp-cfssl:3.0.3               "/bin/ucp-ca serve..."   2 days ago          Up 4 hours (healthy)              0.0.0.0:12381->12381/tcp                                                          ucp-cluster-root-ca
46b78465fb2a        docker/ucp-hyperkube:3.0.3           "kube-proxy --clus..."   2 days ago          Up Less than a second                                                                                               ucp-kube-proxy
7d65c2143f28        docker/ucp-hyperkube:3.0.3           "kubelet --allow-p..."   2 days ago          Up Less than a second                                                                                               ucp-kubelet
82794f34aafc        docker/ucp-agent:3.0.3               "/bin/ucp-agent pr..."   2 days ago          Up Less than a second (healthy)   0.0.0.0:6444->6444/tcp, 0.0.0.0:12378->12378/tcp, 0.0.0.0:12376->2376/tcp         ucp-proxy

(Sunnydockerid) #2

Same here. Usually the monitor should not use so many resource . What is the solution ?


(Billgercken) #3

Agreed. At least I am not the only one. I am trying to resolve whether it is a configuration issue for me or not. I am running in mixed mode. I am going to switch to swarm mode to see if that helps. Will update if I resolve anything.


(Ribragimovalfa) #4

We have the same problem on the same version docker EE.