Unable to reconcile state of Kubernetes addons component

Here are the logs

ERRO[0653] Unable to successfully setup local node. Run “docker logs ucp-reconcile” for more details
FATA[0653] reconcile exited with non-zero status: 1
[root@ucp-test ~]# docker logs ucp-reconcile
{“level”:“info”,“msg”:“Configuring node as agent with the following SANs: [kubernetes.default.svc kubernetes.default.svc.cluster.local 127.0.0.1 localhost proxy.local 172.17.0.1 kubernetes kubernetes.default ucp-controller.kube-system.svc.cluster.local compose-api.kube-system.svc 10.0.0.1]”,“time”:“2019-01-14T15:15:25Z”}
{“level”:“info”,“msg”:“Reconciling state of component Docker Proxy”,“time”:“2019-01-14T15:15:25Z”}
{“level”:“info”,“msg”:“Reconciling state of component Certificates”,“time”:“2019-01-14T15:15:26Z”}
{“level”:“info”,“msg”:“Reconciling state of component Concurrent [Client CA Cluster CA Analytics Kubelet Kubernetes Proxy legacymetrics Concurrent [ucp-agent-service ucp-agent-win-service ucp-agent-s390x-service] interlockservice [etcd Exclusive RethinkDB Concurrent [eNZi Secret Kubernetes API Server] Concurrent [Swarm-Classic Manager Concurrent [eNZi API x86_64 service eNZi API s390x service] Concurrent [eNZi Worker x86_64 service eNZi Worker s390x service] Kubernetes Scheduler Kubernetes Controller Manager]]]”,“time”:“2019-01-14T15:15:28Z”}
{“level”:“info”,“msg”:“Reconciling components [etcd Exclusive RethinkDB Concurrent [eNZi Secret Kubernetes API Server] Concurrent [Swarm-Classic Manager Concurrent [eNZi API x86_64 service eNZi API s390x service] Concurrent [eNZi Worker x86_64 service eNZi Worker s390x service] Kubernetes Scheduler Kubernetes Controller Manager]]”,“time”:“2019-01-14T15:15:28Z”}
{“level”:“info”,“msg”:“Reconciling state of component etcd”,“time”:“2019-01-14T15:15:28Z”}
{“level”:“info”,“msg”:“successfully reconciled state of legacymetrics component”,“time”:“2019-01-14T15:15:28Z”}
{“level”:“info”,“msg”:“successfully reconciled state of ucp-agent-s390x-service component. This component will enable UCP on s390x linux nodes if they are added to the cluster”,“time”:“2019-01-14T15:15:28Z”}
{“level”:“info”,“msg”:“successfully reconciled state of interlockservice component. This component will enable the interlock load balancing solution on the UCP cluster.”,“time”:“2019-01-14T15:15:28Z”}
{“level”:“info”,“msg”:“successfully reconciled state of ucp-agent-service component. This component will enable UCP on x86_64 linux nodes if they are added to the cluster”,“time”:“2019-01-14T15:15:28Z”}
{“level”:“info”,“msg”:“successfully reconciled state of ucp-agent-win-service component. This component will enable UCP on x86_64 windows nodes if they are added to the cluster”,“time”:“2019-01-14T15:15:28Z”}
{“level”:“info”,“msg”:“successfully reconciled state of Concurrent [ucp-agent-service ucp-agent-win-service ucp-agent-s390x-service] component”,“time”:“2019-01-14T15:15:28Z”}
{“level”:“warning”,“msg”:“Error when accessing /var/lib/docker/ucp/ucp-kv/member/snap: lstat /var/lib/docker/ucp/ucp-kv/member/snap: no such file or directory”,“time”:“2019-01-14T15:15:28Z”}
{“level”:“warning”,“msg”:“Error when accessing /var/lib/docker/ucp/ucp-kv/datav3/member/snap: lstat /var/lib/docker/ucp/ucp-kv/datav3/member/snap: no such file or directory”,“time”:“2019-01-14T15:15:28Z”}
{“level”:“info”,“msg”:“successfully reconciled state of Analytics component”,“time”:“2019-01-14T15:15:28Z”}
{“level”:“info”,“msg”:“Starting up ucp-kube-proxy container”,“time”:“2019-01-14T15:15:28Z”}
{“level”:“info”,“msg”:“Starting up ucp-kubelet container”,“time”:“2019-01-14T15:15:28Z”}
{“level”:“info”,“msg”:“successfully reconciled state of Kubernetes Proxy component”,“time”:“2019-01-14T15:15:28Z”}
{“level”:“info”,“msg”:“successfully reconciled state of Kubelet component”,“time”:“2019-01-14T15:15:29Z”}
{“level”:“info”,“msg”:“successfully reconciled state of Client CA component”,“time”:“2019-01-14T15:15:29Z”}
{“level”:“info”,“msg”:“successfully reconciled state of Cluster CA component”,“time”:“2019-01-14T15:15:29Z”}
{“level”:“info”,“msg”:“Reconciling state of component Exclusive RethinkDB”,“time”:“2019-01-14T15:15:31Z”}
{“level”:“info”,“msg”:“Creating the UCP database”,“time”:“2019-01-14T15:15:36Z”}
{“level”:“info”,“msg”:“Waiting for database ucp to exist”,“time”:“2019-01-14T15:15:36Z”}
{“level”:“info”,“msg”:“Creating initial collections”,“time”:“2019-01-14T15:15:37Z”}
{“level”:“info”,“msg”:“Reconciling state of component Concurrent [eNZi Secret Kubernetes API Server]”,“time”:“2019-01-14T15:15:37Z”}
{“level”:“info”,“msg”:“successfully reconciled state of eNZi Secret component”,“time”:“2019-01-14T15:15:37Z”}
{“level”:“info”,“msg”:“Created a new Kubernetes master config and stored in etcd”,“time”:“2019-01-14T15:15:37Z”}
{“level”:“info”,“msg”:“Starting up ucp-kube-apiserver container”,“time”:“2019-01-14T15:15:37Z”}
{“level”:“info”,“msg”:“successfully reconciled state of Kubernetes API Server component”,“time”:“2019-01-14T15:15:38Z”}
{“level”:“info”,“msg”:“Reconciling state of component Concurrent [Swarm-Classic Manager Concurrent [eNZi API x86_64 service eNZi API s390x service] Concurrent [eNZi Worker x86_64 service eNZi Worker s390x service] Kubernetes Scheduler Kubernetes Controller Manager]”,“time”:“2019-01-14T15:15:38Z”}
{“level”:“info”,“msg”:“Service ucp-auth-api-s390x is desired to be running but is not running”,“time”:“2019-01-14T15:15:38Z”}
{“level”:“info”,“msg”:“Starting up ucp-kube-controller-manager container”,“time”:“2019-01-14T15:15:38Z”}
{“level”:“info”,“msg”:“Starting up ucp-kube-scheduler container”,“time”:“2019-01-14T15:15:38Z”}
{“level”:“info”,“msg”:“Service ucp-auth-api is desired to be running but is not running”,“time”:“2019-01-14T15:15:38Z”}
{“level”:“info”,“msg”:“Service ucp-auth-worker-s390x is desired to be running but is not running”,“time”:“2019-01-14T15:15:38Z”}
{“level”:“info”,“msg”:“Service ucp-auth-worker is desired to be running but is not running”,“time”:“2019-01-14T15:15:38Z”}
{“level”:“info”,“msg”:“successfully reconciled state of eNZi API s390x service component. This component will enable eNZi API servers on s390x linux nodes if they are added to the cluster”,“time”:“2019-01-14T15:15:38Z”}
{“level”:“info”,“msg”:“successfully reconciled state of eNZi Worker s390x service component. This component will enable eNZi workers on s390x linux nodes if they are added to the cluster”,“time”:“2019-01-14T15:15:38Z”}
{“level”:“info”,“msg”:“successfully reconciled state of Kubernetes Scheduler component”,“time”:“2019-01-14T15:15:38Z”}
{“level”:“info”,“msg”:“successfully reconciled state of Kubernetes Controller Manager component”,“time”:“2019-01-14T15:15:39Z”}
{“level”:“info”,“msg”:“successfully reconciled state of Swarm-Classic Manager component”,“time”:“2019-01-14T15:15:39Z”}
{“level”:“info”,“msg”:“successfully reconciled state of eNZi Worker x86_64 service component. This component will enable eNZi workers on x86_64 linux nodes if they are added to the cluster”,“time”:“2019-01-14T15:15:43Z”}
{“level”:“info”,“msg”:“successfully reconciled state of Concurrent [eNZi Worker x86_64 service eNZi Worker s390x service] component”,“time”:“2019-01-14T15:15:43Z”}
{“level”:“info”,“msg”:“successfully reconciled state of eNZi API x86_64 service component. This component will enable eNZi API servers on x86_64 linux nodes if they are added to the cluster”,“time”:“2019-01-14T15:15:45Z”}
{“level”:“info”,“msg”:“successfully reconciled state of Concurrent [eNZi API x86_64 service eNZi API s390x service] component”,“time”:“2019-01-14T15:15:45Z”}
{“level”:“info”,“msg”:“successfully reconciled state of [etcd Exclusive RethinkDB Concurrent [eNZi Secret Kubernetes API Server] Concurrent [Swarm-Classic Manager Concurrent [eNZi API x86_64 service eNZi API s390x service] Concurrent [eNZi Worker x86_64 service eNZi Worker s390x service] Kubernetes Scheduler Kubernetes Controller Manager]] component”,“time”:“2019-01-14T15:15:45Z”}
{“level”:“info”,“msg”:“Reconciling state of component UCP Controller”,“time”:“2019-01-14T15:15:45Z”}
{“level”:“info”,“msg”:“Reconciling state of component Kubernetes CNI Plugin”,“time”:“2019-01-14T15:15:54Z”}
{“level”:“info”,“msg”:“Deploying addon calico”,“time”:“2019-01-14T15:15:54Z”}
{“level”:“info”,“msg”:“Waiting for kubernetes node ucp-test.novalocal to become ready”,“time”:“2019-01-14T15:16:03Z”}
{“level”:“info”,“msg”:“Addon calico was deployed successfully”,“time”:“2019-01-14T15:16:03Z”}
{“level”:“info”,“msg”:“Reconciling state of component Kubernetes addons”,“time”:“2019-01-14T15:16:03Z”}
{“level”:“info”,“msg”:“Deploying addon kubedns”,“time”:“2019-01-14T15:16:03Z”}
{“level”:“info”,“msg”:“Addon kubedns was deployed successfully”,“time”:“2019-01-14T15:16:05Z”}
{“level”:“info”,“msg”:“Deploying addon ucp-controller”,“time”:“2019-01-14T15:16:05Z”}
{“level”:“info”,“msg”:“Deploying addon ucp-metrics”,“time”:“2019-01-14T15:16:06Z”}
{“level”:“info”,“msg”:“Deploying addon compose”,“time”:“2019-01-14T15:16:07Z”}
{“level”:“info”,“msg”:“Checking installation state”,“time”:“2019-01-14T15:16:07Z”}
{“level”:“info”,“msg”:“Install image with tag “e9c8673f4fd3df10a90e1542aa9bfde8e300f582” in namespace “kube-system””,“time”:“2019-01-14T15:16:07Z”}
{“level”:“fatal”,“msg”:“unable to reconcile state of Kubernetes addons component: error while deploying addon compose: context deadline exceeded”,“time”:“2019-01-14T15:26:04Z”}

1 Like