Docker Community Forums

Share and learn in the Docker community.

UCP 3.2.1 Installation Fails on Ubuntu 18.04/Docker EE 19.03

I am trying to install UCP 3.2.1 on a single node Ubuntu 18.03/Docker EE 19.03 env. This is a trial instance to try out UCP console.

My Attempt to bring up UCP 3.21 is unsuccessful as the UCP installation fails with

mkdir /var/lib/docker permission denied.

All my other docker cli commands execute fine and not sure what manual permission needs to be given on /var/lib/docker folder which is owned by root as created by root as part of docker ee engine creation.

here are the error messages from the container (docker/ucp-controller:3.1.10 ) log.

docker: permission denied",“time”:“2019-09-27T03:51:30Z”}

{“level”:“fatal”,“msg”:“unable to initialize authenticator: unable to lookup authenticator config in kv store: Key not found in store”,“time”:“2019-09-27T03:51:30Z”}

{“level”:“info”,“msg”:“orca version 3.1.10 (2828b02)”,“time”:“2019-09-27T03:52:31Z”}

{“level”:“warning”,“msg”:“Unable to create a kube client: unable to copy kube certs from ucp-proxy: unable to create directory: mkdir /var/lib/docker: permission denied”,“time”:“2019-09-27T03:52:32Z”}

{“level”:“warning”,“msg”:“Unable to create a kube client: unable to copy kube certs from ucp-proxy: unable to create directory: mkdir /var/lib/docker: permission denied”,“time”:“2019-09-27T03:52:32Z”}

{“level”:“fatal”,“msg”:“unable to initialize authenticator: unable to lookup authenticator config in kv store: Key not found in store”,“time”:“2019-09-27T03:52:32Z”}

{“level”:“info”,“msg”:“orca version 3.1.10 (2828b02)”,“time”:“2019-09-27T03:53:33Z”}

{“level”:“warning”,“msg”:“Unable to create a kube client: unable to copy kube certs from ucp-proxy: unable to create directory: mkdir /var/lib/docker: permission denied”,“time”:“2019-09-27T03:53:34Z”}

{“level”:“warning”,“msg”:"Unable to create a kube client: unable to copy kube certs from ucp-proxy: unable to create directory: mkdir /var/lib/

ucp install command

docker container run --rm -it --name ucp
-v /var/run/docker.sock:/var/run/docker.sock
docker/ucp:3.2.1 install
–host-address
–interactive

docker/ucp-controller:3.1.10 is restarting and not successfully started.

7ec407b735fd docker/ucp-controller:3.1.10 “/bin/controller ser…” 20 seconds ago Restarting (1) 3 seconds ago ucp-controller
7fad58250a9a docker/ucp-hyperkube:3.1.10 “/bin/apiserver_entr…” 21 seconds ago Up 19 seconds 0.0.0.0:12388->12388/tcp ucp-kube-apiserver
b566c71e0f76 docker/ucp-agent:3.1.10 “/bin/ucp-agent reco…” 23 seconds ago Up 21 seconds 2376/tcp ucp-reconcile
36428142b689 docker/ucp-agent:3.1.10 “/bin/ucp-agent agent” 31 seconds ago Up 25 seconds 2376/tcp ucp-agent.upl7ngtl589s27nm4yhcr4xjp.bl96lt3luy0s2neivaakru0v2
dcc261ae137f docker/ucp-auth:3.1.10 “/usr/local/bin/enzi…” 5 minutes ago Up 5 minutes (healthy) ucp-auth-api.upl7ngtl589s27nm4yhcr4xjp.mkf5xr5yjz42mhx9i3chdtkx6
28883e8c5b95 docker/ucp-auth:3.1.10 “/usr/local/bin/enzi…” 5 minutes ago Up 5 minutes (healthy) ucp-auth-worker.upl7ngtl589s27nm4yhcr4xjp.xadp7afh0o2bpy2p20i7qlffl
53006ed2e619 docker/ucp-swarm:3.1.10 “/bin/swarm manage -…” 5 minutes ago Up 5 minutes (healthy) 0.0.0.0:2376->2375/tcp ucp-swarm-manager
595f6adce400 docker/ucp-hyperkube:3.1.10 “kube-controller-man…” 5 minutes ago Up 5 minutes (healthy) ucp-kube-controller-manager
7f5110584d48 docker/ucp-hyperkube:3.1.10 “kube-scheduler --ku…” 5 minutes ago Up 5 minutes (healthy) ucp-kube-scheduler
1fdc87e1e936 docker/ucp-auth-store:3.1.10 “rethinkdb --bind al…” 5 minutes ago Up 5 minutes (healthy) 0.0.0.0:12383-12384->12383-12384/tcp ucp-auth-store
580b0c71abce docker/ucp-hyperkube:3.1.10 “/bin/kubelet_entryp…” 5 minutes ago Up 5 minutes ucp-kubelet
3a20acb1dfdc docker/ucp-etcd:3.1.10 “/bin/entrypoint.sh …” 5 minutes ago Up 5 minutes (healthy) 2380/tcp, 4001/tcp, 7001/tcp, 0.0.0.0:12380->12380/tcp, 0.0.0.0:12379->2379/tcp ucp-kv
f978ae53fb36 docker/ucp-cfssl:3.1.10 “/bin/ucp-ca serve -…” 5 minutes ago Up 5 minutes (healthy) 0.0.0.0:12382->12382/tcp ucp-client-root-ca
6600c10a2d84 docker/ucp-cfssl:3.1.10 “/bin/ucp-ca serve -…” 5 minutes ago Up 5 minutes (healthy) 0.0.0.0:12381->12381/tcp ucp-cluster-root-ca
813b26398eca docker/ucp-hyperkube:3.1.10 “kube-proxy --cluste…” 5 minutes ago Up 5 minutes ucp-kube-proxy
d37e1302955c docker/ucp-agent:3.1.10 “/bin/ucp-agent prox…” 5 minutes ago Up 5 minutes (healthy) 0.0.0.0:6444->6444/tcp, 0.0.0.0:12378->12378/tcp, 0.0.0.0:12376->2376/tcp ucp-proxy

any help to resolve this is highly appreciated.

Did you remove the value for the parameter --host-address before you pasted it? it is missing. Are you required to use the parameter?

To be able to get an understanding of your situation, please run following commands and paste their output:

  1. id
  2. ls -l /var/lib/ | grep -E 'docker|kubelet'
  3. df -h /var/lib/docker (more than 20gb spacerecommended)
  4. df -h /var/lib/kubelet (requires at least 6gb space)

If you didn’t tinker around with the /var/lib/docker folder, all permissions should be fine.

Metin,

I did not remove the value for --host address. Here is the actual command that ran

docker container run --rm -it --name ucp
-v /var/run/docker.sock:/var/run/docker.sock
docker/ucp:3.1.10 install
–host-address 150.136.244.202
–interactive

Other request command output

id

groups=1001(ubuntu),4(adm),20(dialout),24(cdrom),25(floppy),27(sudo),29(audio),30(dip),44(video),46(plugdev),108(lxd),114(netdev),999(docker)

ubuntu@instance-20190926-1434:~$ ls -l /var/lib/ | grep -E ‘docker|kubelet’
drwx–x--x 14 root root 4096 Sep 27 03:16 docker
drwxr-xr-x 2 root root 4096 Sep 27 03:16 docker-engine
drwxr-xr-x 6 root root 4096 Sep 27 02:06 kubelet

ubuntu@instance-20190926-1434 : ~ $ df -h /var/lib/docker
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 45G 4.8G 41G 11% /

ubuntu@instance-20190926-1434:~$ df -h /var/lib/kubelet
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 45G 4.8G 41G 11% /

I did not tinker with/var/lib/docker.

it is very consistent with this problem. Tried to provision a brand new instance and tried again…same issue.

UCP 3.2.1 times out .out. ucp 3.1.10 get stuck at the above error stage… can’t get past this error event after trying many times.

regards,
Jag

Your user is in the docker group and is therefor able to use the docker command to send commands to the engine using the daemon socket /var/run/docker.sock. The diskspace should be more then sufficient to install UCP.

I can’t see anything wrong… I would suggest to run the command again with --debug. Could it be a problem with AppArmor or SELinux?

here is the last line output from ucp installation in debug mode…

{“level”:“info”,“msg”:“successfully reconciled state of Kubernetes Scheduler component”,“time”:“2019-09-27T17:56:20Z”}

{“level”:“info”,“msg”:“successfully reconciled state of Kubernetes Controller Manager component”,“time”:“2019-09-27T17:56:20Z”}

{“level”:“debug”,“msg”:"Connected to https://150.136.244.202:2376 a2ae8302fb8c ",“time”:“2019-09-27T17:56:20Z”}

{“level”:“debug”,“msg”:“Swarm Manager component reconciled successfully”,“time”:“2019-09-27T17:56:20Z”}

{“level”:“info”,“msg”:“successfully reconciled state of Swarm-Classic Manager component”,“time”:“2019-09-27T17:56:20Z”}

{“level”:“error”,“msg”:“Auth Worker server didn’t come up within 5m0s”,“time”:“2019-09-27T18:01:19Z”}

{“level”:“error”,“msg”:“unable to reconcile state of eNZi Worker x86_64 service component: unable to connect to system”,“time”:“2019-09-27T18:01:19Z”}

{“level”:“error”,“msg”:“unable to reconcile state of Concurrent [eNZi Worker x86_64 service] component: unable to reconcile state of eNZi Worker x86_64 service component: unable to connect to system”,“time”:“2019-09-27T18:01:19Z”}

{“level”:“error”,“msg”:“Auth API server didn’t come up within 5m0s”,“time”:“2019-09-27T18:01:19Z”}

{“level”:“error”,“msg”:“unable to reconcile state of eNZi API x86_64 service component: unable to connect to system”,“time”:“2019-09-27T18:01:19Z”}

{“level”:“error”,“msg”:“unable to reconcile state of Concurrent [eNZi API x86_64 service] component: unable to reconcile state of eNZi API x86_64 service component: unable to connect to system”,“time”:“2019-09-27T18:01:19Z”}

{“level”:“error”,“msg”:“unable to reconcile state of [etcd Exclusive RethinkDB Concurrent [eNZi Secret Kubernetes API Server] Concurrent [Swarm-Classic Manager Concurrent [eNZi API x86_64 service] Concurrent [eNZi Worker x86_64 service] Kubernetes Scheduler Kubernetes Controller Manager]] component: unable to reconcile state of Concurrent [Swarm-Classic Manager Concurrent [eNZi API x86_64 service] Concurrent [eNZi Worker x86_64 service] Kubernetes Scheduler Kubernetes Controller Manager] component: unable to reconcile state of Concurrent [eNZi Worker x86_64 service] component: unable to reconcile state of eNZi Worker x86_64 service component: unable to connect to system”,“time”:“2019-09-27T18:01:19Z”}

{“level”:“fatal”,“msg”:“unable to reconcile state of Concurrent [Client CA Cluster CA Analytics Kubelet Kubernetes Proxy legacymetrics Deprecated UCP services Concurrent [ucp-agent-service ucp-agent-win-service] interlockservice [etcd Exclusive RethinkDB Concurrent [eNZi Secret Kubernetes API Server] Concurrent [Swarm-Classic Manager Concurrent [eNZi API x86_64 service] Concurrent [eNZi Worker x86_64 service] Kubernetes Scheduler Kubernetes Controller Manager]]] component: unable to reconcile state of [etcd Exclusive RethinkDB Concurrent [eNZi Secret Kubernetes API Server] Concurrent [Swarm-Classic Manager Concurrent [eNZi API x86_64 service] Concurrent [eNZi Worker x86_64 service] Kubernetes Scheduler Kubernetes Controller Manager]] component: unable to reconcile state of Concurrent [Swarm-Classic Manager Concurrent [eNZi API x86_64 service] Concurrent [eNZi Worker x86_64 service] Kubernetes Scheduler Kubernetes Controller Manager] component: unable to reconcile state of Concurrent [eNZi Worker x86_64 service] component: unable to reconcile state of eNZi Worker x86_64 service component: unable to connect to system”,“time”:“2019-09-27T18:01:19Z”}

{“level”:“info”,“msg”:“ucp-reconcile container exited with status code: 1”,“time”:“2019-09-27T18:01:20Z”}

{“level”:“info”,“msg”:“Completed state reconciliation, system is ready.”,“time”:“2019-09-27T18:01:20Z”}

ERRO[0340] Unable to successfully setup local node. Run “docker logs ucp-reconcile” for more details

FATA[0340] reconcile exited with non-zero status: 1

Debug also not giving anything better,
tried even UCP 3.0.7 . no luck either…same error

{“level”:“warning”,“msg”:“Unable to create a kube client: unable to copy kube certs from ucp-proxy: unable to create directory: mkdir /var/lib/docker: permission denied”,“time”:“2019-09-27T18:27:42Z”}

{“level”:“debug”,“msg”:“Copying /etc/docker/ssl certs from ucp-proxy”,“time”:“2019-09-27T18:27:42Z”}

{“level”:“debug”,“msg”:“writing file /var/lib/docker/ucp/ucp-node-certs/ssl”,“time”:“2019-09-27T18:27:43Z”}

{“level”:“warning”,“msg”:“Unable to create a kube client: unable to copy kube certs from ucp-proxy: unable to create directory: mkdir /var/lib/docker: permission denied”,“time”:“2019-09-27T18:27:43Z”}

{“level”:“debug”,“msg”:“Detected ucp-auth-store, marking current node state as a manager”,“time”:“2019-09-27T18:27:43Z”}

{“level”:“debug”,“msg”:“Initializing configurations”,“time”:“2019-09-27T18:27:43Z”}

{“level”:“debug”,“msg”:“Starting signal watcher to detect config updates”,“time”:“2019-09-27T18:27:43Z”}

{“level”:“fatal”,“msg”:“Unable to setup manager: unable to setup authenticator: unable to setup auth config subsystem: Key not found in store”,“time”:“2019-09-27T18:27:43Z”}

Honestly, I never experienced such problems…

@bryceryan could you breach in an add same insights here?