We have set up a Docker Enterprise 2.1 cluster with 3 worker nodes (docker engine v18.09, UCP v3.1.0), and we are using an external Ceph cluster to provide the persistent storage.
We have attempted to set up Ceph Rados Block Device (RBD) as a Storage Class for Kubernetes, according to an online tutorial (http://tutorial.kubernetes.noverit.com/content/ceph.html), and we are able to create a PVC with bounded PV using the Ceph RDB Storage Class we created. We have checked that the RBD image was in fact dynamically provisioned on the Ceph cluster.
However, during the installation of application using Helm, we are getting a warning event during Pod creation:
MountVolume WaitForAttach failed for volume <name of volume>: fail to check rbd image status with (executable file not found in $PATH), RBD OUTPUT: ()
The installation does not proceed and the containers in the pod never get successfully created. The pod remains stuck with a “ContainerCreating” status.
All worker and manager nodes in the cluster have been installed with the Ceph client libraries, and we are able to communicate with the Ceph cluster from the host OS of all nodes using the “rbd” command.
Can you advice on the error message? Especially the part about “executable file not found in $PATH”. This error appears in the ucp-kubelet logs, but we are unsure of its origin.
Is this a limitation because Docker EE is using ucp-kubelet instead of OS level kubelet?