SOLVED: iSCSI login fails from docker plugin

Hi,

I am facing issue while iSCSI login to iSCSI target from within a docker plugin.
I have a GO code which performs iSCSI operations from within the plugin, but when I try to perform iSCSI login from within my plugin code, it cannot log in to the portals, it throws an error saying, iSCSI driver not found, but then if I perform iSCSI login from the docker daemon host manually the login succeeds, and if I run the code again then it returns saying iscsiadm: default: 1 session requested, but 1 already present

So, basically I cannot log in to iSCSI target from the docker plugin, but all the iSCSI operations work successfully from the docker daemon host.

Please help me, I want to run the iSCSI login from the plugin code

Code running iSCSI login:

out, err := exec.Command("iscsiadm", "-m", "node", "-l", "-T", targetIQN).CombinedOutput()
if err != nil {
	log.Fatalf("Error encountered in iscsi login to target '%s': Error code '%s', Error : '%s'", targetIQN, err, string(out))
	return "", errors.New("error doing iscsi Login cmd")
}
log.Debug("Output of login command\n: ", string(out))

Issue:

The iSCSI login fails with the following error:

iscsiadm: initiator reported error (12 - iSCSI driver not found. Please make sure it is loaded, and retry the operation)
iscsiadm: Could not log into all portals

Environment:

1. Docker Daemon Host: Ubuntu 18.04
Linux stack-virtual-machine 4.15.0-45-generic #48~16.04.1-Ubuntu SMP Tue Jan 29 18:03:48 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

2. iSCSI package: open-iscsi

3. Docker Plugin: Ubuntu 18.04
Dockerfile

FROM ubuntu:18.04
ENV GO_VERSION=1.11.4 \
    GOROOT=/goroot \
    GOPATH=/gopath
ENV PATH $PATH:$GOROOT/bin:$GOPATH/bin

# Check if we need to copy initiator file from host to plugin container
RUN mkdir -p /var/lib/docker-volumes/plugin
RUN apt-get update -y && \
    # Install needed packages
    apt-get install --no-install-recommends -y -q curl build-essential git vim apt-transport-https software-properties-common ca-certificates open-iscsi && \
    # Install GOLang
    mkdir ${GOROOT} && curl https://storage.googleapis.com/golang/go${GO_VERSION}.linux-amd64.tar.gz | tar xvzf - -C ${GOROOT} --strip-components=1 && \
    mkdir ${GOPATH} && \
    # remove apt cache from image
    apt-get clean all

# it will work only after making the repo public
COPY . ${GOPATH}/src/github.com/temp/plugins/plugin
COPY initiatorname.iscsi /etc/iscsi/initiatorname.iscsi
WORKDIR ${GOPATH}/src/github.com/temp/plugins/plugin
RUN go get github.com/docker/go-plugins-helpers/volume
RUN go get github.com/Sirupsen/logrus
RUN set -ex \
    && apt-get install gcc libc-dev \
    && go install --ldflags '-extldflags "-static"'
RUN mkdir /etc/plugin
CMD ["/gopath/bin/plugin"]

OS Version/build:
root@virtual-machine:/home/stack# docker version
Client:
Version: 18.03.0-ce
API version: 1.37
Go version: go1.9.4
Git commit: 0520e24
Built: Wed Mar 21 23:10:01 2018
OS/Arch: linux/amd64
Experimental: false
Orchestrator: swarm

This is my config.json for the plugin:

{
  "description": "Docker volume plugin",
  "documentation": "https://docs.docker.com/engine/extend/plugins/",
  "entrypoint": ["/gopath/bin/docker-plugin"],
  "network": {
    "type": "host"
  },
  "Env": [
    {
      "Description": "Config file from /etc/plugin on host",
      "Name": "config",
      "Settable": [
        "value"
      ],
      "Value": "/etc/plugin/plugin-config.json"
    },
    {
      "Description": "Default location of Log file on host",
      "Name": "log-location",
      "Settable": [
        "value"
      ],
      "Value": "/var/log/plugin/plugin.json"
    }
  ],
  "interface" : {
    "types": ["docker.volumedriver/1.0"],
    "socket": "plugin.sock"
  },
  "mounts": [
    {
      "type": "bind",
      "source": "/etc/plugin",
      "destination": "/etc/plugin",
      "options": ["rbind"]
    },
    {
      "type": "bind",
      "source": "/var/log/plugin",
      "destination": "/var/log/plugin",
      "options": ["rbind"]
    },
    {
      "type": "bind",
      "source": "/dev",
      "destination": "/dev",
      "options": ["rbind"]
    }
  ],
  "PropagatedMount": "/var/lib/docker-volumes/plugin",
  "linux": {
    "capabilities": ["CAP_SYS_ADMIN"],
    "allowAllDevices": true
  }
}

I myself know nothing about ISCSI or the ISCSI driver.
According to the error message: iSCSI driver not found. Please make sure it is loaded, and retry the operation

Seems like you need to install the ISCSI driver in your Docker Login Plugin when you build it, and then load it.

https://bbs.archlinux.org/viewtopic.php?id=233606

But if the iSCSI driver wasn’t found, it wouldn’t have performed the iSCSI discovery in first place.
Also, if the iSCSI driver wasn’t found it wouldn’t have worked in the 2nd try of login from the plugin code.

Also, I would like to know if there is a way to log in to a running docker plugin container? so that I can check what exactly is wrong with the docker plugin environment.

Do let me know.

I myself have never written a Docker Plugin or debugged one.
But checking the documentation I found this page: Debugging plugins

Using docker-runc to obtain logfiles and shell into the plugin.

docker-runc , the default docker container runtime can be used for debugging plugins. This is specifically useful to collect plugin logs if they are redirected to a file.

$ sudo docker-runc --root /var/run/docker/plugins/runtime-root/moby-plugins list

ID                                                                 PID         STATUS      BUNDLE                                                                                                                                       CREATED                          OWNER
93f1e7dbfe11c938782c2993628c895cf28e2274072c4a346a6002446c949b25   15806       running     /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby-plugins/93f1e7dbfe11c938782c2993628c895cf28e2274072c4a346a6002446c949b25   2018-02-08T21:40:08.621358213Z   root
9b4606d84e06b56df84fadf054a21374b247941c94ce405b0a261499d689d9c9   14992       running     /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby-plugins/9b4606d84e06b56df84fadf054a21374b247941c94ce405b0a261499d689d9c9   2018-02-08T21:35:12.321325872Z   root
c5bb4b90941efcaccca999439ed06d6a6affdde7081bb34dc84126b57b3e793d   14984       running     /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby-plugins/c5bb4b90941efcaccca999439ed06d6a6affdde7081bb34dc84126b57b3e793d   2018-02-08T21:35:12.321288966Z   root

Just in case you were not already aware, Splunk and SumoLogic have Docker Logging Plugins which are opensource and stored in GitHub. At the Splunk GitHub page they have a troubleshooting section. Some of that information might be useful for you.


Hi @gforghetti,
I tried runnung the docker-runc it doesn’t give any output.

Please find below the execution:

root@virtual-machine:/home# docker plugin ls
ID                  NAME                          DESCRIPTION                              ENABLED
0b237c081040        apmsys/qs-plugin:1.0    Docker volume driver plugin                     true

root@virtual-machine:/home# sudo docker-runc --root /var/run/docker/plugins/runtime-root/moby-plugins list
ID          PID         STATUS      BUNDLE      CREATED     OWNER

root@virtual-machine:/home#

I am not able to understand what could be the problem.
Please help.

Sorry, I have never coded a Docker Volume plugin.
Hopefully someone else here has and can respond.

You can try posting this to the Moby Project forum https://forums.mobyproject.org/c/support

Found a solution to this issue.
I was installing the package “open-iscsi” in the plugin container via the Dockerfile.
I guess the plugin container uses the docker daemon host’s iSCSI driver and I was installing the package again on top of it in the container which was causing the problem.

I removed the package from Dockerfile and exposed the locations where the iSCSI driver file resides from docker daemon host to plugin container and it worked.

Thanks for your help.

Excellent. Glad you got it working and thanks for posting what the actual problem was and how you resolved it.