Seeking advice for a Storage Volume Plugin

Docker Engine Swarm
1

I am about to modify my homelab heavily, though there is one thing realy challenging me: which storage plugin suits my needs?

I have a manager nodes only homelab with 2x x86_64 hosts having nvme and ssd storages attached to them, on top i have a Raspberry Pi which I added as a manager node without any storage attached. The Pi is my sacrifice to keep the swarm manager consensus happy in case one of the x86_64 are down for maintanance.

My objective is to have my container storage beeing replicated on both hosts. I figured the way is either a storage cluster (glusterfs, moosefs, ceph) where replication is done on storage level or a cluster of storages (implemenations of Cloud Native Storage or Container Storage Interface like portworx, storageos, openebs) where replication is done on container level.

Now the question is which one of those to pick, as I am quite sure that most of these solutions use a consensus algorithm, like raft, to organize the replication. While the Pi works like a charm as a swarm manager node, I doubt it will be of any use as a consensus node for any sort of storage solution.

The local storage volume plugin (and if required a backend solution) which is fine for host bound containers (local disk/nfs), but is not sufficient for what I want.

What i need is a storage plugin that replicates the data on both host so scheduling on one of both x86_64 nodes is possible. The volumes are mainly for file based configuration, but some will be database files (which works terrible on nfs shares).

What is your experience with Storage plugins and which one would you recommend on my setup?

2

I would recommend looking into GlusterFS and mount the shared storage into a consistent location on all nodes. Portworx is another good option, I haven’t used it personally but I heard good things about it.

3

Thank you for the response.

Do you by any chance know if GlusterFS or Portworx will work if only two “Storage nodes” are available?

4

GlusterFS requires 3 nodes to avoid a split brain, however it has the option to use Arbiter, I suppose is exactly what you’re looking for - it allows you to add a node that will be a part of a cluster to prevent a split brain but won’t actually hold any data.

https://docs.gluster.org/en/latest/Administrator%20Guide/arbiter-volumes-and-quorum/

5

Indeed, It is worth trying if the Pi is powerfull enough to be used as an Arbiter.

Thanks :slight_smile:

Update:
I am still gathing facts for the other candidates. At least i know now, that GlusterFS is the fallback solution if i am not able to find a better option.

Seems like Portworx provides a simiar solution for the commercial product: Storage-less Nodes with PX Enterprise, though the free version PX Developer does not.

StorageOS depends on ETCD for the consenus, which itself required a 3 node setup. One of the etcd nodes would need to be run on the PI.

Ceph uses monitoring nodes for the consensus, thus one of the monitors would be run on the PI.

6

We did a PoC with GlusterFS on a heavy write scenario (SolrCloud). But I didnt make it do production because:

  1. wirte performance is 3 times as slow with replaction 3 (that we knew beforhand)
  2. GlusterFS needs utilized a lot of CPU
  3. we were able to crash glusterfs on very heavy write scenarios (high cpu utilization)
7

Thanks for sharing your insights!

8

I finaly found time to setup StorageOS for my mixed architecure homelab (2x on x86_64, 1x on arm64) with Docker-CE 18.09.5 on Ubuntu 18.04.2.

As a precondition to permit a fault tollerate cluster state, I created a compose.yml for a mixed architecture 3 node etcd3 swarm deployment and started the stack.

Once the etcd3 nodes joined the cluster, I followed the instrcutions on the hompage:

Install StorageOS CLI (on the x86_64 nodes):

curl -sSL https://github.com/storageos/go-cli/releases/download/1.2.0/storageos_linux_amd64 > /usr/local/bin/storageos chmod +x /usr/local/bin/storageos

Set the usercredentials and node ip for StorageOS:

export STORAGEOS_USERNAME=storageos STORAGEOS_PASSWORD=storageos STORAGEOS_HOST=192.168.200.35

Run a one time container to initialize the dependenies on the host:

docker run --name enable_lio \
           --privileged \
           --rm \
           --cap-add=SYS_ADMIN \
           --volume /lib/modules:/lib/modules \
           --volume /sys:/sys:rshared \
           storageos/init:0.1

Install the StorageOS node and containerized plugin:

docker run -d \
  --name=storageos \
  --env=LOG_LEVEL=debug \
  --env=HOSTNAME=docker1 \
  --env=ADVERTISE_IP=192.168.200.35 \
  --env=JOIN=192.168.200.36 \
  --env=DISABLE_TELEMETRY=true \
  --env=KV_BACKEND=etcd \
  --env=KV_ADDR=192.168.200.34:2379,192.168.200.35:2379,192.168.200.36:2379 \
  --net=host \
  --pid=host \
  --privileged \
  --cap-add=SYS_ADMIN \
  --device=/dev/fuse \
  --volume=/var/lib/storageos:/var/lib/storageos:rshared \
  --volume=/run/docker/plugins:/run/docker/plugins \
  --volume=/sys:/sys \
  storageos/node:1.2.0 server

ADVERTISE_IP is the ip of the current node to install. On the first node, JOIN will use the same ip. The installation of the next node needs its own ip, but JOIN will still point to the ip of the first node.

KV_BACKEND sets the key/value store for the cluster state. KV_ADDR is a commana separated list of “ip:port” for the key/value store members.

Notes:
If and KV_BACKEND and KV_ADDR are not provided, an internal etcd instance will be used which will be installed on the first initilized instance - with no faul tollerance. In previous versions consul was prefered over etcd, but in one of the recent updates they obsoleted consul and switched to etcd.

There is a Volume Plugin available in the Docker Store, thogh I was not able to install it due a missing latest tag.

The default license allows a total of 100GB storage. StorageOS offers a free developer license which allows a total of 500gb storage, which should be plenty for most development/evaluation tasks.

StorageOS can not be operated as a swarm stack.