I am about to modify my homelab heavily, though there is one thing realy challenging me: which storage plugin suits my needs?
I have a manager nodes only homelab with 2x x86_64 hosts having nvme and ssd storages attached to them, on top i have a Raspberry Pi which I added as a manager node without any storage attached. The Pi is my sacrifice to keep the swarm manager consensus happy in case one of the x86_64 are down for maintanance.
My objective is to have my container storage beeing replicated on both hosts. I figured the way is either a storage cluster (glusterfs, moosefs, ceph) where replication is done on storage level or a cluster of storages (implemenations of Cloud Native Storage or Container Storage Interface like portworx, storageos, openebs) where replication is done on container level.
Now the question is which one of those to pick, as I am quite sure that most of these solutions use a consensus algorithm, like raft, to organize the replication. While the Pi works like a charm as a swarm manager node, I doubt it will be of any use as a consensus node for any sort of storage solution.
The local storage volume plugin (and if required a backend solution) which is fine for host bound containers (local disk/nfs), but is not sufficient for what I want.
What i need is a storage plugin that replicates the data on both host so scheduling on one of both x86_64 nodes is possible. The volumes are mainly for file based configuration, but some will be database files (which works terrible on nfs shares).
What is your experience with Storage plugins and which one would you recommend on my setup?