This is my current network setup:
Changes I want:
All services (Plex, Transmission, etc.) moved to docker (not an issue - I know how to do this, the other requirements are where I get stuck!)
Docker running in swarm mode on Server 1 and Server 2. Docker needs to have its own network with a virtual IP on the 192.168.1.x 1GB/s network and allow local communication on the 10.0.0.x 40GB/s InfiniBand network. Containers must be able to all use the 192.168.1.x network IP. I would also like to run 2 instanstances of PiHole on unique 192.168.1.x IPs, currently these are running on raspberry Pis, but stability is an issue and manual reboots a pain.
If Server 1 or Server 2 goes down, Docker Swarm on the unaffected server should relaunch all affected containers on the other server.
Persistent data for docker containers must be stored locally on both Server 1 and Server 2. There should be immediate duplication between the servers via the 40GB/s 10.0.0.x InfiniBand network. I would prefer to use the boot SSD drives for this data and run a scheduled task or cron within a docker container to backup this data to the drivepool. The drivepool would then have this data replicated to 2 or 3 physical drives.
Non persistant data for docker containers (I.E. container images) can be stored on the boot SSD drives but does not require duplication.
Use of VMs on Server 1 and/or Server 2 with the available hypervisors for Docker Swarm is optional but not preferred. I’ll only do this if it makes it much easier to configure.
Windows subsystem for linux is available on both Server 1 and Server 2 so running linux based docker images directly on the server is not an issue.
1 Is it possible to set up docker to make use of a virtual IP on the 192.168.1.x subnet (while allowing communication between containers on the InfiniBand 10.0.0.x subnet) with automated failover if one server becomes unavailable? If this is possible, what would be the best way to do it?
2. Is it possible to run PiHole on dedicated IPs on the 192.168.1.x subnet (one copy running on each server)? If this is possible, how should I set this up? I’m happy to use dedicated VMs on both Server 1 and Server 2 rather than docker if this makes it easier.
3. For creating and using VMs with this setup, should I use Virtualbox or Hyper-V? VMs will not need failover and only need to be present on one server. VM image files would be stored on the DrivePool and thus could be launched on either server. What is the best way to set up a VM with normal networking on the 192.168.1.x subnet and private communication (data transfer etc) on the 10.0.0.x subnet?
4. I would like to use a container manager like Portainer with this setup to allow for simplicity. I am also not averse to using Docker Create yml files to specify container settings. What container manager would be suggested for my setup?
5. I would like to have additional malware protection and run ClamAV within either a docker container or VM and have it periodically scan the pooled drives. I’ve not seen a container image that will do this - any suggestions?
6. If I wanted to add ClamAV (or similar) scanning to the data in each of my running docker containers, is it possible to create a fork of these containers that adds a ClamAV service to scan the internal container data that will still automatically update when the master is updated? Alternatively is there a dedicated service container that will periodically connect to running docker containers and scan them? Depending on the answer what would be the best way to collated the logs from each container’s scans and flag issues? I would prefer a single “scanning container” that can connect to all other docker images.
7. If the answer to question 1 is “no it isn’t possible”, what alternatives do I have for automated failover. I’m planning to implement Traefik as a virtual edge router allowing domain name based access to containers without worrying about ports, would this work better and simplify things?
8. What would be the simplest method through which to make the pooled drives accessible from outside the network so I can access all the data while not at home? VPN? Dedicated file sharing container?