Database (Mongo) volume persistance Docker Swarm

We are implementing a 3 nodes Swarm cluster for deploying microservices apps. All of the microservice in this apps use their own DB container (Database per service pattern), and use an Event-Bus for sharing data across Microservices. Example:

  ms_1:
    image: AnImageWebApp
    depends_on:
      - ms_2-mongo
    networks:
      - traefik_public_ntw
      - ms_1_mongo_ntw
  #### Microservice ####
  ms_1-mongo:
    image: mongo:3.6.19-xenial
    networks:
      - ms_1_mongo_ntw
    volumes:
      ms_1-mongo:/data/db
  #### Microservice ####
  ms_2:
    image: AnImageWebApp2
    depends_on:
      - ms_2-mongo
    networks:
      - traefik_public_ntw
      - ms_2_mongo_ntw
  #### Microservice ####
  ms_2-mongo:
    image: mongo:3.6.19-xenial
    networks:
      - ms_2_mongo_ntw
    volumes:
      ms_2-mongo:/data/db

So far, everything works, but I have some questions on the behaviour of Docker Swarm and how to avoid problem with volumes (data persistence).

Node Fail

If a node fail, Docker Swarm will detect that the desired state is different of the actual state, and it will reconcile automatically by rescheduling missing containers on other available nodes.

This will cause problem in the case of the Mongo service because as the volume is missing on the other node, the data will not be available in the new Task. One possible solution would be to store the mongo volumes on an NFS storage (shared with all nodes in the cluster). In this case, if the node fail, the task will be reschedule on other nodes and the volume will be find by the task.

Question: Does putting volumes on an NFS storage is a good idea especially in term of performance ?

Performance for some microservice

Some of our microservice will have to manage a lots of write and read operations (IoT telemetry datas). This can cause, in the future, some performances issues. We are thinking about deploying a Sharded Cluster for this particular microservices. Instead of one mongo service, we will implement several services that will compose the cluster (because we, of course, cannot use the replicas feature for this use case). We can avoid using NFS share in this case but we will have to “stick” service task to one node by ourself (not letting swarm spreading the task but do it ourself basically).

Another idea would be to build a shared cluster (such as above) but for all the microservice of the app. The microservices will still have their own Database but, instead of deploying a mongo service per microservice, we just have one cluster for all the microservice app (all of our microservice use Mongo as storage) and the microservices (web app) will use the cluster as storage.

What would be the best strategy for this use case ? Does anyone faced this problem before ?