Using NVIDIA GPU with docker swarm started by docker-compose file

Docker Engine Swarm
1

Hi there,

I have multiple GPU machines and want to run docker swarm on them where each image uses 1 of the available Nvidia GPUs.

I can’t find a good solution online. I can run the following docker-compose file with docker-compose up:

version: '3.7'
services:
  test:
    image: nvidia/cuda:10.2-base
    command: nvidia-smi
    deploy:
      resources:
        reservations:
          devices:
          - driver: nvidia
            count: 1
            capabilities: [gpu, utility]

but I cannot run it with docker swarm:

> docker stack deploy -c docker-compose.yml gputest
services.test.deploy.resources.reservations Additional property devices is not allowed

running a compose file like this

version: '3.7'
services:
  test:
    image: nvidia/cuda:10.2-base
    command: nvidia-smi -l
    deploy:
      resources:
        limits:
          cpus: '0.1'
          memory: 500M
        reservations:
          cpus: '0.001'
          memory: 200M
          generic_resources:
            - discrete_resource_spec:
                kind: 'gpu'
                value: 1

i can deploy it to the stack but it cannot find any nodes to run the images

What would be the best configuration? If possible: I want to use docker-compose files because they are quite easy to handle.

2

bumping this thread. Has somebody any idea how to do this?

3

Have you tried using the nvidia-container-toolkit? You might have to switch your default runtime to nvidia.

4

This might be relevant for you too: Docker Swarm GPU Support