Engine/Desktop - Cuda - Model Runner - Ubuntu

General
1

Hello,

I am creating a new Ubuntu 24 host. I installed docker engine so I could have all the data on a separate SSD. I installed the nvidia cuda container tools and got nvidia-smi to recognize my 2 gpus. Then I tried to install and then reinstall docker model runner with cuda support. But, when I reinstalled, I still got the “python3 not found” even after specifically installing the backends for vLLM, diffuser, etc.

Then I read that model runner was a Docker Desktop plugin, so I tried to install Docker Desktop, but still no cuda love.

So, IF I was to start all over, is there a good tutorial somewhere on installing Docker with CUDA support on a linux host?

Thanks for any help or pointers.

Best,
Steven

2

GPU support in Docker Desktop is only available on Windows with the WSL2 backend.

But you can find guides on NVIDIA’s website

https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installing-the-nvidia-container-toolkit

and for Docker-specific configuration

https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/docker-specialized.html

But I assume you already read these.

When did you get this error message exactly? Was it in containers?

3

This was the name of the image for the model runner: docker/model-runner:latest-cuda

When I run “docker model status” I get:

Docker Model Runner is running

BACKEND    STATUS  DETAILS
diffusers  Error   python3 not found in PATH
llama.cpp  Error   running llama.cpp version: c55bce4
mlx        Error   not installed
sglang     Error   python3 not found in PATH
vllm       Error   vLLM binary not found

These errors persist even after running:

docker model reinstall-runner --backend vllm --gpu cuda
docker model reinstall-runner --backend diffusers --gpu cuda
docker model reinstall-runner --backend sglang --gpu cuda
docker model reinstall-runner --backend llama.cpp --gpu cuda