Nvidia runtime error

General
1

Hi

I am running a desktop with a 3700x, 64GB ram, a 3060 12GB and a (lot of) fresh install(s) of Linux Mint Mate.

I am trying to install an AI based document management-search system from https://github.com/theaiautomators/self-hosted-ai-starter-kit

The commands are:-

git clone https://github.com/theaiautomators/self-hosted-ai-starter-kit.git

cd self-hosted-ai-starter-kit

cp .env.example .env # you should update secrets and passwords inside

docker compose --profile gpu-nvidia up

I can not get the Nvidia gpu to work with Docker. I have tried installing Docker with the instructions at https://docs.docker.com/engine/install/ubuntu/ also (on a new install) the Docker desktop version with the instructions at https://docs.docker.com/desktop/setup/install/linux/ubuntu/

Also, I have tried without and with installing the Nvidia toolkit from https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html

Unfortunately I always get the same result.

After I run the pasted commands the install process continues until it finds the following error

✘ Container docling Error response from daemon: unknown or invalid runtime name: nvidia

I have had many long chats with the ever helpful Gordon but have reached an endpoint with that process, Gordon starts suggesting I should stop trying to use the GPU and instead work with the cpu.

I am hopeful someone might have a solution, a suggestion to help resolve my problem

Thanks

2

Docker Desktop for Linux does not pass through the host GPU. Docker Desktop for Windows (with WSL2 backend) and Docker Desktop for Mac are able to use GPU acceleration, as the underlying virtualization solutions have this feature built-in.

On Linux, your only way to make GPU acceleration work with containers, is by using the standalone Docker Engine (=the docker-ce package) and install and configure the NVIDIA toolkit. Your link to the NVIDIA toolkit docs points to the required documenation.