Nvdia-smi showing CUDA:N/A since latest Nvidia driver update on host

[shawneau (Shawn Oliver)]

Hi everyone, I’m on Ubuntu 22.04 and recently updated to Cuda 12.8 and 570 driver from Nvidia.

The host shows everything proper in nvidia-smi, but even with the toolkit installed, all of my docker containers show nvdia-smi output but with CUDA being N/A.

I’ve searched google, I’ve tried troubleshooting with LLMs, I reinstalled docker etc. but nothing is working. I need some help. Is there anything stupid I’m missing? Maybe my version on Ubuntu doesn’t jive well with the latest Nvidia driver? It’s stable on host…

Someone please enlighten me I’m out of ideas!

Thanks a lot!

[rimelek (Ákos Takács)]

Without knowing what you did exactly, how you installed Docker and the nvidia driver, and how you start a container, we can’t tell what you do wrong.

Normally, you would install Docker from the official repo

Ubuntu

Install the correct nvidia driver, and also install the NVidia container toolkit

https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html

This steps are mentioned in the Docker documentation in the GPU section

https://docs.docker.com/engine/containers/resource_constraints/#gpu

which also shows a command to test the installation

docker run -it --rm --gpus all ubuntu nvidia-smi

It also mentions environment variables you can set or use the cuda base image.

Update:

I just missed the fact that it worked before and the update broke it. So I’m not sure why it happened.

Update 2:

You can check this compatibility documentation

https://docs.nvidia.com/deploy/cuda-compatibility/#cuda-11-and-later-defaults-to-minor-version-compatibility