Docker: Error response from daemon: failed to create shim task: OCI runtime create failed

hi,

I updated the docker and restarted, still getting the same error:

docker run --rm --gpus all nvidia/cuda:12.3.0-runtime-ubuntu20.04 nvidia-smi
Unable to find image ‘nvidia/cuda:12.3.0-runtime-ubuntu20.04’ locally
12.3.0-runtime-ubuntu20.04: Pulling from nvidia/cuda
96d54c3075c9: Already exists
f35fac6f9729: Already exists
c16243f33326: Already exists
752b1f8b6764: Already exists
7d4f0f8effa7: Already exists
71301cc2f14c: Pull complete
0e0ac8025572: Pull complete
e06ff93fa7b0: Pull complete
ce791b2b9978: Pull complete
Digest: sha256:028088b95354c8cf7578c670f338d32719fef19c9a2ccdfdd5e987d3d2d4ccb0
Status: Downloaded newer image for nvidia/cuda:12.3.0-runtime-ubuntu20.04
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as ‘legacy’
nvidia-container-cli: initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: no such file or directory: unknown.

nvidia-smi
Sun Jan 28 18:14:27 2024
±--------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.29.06 Driver Version: 545.29.06 CUDA Version: 12.3 |

docker --version
Docker version 25.0.1, build 29cf629

echo $LD_LIBRARY_PATH
/opt/ros/noetic/lib:/opt/ros/noetic/lib/x86_64-linux-gnu:/usr/lib:/usr/lib/x86_64-linux-gnu:/usr/local/cuda-12.3/lib64::/usr/lib64

ls -l /usr/lib/libnvidia-ml.so.1
lrwxrwxrwx 1 root root 41 Jan 28 17:49 /usr/lib/libnvidia-ml.so.1 → /usr/lib/i386-linux-gnu/libnvidia-ml.so.1
ls -l /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1
lrwxrwxrwx 1 root root 25 Jan 26 21:28 /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1 → libnvidia-ml.so.545.29.06

From a tutorial:

Before diving into the integration of the CUDA Toolkit with Docker, it’s imperative to have the NVIDIA drivers correctly installed on your system. These drivers act as the bridge between your operating system and the NVIDIA GPU hardware