Enabling GPU Support for Docker on My Laptop

General
1

I’m new to programming, and I’m unfamiliar with the Docker platform. I wanted to use LLAMU 3.1 8B, but it runs too slowly on my laptop, whereas Mistral 7B runs very quickly. It seems that LLAMU 3.1 8B is not utilizing my GPU, as it runs very slowly. How can I configure Docker to recognize and use my GPU (GeForce RTX 4050 Laptop GPU)?

I would greatly appreciate your help.

2

We usually need the following information to understand the situation:

  1. What platform are you using? Windows, Linux or macOS? Which version of the operating systems? In case of Linux, which distribution?

  2. How did you install Docker? Sharing the platform almost answers it, but only almost. Direct links to the followed guide can be useful.

  3. On debian based Linux, the following commands can give us some idea and recognize incorrectly installed Docker:

    docker info
docker version

    Review the output before sharing and remove confidential data if any appears (public IP for example)

    dpkg -l | grep docker
snap list docker

    When you share the outputs, always format your posts according to the following guide: How to format your forum posts

1 Like
3

Thank you for your assistance. As I’m new to programming, I’ll do my best to follow the forum’s structure and culture. I appreciate your patience as I learn, and I apologize for any inadvertent mistakes I might make along the way.

Thank you for your concern about security. I’ve reviewed the information and ensured there are no sensitive details, such as IP addresses, included. I am not sure whether they are exposed or not. However, I understand that it’s always better to err on the side of caution regarding data security. With that in mind, I’m comfortable sharing the following information with you

Below are the details of the Docker Desktop installation on my Windows 11 laptop:

Docker version:

Client:
 Version:           27.1.1
 API version:       1.46
 Go version:        go1.21.12
 Git commit:        6312585
 Built:             Tue Jul 23 19:57:57 2024
 OS/Arch:           windows/amd64
 Context:           desktop-linux

Server: Docker Desktop 4.33.1 (161083)
 Engine:
  Version:          27.1.1
  API version:      1.46 (minimum version 1.24)
  Go version:       go1.21.12
  Git commit:       cc13f95
  Built:            Tue Jul 23 19:57:19 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.19
  GitCommit:        2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
 runc:
  Version:          1.7.19
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

As for the Docker info:

Client:
 Version:    27.1.1
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
   .....

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 27.1.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.153.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 11.54GiB
 Name: docker-desktop
 ID: 771596b7-cb47-4f1d-ac57-faf164d09948
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=npipe://\\.\pipe\docker_cli
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:
  127.
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

I hope this information helps give you some answers on how I can use the GPU on my Docker.
Thank you.

4

I think I have solved my problem or not? I rewrote the code in Docker Engine to do this:

{
  "runtimes": {
    "nvidia": {
      "path": "nvidia-container-runtime",
      "runtimeArgs": []
    }
  }
}

Anyway, LLAMA 3.1 8B started responding faster. But I am not sure if this is achieved by activating the GPU.

5

Have you checked the Docker Desktop on Windows docs about gpu support?

6

I’m feeling a bit stuck and unsure about what to do next. I’ve tried everything step by step, but it seems like the problem might be related to the limits of my RTX 4050 graphics card, which has 6GB of VRAM. From what I understand, models like LLAMA 3.12 8B or Gemma2:9B require a minimum of 8GB of VRAM. This might be the issue, even though they were running smoothly and quickly yesterday!

7

I can’t provide any other pointers than the official documentation.

You will have to wait for someone who actually has experience with this.