“Nvidia-smi not found in $PATH” error

[darribas]

I’m running into the following issue:

dani@ymir:~$ docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "nvidia-smi": executable file not found in $PATH: unknown.
[0.498s] 127  dani@ymir:~$

I’m aware of #148 but, going by the README of this repo, I think that should have been implemented and available? I’m running this on Ubuntu server.

Here’re my specs

• OS version

 lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 24.04.1 LTS
Release: 24.04
Codename: noble

• Kernel

Linux ymir 6.8.0-45-generic #45-Ubuntu SMP PREEMPT_DYNAMIC Fri Aug 30 12:02:04 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

• NVIDIA driver/CUDA on the host

Thu Sep 26 17:05:57 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.120                Driver Version: 550.120        CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
[…]

• Snap version

snap version 
snap    2.63.1+24.04
snapd   2.63.1+24.04
series  16
ubuntu  24.04
kernel  6.8.0-45-generic

• Docker info

snap info docker
name:      docker
summary:   Docker container runtime
publisher: Canonical✓
store-url: https://snapcraft.io/docker
contact:   https://github.com/docker-snap/docker-snap/issues?q=
license:   (Apache-2.0 AND MIT AND GPL-2.0)
description: |
  Build and run container images with Docker.
  
  **Usage**
  
  * This build can only access files in the home directory. So Dockerfiles and all other files used
  in commands like `docker build`, `docker save` and `docker load` need to be in $HOME.
  * You can change the configuration of this build by modifying the files in
  `/var/snap/docker/current/`.
  * Additional certificates used by the Docker daemon to authenticate with registries need to be
  added in `/var/snap/docker/current/etc/docker/certs.d` (instead of `/etc/docker/certs.d`). This
  directory can be accessed by other snaps using the `docker-registry-certificates` content
  interface.
  
  **Running Docker as normal user**
  
  By default, Docker is only accessible with root privileges (`sudo`). If you want to use docker as
  a regular user, you need to add your user to the `docker` group.
  
      sudo addgroup --system docker
      sudo adduser $USER docker
      newgrp docker
      sudo snap disable docker
      sudo snap enable docker
  
  **Warning:** if you add your user to the `docker` group, it will have similar power as the `root`
  user. For details on how this impacts security in your system, see
  https://docs.docker.com/engine/security/#docker-daemon-attack-surface
  
  **Authors**
  
  This snap is built by Canonical based on source code published by Docker, Inc. It is not endorsed
  or published by Docker, Inc.
  
  Docker and the Docker logo are trademarks or registered trademarks of Docker, Inc. in the United
  States and/or other countries. Docker, Inc. and other parties may also have trademark rights in
  other terms used herein.
commands:
  - docker.compose
  - docker
  - docker.help
services:
  docker.dockerd:                  simple, enabled, active
  docker.nvidia-container-toolkit: oneshot, enabled, inactive
snap-id:      sLCsFAO8PKM5Z0fAKNszUOX0YASjQfeZ
tracking:     latest/stable
refresh-date: 22 days ago, at 04:47 UTC
channels:
  latest/stable:    24.0.5   2024-09-03 (2932) 138MB -
  latest/candidate: 24.0.5   2024-09-03 (2932) 138MB -
  latest/beta:      27.2.0   2024-09-19 (2963) 146MB -
  latest/edge:      27.2.0   2024-09-20 (2969) 146MB -
  core18/stable:    20.10.17 2023-03-13 (2746) 146MB -
  core18/candidate: ↑                                
  core18/beta:      ↑                                
  core18/edge:      ↑                                
installed:          24.0.5              (2932) 138MB -

Do you have any sense what might be going on? Thanks!

[farshidtz]

The binary discovery path is explained in this section of the README. The nvidia-smi is not in PATH.

Please try one of the following:

• Add --env PATH="${PATH}:/var/lib/snapd/hostfs/usr/bin" to the docker run flags.
• Execute the binary with the full path: /var/lib/snapd/hostfs/usr/bin/nvidia-smi

[YamiYukiSenpai]

$ docker run --env PATH="${PATH}:/var/lib/snapd/hostfs/usr/bin" --rm --runtime=nvidia ubuntu nvidia-smi
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "nvidia-smi": executable file not found in $PATH: unknown.

$ docker run --env PATH="${PATH}:/var/lib/snapd/hostfs/usr/bin" --rm --runtime=nvidia ubuntu /var/lib/snapd/hostfs/usr/bin/nvidia-smi
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/var/lib/snapd/hostfs/usr/bin/nvidia-smi": stat /var/lib/snapd/hostfs/usr/bin/nvidia-smi: no such file or directory: unknown.

This is my nvidia-smi output on native Ubuntu 24.04:

$ nvidia-smi
Wed Oct 23 05:40:23 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.90.07              Driver Version: 550.90.07      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA T400 4GB                Off |   00000000:01:00.0 Off |                  N/A |
| 66%   52C    P0             N/A /   31W |       1MiB /   4096MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

$ snap --version
snap    2.66
snapd   2.66
series  16
ubuntu  24.04
kernel  6.8.0-47-generic

$ snap info docker
name:      docker
summary:   Docker container runtime
publisher: Canonical✓
store-url: https://snapcraft.io/docker
contact:   https://github.com/docker-snap/docker-snap/issues?q=
license:   (Apache-2.0 AND MIT AND GPL-2.0)
...
snap-id:      sLCsFAO8PKM5Z0fAKNszUOX0YASjQfeZ
tracking:     latest/edge
refresh-date: 31 days ago, at 10:55 UTC
channels:
  latest/stable:    24.0.5   2024-09-03 (2932) 138MB -
  latest/candidate: 24.0.5   2024-09-03 (2932) 138MB -
  latest/beta:      27.2.0   2024-09-19 (2963) 146MB -
  latest/edge:      27.2.0   2024-09-20 (2969) 146MB -
  core18/stable:    20.10.17 2023-03-13 (2746) 146MB -
  core18/candidate: ↑                                
  core18/beta:      ↑                                
  core18/edge:      ↑                                
installed:          27.2.0              (2969) 146MB -