"CUDA_ERROR_NO_DEVICE" despite installed GPU

[siebentausend]

Hi everyone!

I'm running a Ubuntu 18.04.6 VM on Proxmox and I'm passing a NVIDIA K4000 GPU and an Osprey 845e SDI-Interface to the VM (The Osprey card is the actual reason for running the old Ubuntu Version since the driver support is quite limited).
Running the cuda-build of the Restreamer-Container, I get to pick the "H.264 (NVENC)"-Option, but when I try to connect the stream, I get the error "[h264_nvenc @ 0x55a7014df380] dl_fn->cuda_dl->cuInit(0) failed -> CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected". With libx264 everything works perfectly fine, only the CPU usage is higher, of course. Running nvidia-smi from the container I got the result below. Therefor I assume, that the GPU is passed through correctly?!
I followed several instructions and ended up with the docker-compose file below. As you can see, I tried "runtime: nvidia" and the "deploy" setting with the same result. At least with this config the GPU is recognized, but NVENC still isn't working. Do you have any idea what I'm missing here? I'm at the end of the rope here...

Thank you for your help!

$ sudo docker exec restreamer_core nvidia-smi
Wed Apr 24 14:44:26 2024
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.239.06 Driver Version: 470.239.06 CUDA Version: 11.7 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Quadro K4000 Off | 00000000:01:00.0 Off | N/A |
| 30% 37C P0 29W / 87W | 0MiB / 3018MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+

docker-compose.yaml:

version: "3"
services:
  restreamer:
    image: datarhei/restreamer:cuda-latest
    container_name: restreamer_core
    restart: unless-stopped
    #runtime: nvidia
    deploy:
      resources:
        reservations:
          devices:
             - driver: nvidia
               count: all
               capabilities: [gpu]  
    ports:
      - "8080:8080"
      - "8181:8181"
      - "1935:1935"
      - "4020-4040:4020-4040"
      - "4020-4040:4020-4040/udp"
      - "1020-1050:1020-1050/udp"
      - "5000-5015:5000-5015/udp"
    networks:
      - restreamer-net
    environment:> 
      - CORE_METRICS_ENABLE=true
      - CORE_METRICS_ENABLE_PROMETHEUS=true
      - CORE_METRICS_RANGE_SECONDS=300
      - CORE_METRICS_INTERVAL_SECONDS=2
    volumes:
      - restreamer-data-nfs:/core/data
      - restreamer-config-nfs:/core/config
    devices:
      - "/dev/video0:/dev/video0"
      - "/dev/video1:/dev/video1"
      - "/dev/video2:/dev/video2"
      - "/dev/video3:/dev/video3"
    privileged: true
volumes:
  restreamer-data-nfs:
    driver: local
    driver_opts:
      type: "nfs"
      o: addr=10.4.255.60,nolock,soft,rw,nfsvers=3
      device: ":/export/docker/restreamer/data"
  restreamer-config-nfs:
    driver: local
    driver_opts:
      type: "nfs"
      o: addr=10.4.255.60,nolock,soft,rw,nfsvers=3
      device: ":/export/docker/restreamer/config"

networks:
  restreamer-net:

I'm running following software versions:

• Ubuntu 18.04.6 (4.15.0-213-generic)
• Docker version 24.0.2, build cb74dfc
• NVIDIA Driver (nvidia-smi) 470.239.06
• CUDA Version 11.4
• datarhei-core v16.15.0 (linux/amd64) - long-dream-5781
• nvidia-container-runtime
  cli-version: 1.13.5
  lib-version: 1.13.5
  build date: 2023-07-18T11:38+00:00
  build revision: 66607bd046341f7aad7de80a9f022f122d1f2fce
  build compiler: x86_64-linux-gnu-gcc-7 7.5.0
  build platform: x86_64
  build flags: -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -DNDEBUG -std=gnu11 -O2 -g -fdata-sections -ffunction-sections -fplan9-extensions -fstack-protector -fno-strict-aliasing -fvisibility=hidden -Wall -Wextra -Wcast-align -Wpointer-arith -Wmissing-prototypes -Wnonnull -Wwrite-strings -Wlogical-op -Wformat=2 -Wmissing-format-attribute -Winit-self -Wshadow -Wstrict-prototypes -Wunreachable-code -Wconversion -Wsign-conversion -Wno-unknown-warning-option -Wno-format-extra-args -Wno-gnu-alignof-expression -Wl,-zrelro -Wl,-znow -Wl,-zdefs -Wl,--gc-sections

Resulting log for the stream:

@1713969154 ffmpeg version 6.1.1-datahrei Copyright (c) 2000-2023 the FFmpeg developers
@1713969154 built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04)
@1713969154 configuration: --extra-version=datahrei --prefix=/usr/local --extra-cflags=-I/usr/local/cuda/include --extra-ldflags=-L/usr/local/cuda/lib64 --enable-nonfree --enable-gpl --enable-version3 --enable-postproc --enable-static --enable-openssl --enable-libxml2 --enable-cuda --enable-cuvid --enable-nvenc --enable-libnpp --enable-libv4l2 --enable-libfreetype --enable-alsa --enable-libsrt --enable-libx264 --enable-libx265 --enable-libvpx --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libdav1d --enable-librav1e --disable-ffplay --disable-ffprobe --disable-debug --disable-doc
@1713969154 libavutil 58. 29.100 / 58. 29.100
@1713969154 libavcodec 60. 31.102 / 60. 31.102
@1713969154 libavformat 60. 16.100 / 60. 16.100
@1713969154 libavdevice 60. 3.100 / 60. 3.100
@1713969154 libavfilter 9. 12.100 / 9. 12.100
@1713969154 libswscale 7. 5.100 / 7. 5.100
@1713969154 libswresample 4. 12.100 / 4. 12.100
@1713969154 libpostproc 57. 3.100 / 57. 3.100
@1713969154 Input #0, video4linux2,v4l2, from '/dev/video1':
@1713969154 Duration: N/A, start: 1037.560333, bitrate: 622080 kb/s
@1713969154 Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 1920x1080, 622080 kb/s, 25 fps, 25 tbr, 1000k tbn
@1713969154 [aist#1:0/pcm_s16le @ 0x55a7014db6c0] Guessed Channel Layout: stereo
@1713969154 Input #1, alsa, from 'hw:2,1':
@1713969154 Duration: N/A, start: 1713969154.471428, bitrate: 1536 kb/s
@1713969154 Stream #1:0: Audio: pcm_s16le, 48000 Hz, 2 channels, s16, 1536 kb/s
@1713969154 Stream mapping:
@1713969154 Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (h264_nvenc))
@1713969154 Stream #1:0 -> #0:1 (pcm_s16le (native) -> aac (native))
@1713969154 Press [q] to stop, [?] for help
@1713969154 [h264_nvenc @ 0x55a7014df380] dl_fn->cuda_dl->cuInit(0) failed -> CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
@1713969154 [vost#0:0/h264_nvenc @ 0x55a7014df040] Error while opening encoder - maybe incorrect parameters such as bit_rate, rate, width or height.
@1713969154 Error while filtering: Unknown error occurred
@1713969154 [out#0/hls @ 0x55a7014dae40] Nothing was written into output file, because at least one of its streams received no packets.
@1713969154 Conversion failed!