Fail gracefully on GPU OutOfMemoryError

[clstaudt]

Running the example code in the Readme on a real audio file.

nvidia-smi shows the GPU memory (8GB) fill up.

Output exceeds the [size limit](command:workbench.action.openSettings?[). Open the full output data [in a text editor](command:workbench.action.openLargeOutput?42d9deaa-25c9-4785-9999-bb5b8e29fa72)
---------------------------------------------------------------------------
OutOfMemoryError                          Traceback (most recent call last)
Cell In[2], line 7
      4 audio_file = "audio/s4f000.mpga"
      6 # transcribe with original whisper
----> 7 model = whisperx.load_model("large", device)
      8 result = model.transcribe(audio_file)
     10 print(result["segments"]) # before alignment

File ~/miniforge3/envs/whisperx/lib/python3.10/site-packages/whisperx/__init__.py:115, in load_model(name, device, download_root, in_memory)
    112 model = Whisper(dims)
    113 model.load_state_dict(checkpoint["model_state_dict"])
--> 115 return model.to(device)

File ~/miniforge3/envs/whisperx/lib/python3.10/site-packages/torch/nn/modules/module.py:989, in Module.to(self, *args, **kwargs)
    985         return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None,
    986                     non_blocking, memory_format=convert_to_format)
    987     return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
--> 989 return self._apply(convert)

File ~/miniforge3/envs/whisperx/lib/python3.10/site-packages/torch/nn/modules/module.py:641, in Module._apply(self, fn)
    639 def _apply(self, fn):
    640     for module in self.children():
--> 641         module._apply(fn)
    643     def compute_should_use_set_data(tensor, tensor_applied):
...
    985     return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None,
    986                 non_blocking, memory_format=convert_to_format)
--> 987 return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)

OutOfMemoryError: CUDA out of memory. Tried to allocate 26.00 MiB (GPU 0; 7.79 GiB total capacity; 6.20 GiB already allocated; 29.31 MiB free; 6.52 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[clstaudt]

The issue seems to be that the 8GB of the GPU are not enough for the large model. It works with the medium model.

Suggestion: whisperx should fail more gracefully and suggest the next smaller model to the user.

[m-bain]

faster-whisper and other reductions should now help with gpu issues
#159 (comment)

[clstaudt]

@m-bain Agreed, but this was also a usability issue, since users may be confused whether they have chosen a model that cannot work with the given memory or something unexpected is going wrong.