Whisper Fine-tuning Recipe on Aishell1

[yuekaizhang]

This PR supports fine-tuning whisper models using aishell1.

• Fix code comments & lint errors

	test (before fine-tuning)	test (after fine-tuning)	comment
medium	7.23	3.27	--epoch 10 --avg 4, ddp
large-v2	6.56	2.47	--epoch 10 --avg 6, deepspeed zero stage1
large-v3	6.06	2.84	--epoch 5 --avg 3, deepspeed zero stage1

[yuekaizhang]

Nice work!

Would you mind sharing how much GPU memory and how long it takes to fine-tune the Whisper model?

On a 8xA100(80GB) machine, it takes about 15 mins per epoch for large-v2/3 models. Since I am always set the max_duration as large as possible, I think people could fine-tune it on 24GB cards with deepspeed and smaller batch size.

[JinZr]

LGTM! Thank you!

[csukuangfj]

    model = whisper.load_model(filename)
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/whisper/__init__.py", line 147, in load_model
    dims = ModelDimensions(**checkpoint["dims"])
KeyError: 'dims'

It throws the above error while using whisper.load_model() with the checkpoint

Could you make your checkpoint compatible with whisper? For instance, you need to save dims in
the resulting checkpoint.

@yuekaizhang

[yuekaizhang]

    model = whisper.load_model(filename)
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/whisper/__init__.py", line 147, in load_model
    dims = ModelDimensions(**checkpoint["dims"])
KeyError: 'dims'

It throws the above error while using whisper.load_model() with the checkpoint

Could you make your checkpoint compatible with whisper? For instance, you need to save dims in the resulting checkpoint.

@yuekaizhang

Sorry, I added this converion script https://huggingface.co/yuekai/icefall_asr_aishell_whisper/blob/main/convert.sh. I have uploaded a converted medium model. Would you mind trying it again? https://huggingface.co/yuekai/icefall_asr_aishell_whisper/blob/main/exp_medium/whisper-medium-aishell1-epoch-10-avg-4.pt

[csukuangfj]

@yuekaizhang

Thanks! It works perfectly. I have converted it to sherpa-onnx with k2-fsa/sherpa-onnx#565

You can find the converted model at
https://huggingface.co/csukuangfj/sherpa-onnx-whisper-medium-aishell/tree/main

You can also try it in the following huggingface space
https://huggingface.co/spaces/k2-fsa/automatic-speech-recognition-with-whisper