The problem of multilingual mixing in recording

[duwenlong2]

I have a scene like this. The audio file for meeting minutes needs to be converted into text, but after using Whisper, there are Chinese and English sentences in my audio file. He has uniformly output English.
I want to preserve the results of the original language. Can it be implemented in Whisper?

[sandrohanea]

The library is identifying one language at stat-up (if "auto" is used) and then it is used to transcribe the entire file so it makes sense it will uniformly output English (and transcriptions when other languages are spoken).

One idea to fix it (but it is not tested) would be to :

1. Add the WithProbabilities on the builder => which will give you the confidence level for each segment.
2. Once you identify some segment with low confidence level, re-transcribe it (by extracting the frames for that segment from start and end time). Either provide the other language if you know it is always "Chinese and English" or identify it again using "auto".
3. Replace the segments in the result.

It would be probably interesting to have this functionality in the library in the future, but cannot promise that I'll have time to implement it.