Remote Audio Transcription and Idle Client

[gfd2020]

Remote audio transcription with the IPED server/client is excellent. It helps a lot in this difficult task.
However, I noticed something that could perhaps improve performance a little.
For example: a UFDR report has photos, spreadsheets, chats, audio, etc. When the client identifies audio it will send it to the remote server. But sometimes it uses all threads to send to the server, so it remains idle and waiting for the result. Perhaps we could configure a maximum number of transcription threads if it have other types of files not yet processed (maybe half), so as not to leave the client inactive and doing other tasks.

[lfcnassif]

Hi @gfd2020, that's a good idea! Actually we do a similar thing with simultaneous containers expansion, at some point we limited it to half the number of CPU cores, other containers are re-enqueued. We could do something similar to audios.

But this should be Configurable. If your transcription cluster is slow, clients could be very idle and could do other things. If the cluster is very fast, our case, decreasing the number of simultaneous transcription requests can waste the cluster resources and may decrease the overall throughput. To know if there are other types of files different than audios in the queue is another challenge, we don't have this info today, not sure how to do this...

[gfd2020]

I'm still trying other things. I have a question. I think you have a server with some RTX 3090, right? I imagine it must be quick. How long does a transcription of a 30 seconds audio last ( in wav2vec2 ) ? 1 second on average?

[lfcnassif]

Yes, on RTX 3090 the large wav2vec2 model (1B parameters) does the job about 40-50x faster then the audio duration and we are able to run 2 simultaneous jobs per GPU. Today we have 8 servers with 2 GPUs each. That gives up to 1500x faster than audio real time when transcribing many small audios, today one large audio is not broken in separate jobs, which is fine for our current use case.

[lfcnassif]

So our clients are iddle for short periods of time. Of course, we can improve the feature for those without such a great GPU cluster infrastructure.

[gfd2020]

Hi @lfcnassif .
I've finished the heuristic (which can be turned on or off and configurable) and I'm going to open a new PR so that other developers can test and refine it, ok?

[lfcnassif]

Great! Thank you @gfd2020 for this optimization, it should help many users!