Finetuning settings and deepseed for training.

[Dolyfin]

Would love to know how to better optimize epoch/batch for longer datasets (2hrs) and if deepspeed helps training at all.

[erew123]

The DeepSpeed question, that's an easy one! DeepSpeed has no bearing on training, only TTS generation. As AllTalk's scripts are relying on other scripts to train (that other people make) I cannot activate DeepSpeed for training as their scripts don't support it.

As for training the models and how many epochs/how much is enough, have a look here. It's not a one size fits all solution as such, however, the settings I setup in finetuning are the default/recommended for these models.

I don't know of any specific data around batch size as such. This person was training an entirely new language into the model here with a quite a large dataset and for that he used 1000 epochs, but of course, that's an entirely new language rather than training an existing language with a new sounding voice.

The general guide is to train your model on X amount of epochs, then make some samples, compare their waveform+spectrograph, listen to them and see how they sound. If you aren't happy, train more. As for how many will end up being enough, as the FAQ says, its somewhat subjective without a definitive answer. Im not trying to be wishy-washy, its just there's no firm conclusion (or available data) on it.

[Dolyfin]

Thank you. Do you know if I can train 10 epoch then resume training after testing?

All my previous experience was in training RVC2 models and xttsv2 seems a little different.

I’m also unsure, as the dataset is of high recording quality, but consists of <1s to 20s voice lines. If I should remove shorter lines. Using whisper large 2 as it seems to give longer sentences instead of cutting off early.

[erew123]

(off the top of my head as Ive currently wrecked my python environments so cant check) to further train the model you would not go back to step 2 immediately, you would go to step 3, move it over to your fine tuned model, go back to step 2 and choose the finetuned model and continue on from there, re-using the same dataset (or create a new one with step 1 if you wish). It has to be moved first though and looped like that.

Ultimately I believe longer training data will yield better results than very short clips. Really youre after audio clips that will carry some expressive variance, which very very short clips (1 second) wont do. So if you really want to improve the quality in the shortest amount of epochs, Id say try to get plenty of 4 second and longer data.

[erew123]

Ok.. I have a working system now... So after you have trained your model and tested.. if you want to further finetune it, you would choose this option:

and then close and re-open the finetuning script. You should be able to go straight to step 2 (if you are going to re-use the same dataset) and on there you will have an option to use your finetuned model to train it:

I dont have a finetuned model at the moment, so I dont get the option to select it, but you get the idea.

[Dolyfin]

Thank you for the information.

Is there any plans to change how the current exporting fine tunes work? It’s very awkward to potentially train more than 2 models when you have to shuffle them manually between option a and b. Is there a reason why a model can’t be provided a folder name and have a selector to choose between 2+ custom fine tunes?

[erew123]

Coding time to be honest. Everything within AllTalk I've done within the last 44 days, but there's only myself coding. My main goal has been to build a solid base to grow upon. So features such as other model loaders etc are possible, its just been a case of time and juggling everything else. I can make a note of it and look at adding it in future.

[erew123]

@Dolyfin I'm closing this discussion, but I'm not closing it closing it. I'm actually putting up a central list of feature requests and I am linking back to all requests off that list. So your request (extra model save points and load points) will be noted in here #74. I'm also asking on the Feature Requests list that people join in the original thread discussions by following the links.

Thanks