Recovering an incomplete finetune session?

[Jidis83]

Hi,

I tried running the finetune procedure the other night, but (I believe) in the final stages, I ended up with an empty box on that last "what to do next" page for the "XTTS checkpoint path", and I couldn't proceed to the cleanup and move functions. It had run for a couple hours, and seemed able to use my trained data on the testing page, but somehow didn't get finalized correctly and it was getting too late to keep messing with it, so I figured I'd pickup where I left off later on. Unfortunately, it doesn't appear to reload or see any of the saved training data or names on relaunch, but the folders and data are still present.

Did I do something wrong that's unrecoverable, or is there some trick to pointing it back to the original folders and reloading it all? It's no big deal if I need to do it again, but I guess I'll need to look into why that box ended up unpopulated, so I don't just repeat the whole thing or get stuck back where I was.

Thanks!

[erew123]

Sounds like youre on an older build of Finetune. Not sure if youre on AllTalk V1 or V2... but here is a rough guide.

in the finetune/tmp-trn/.../XT{longanmehere}/ you will find there is a best_model.pth and also there should be a best_model_XX.pth (some number).

best_model.pth is THE best model that it has created so far.
best_model_XX.pth is the last epoch of the training run, which may or may not be better than the last (it compares the 2 and keeps one and throws the other).

So in theory, your best_model.pth can be used as a model file it just needs the other files from whatever matching model it was you started training as the base e.g. XTTS 2.0.2 or XTTS 2.0.3... all the 5/6 files that are from that version... bascially you want a folder like this:

the best_model.pth could simply be re-named model.pth and it would work... obviously from that point you can either use it OR further train it.

The only thing you will notice is that is 5GB in size......so needs compacting. (will work without that, but Im guessing you dont want a 5GB file hanging around).

So save this as compact.py

import torch
import os
MODEL_PATH = "best_model.pth"
# Check if best_model.pth exists
if not os.path.exists(MODEL_PATH):
    print(f"\nCannot find \033[94m{MODEL_PATH}\033[0m in the current folder.")
    
    # Check if model.pth exists as an alternative
    ALTERNATIVE_PATH = "model.pth"
    if os.path.exists(ALTERNATIVE_PATH):
        print(f"\nIf your model is currently called \033[93m{ALTERNATIVE_PATH}\033[0m please rename it to \033[94mbest_model.pth\033[0m.")
    else:
        print(f"Please put \033[91mbest_model.pth\033[0m in the current folder or rename your model file to \033[94mbest_model.pth\033[0m.")
    exit(1)
# Attempt to load the model
try:
    checkpoint = torch.load(MODEL_PATH, map_location=torch.device("cpu"))
except Exception as e:
    print("Error loading checkpoint:", e)
    raise
del checkpoint["optimizer"]
for key in list(checkpoint["model"].keys()):
    if "dvae" in key:
        del checkpoint["model"][key]
torch.save(checkpoint, "model.pth")
print(f"\nYour compacted model is created \033[93mmodel.pth\033[0m.")

put the file in the same folder as the best_model.pth file, start the alltalk python environment (or at least have access to python at your command prompt and python compact.py

That should give you a model.pth file.

Does that work for you?

[Jidis83]

in the finetune/tmp-trn/.../XT{longanmehere}/ you will find there is a best_model.pth and also there should be a best_model_XX.pth (some number).

FWIW, my tmp-trn folder actually has nothing but an empty "wavs" folder inside, but I believe the files you are referring to are in a "training" folder, inside a folder I named the model to (in the finetune folder).

[erew123]

They should be sub the finetune folder and they will be 5GB each. I cant recall the exact path off my head. but you should find them in there :)

[erew123]

@Jidis83 and that is of course assuming nothing is corrupted!

[Jidis83]

@erew123 - Much thanks! Got my work cut out for me tonight. I'll let you know what it does. Like I said, if I do need to restart everything, it's no biggie.

-and yes, this is all v.2 AllTalk stuff

[erew123]

@Jidis83 Good luck. Its not too hard :) Im going close off the ticket, im sure you will get back to me if needed, but what Ive given you is pretty much the entire thing! Id say make sure you update finetune (all of alltalk to the latest).

Thanks