add new fairseq_pretraining function for starting from checkpoint

[AndreasPlt]

For doing scheduled hard_negatives training, I added run_fairseq_pretraining_from_checkpoint function that takes an additional checkpoint parameter and runs the pretraining job from that checkpoint

[AndreasPlt]

I noticed that this, when using the checkpoint parameter, it is not possible to resume training from the last checkpoint when interrupted.

The reason for that is that, when checkpoint is given, fairseq always starts training from that checkpoint and not from checkpoint_last.pt. On the other hand, when removing the checkpoint parameter in order to make fairseq train from the last checkpoint, the hash changes and checkpoint_last.pt is in fact not available anymore (because it was saved in the previous job with the other hash). Any ideas, how to solve that elegantly?

[AndreasPlt]

To tackle the problem described above, I proposed to change the behavior in the FairseqHydraTrainingJob in the i6_core repo (see this PR).

[vieting]

The function run_fairseq_pretraining_from_checkpoint that you introduce has huge overlap with the existing run_fairseq_pretraining. Instead of copying the whole function and only changing a small part of it, please add checkpoint: Optional[tk.Path] = None as an argument to the existing function and do the modifications you need if it is given, e.g.

if checkpoint is not None:
    fairseq_args["checkpoint"]["continue_once"] = checkpoint

[AndreasPlt]

Done. Instead of tk.Path however, I added str as a typehint since I am unsure if fairseq accepts a tk.Path. Feel free to correct me if I'm wrong

[vieting]

fairseq will never see a tk.Path object because it is written to the config like a normal string and fairseq only interacts with the config file, not with what you have in your sisyphus graph, right?