Best way to store .pth files in GitHub

[henry-AY]

Select Topic Area

Question

Body

Hey,

I am working on an LLM and recently implemented checkpointing in my models. Because of this, I have files that are too large to push (at the moment small but 100-400 MB). I am aware of LFS, however, what other solutions exist? If any are considered better? Especially given that I am in the training phase and will constantly update the .pth files.

Many Thanks

[Suhaib3100]

Thats Great Mate

[AnuraagR9251]

How

[Topromm]

Cloud Storage + Links: You could store your .pth files on a cloud storage service like Google Drive, AWS S3, or Azure, and then add links to them in your repo.

DVC (Data Version Control): If you're looking for something more integrated, DVC is a great option. It’s designed for handling bigger datasets and model files while keeping version control in sync with Git. It can store your checkpoints remotely (cloud, S3, etc.) and manage them through Git without committing the actual files.

Since you're in the training phase, DVC might be ideal since it tracks different versions of your files and works well with Git. Either that or stick to LFS.

[henry-AY]

Thanks! I think I'm going to be going with DVC especially given it works well with Git. I also like some other options, maybe for projects that won't be scaled to the same extent.

Thanks again :)

[prblsing]

1. Cloud Storage (AWS S3, GCP, Azure) - Recommended

   • Why: Best for frequently updated checkpoints; offers scalable, unrestricted storage with robust API support.
   • How: Upload files to cloud storage, automate sync with a script, and download as needed during training.
   • Pros: Unlimited storage, easy scalability, suitable for dynamic workflows.
   • Cons: Requires cloud account setup, credential management, and custom script integration.

2. DVC (Data Version Control)

   • Why: Designed specifically for data science and ML workflows, providing advanced version control for large files and seamless integration with cloud storage.
   • How: Install DVC, initialize (dvc init), track files (dvc add model.pth), and push to cloud storage using dvc remote.
   • Pros: Tracks file changes, integrates well with ML pipelines, supports multiple remote storage options.
   • Cons: Learning curve is steeper, requires additional configuration and tools.

[henry-AY]

Thank you for the detailed response! As I mentioned in another post here, I will be going with DVC because it works well with git. Thanks for the quick tutorial as well :)

I appreciate it a lot