New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reduce peak memory when using FSDP on 2+ GPUs #2

Open

gordicaleksa opened this issue Sep 9, 2023 · 0 comments

Labels

bug help wanted question

Owner

gordicaleksa commented Sep 9, 2023

Figure out the peak memory issue with FSDP when running the 615 M parameter on 2 GPUs I linked here:

facebookresearch/fairseq#5318

gordicaleksa added the question label

gordicaleksa added this to Open-NLLB: features & tasks

gordicaleksa moved this to Todo in Open-NLLB: features & tasks

gordicaleksa changed the title ~~Reduce peak memory when using FSDP on 2 GPUs~~ Reduce peak memory when using FSDP on 2+ GPUs

gordicaleksa added bug help wanted labels

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment