Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Using Mistral 7B with transformers v4.38.1 on MATH dataset, and facing memory leaks #80

Open
Jayant1234 opened this issue May 4, 2024 · 0 comments
Open

Using Mistral 7B with transformers v4.38.1 on MATH dataset, and facing memory leaks #80

Jayant1234 opened this issue May 4, 2024 · 0 comments

Comments

@Jayant1234
Copy link

Jayant1234 commented May 4, 2024 •
edited
Loading

In both the Trainers, Basic, and FSDP, there is an underlying pattern of GPU memory not being freed. Allocation keeps increasing in steps while utilization remains roughly constant.
image

Does anyone have any suggestions of what might have gone wrong?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Jayant1234