Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Only skip benchmarking if split results are the same too #1260

Open
Muennighoff opened this issue Sep 29, 2024 · 1 comment · May be fixed by #1525
Open

Only skip benchmarking if split results are the same too #1260

Muennighoff opened this issue Sep 29, 2024 · 1 comment · May be fixed by #1525

Comments

@Muennighoff
Copy link
Contributor

Muennighoff commented Sep 29, 2024

We currently skip evaluation if the results file already exists, however, it may be that the file exists but not with the same splits/subsets as requested by the current evaluation. I think we may want to also check equivalence across splits & subsets and then run the ones still missing and edit the existing results file to put them back in.

@thivyanth
Copy link

#1268 I tried my best but I need to make myself more familiar with the codebase and try again.

@isaac-chung isaac-chung linked a pull request Nov 28, 2024 that will close this issue
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants