Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Structured benchmark #2

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

willkurt
Copy link

This is a revised version of the old PR that fixes some small issues and adds support + data for all the Mistral evals we did as well.
In this PR we should have:

- README documenting all the steps to reproduce.
- Necessary model handlers and additional code necessary to run the models.
- Modal files for deployment for both structured and unstructured generation.
- Data from all the runs so that re-running is not necessary.
- A few Python notebooks so that visualization and error analysis is easier to perform.

@rlouf
Copy link
Member

rlouf commented Jun 18, 2024

Is this good for review?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants