Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Evaluate Model Responses #19

Merged
merged 4 commits into from
Oct 8, 2024
Merged

Evaluate Model Responses #19

merged 4 commits into from
Oct 8, 2024

Conversation

bbengfort
Copy link
Contributor

@bbengfort bbengfort commented Oct 7, 2024

Scope of changes

Adds helper tasks to perform evaluations of model responses and collect aggregate numbers for an entire model. Right now these tasks must be run manually using a management command, but in the future they will be automated.

Type of change

  • bug fix
  • new feature
  • documentation
  • other (describe)

Acceptance criteria

This PR will likely be merged without review.

Author checklist

  • I have manually tested the change and/or added automation in the form of unit tests or integration tests
  • I have updated the dependencies list
  • I have added new test fixtures as needed to support added tests
  • I have added or updated the documentation

@bbengfort bbengfort merged commit f09bbd6 into main Oct 8, 2024
6 checks passed
@bbengfort bbengfort deleted the evaluation branch October 8, 2024 10:20
Copy link

sentry-io bot commented Oct 8, 2024

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

  • ‼️ AttributeError: 'NoneType' object has no attribute 'count' parley.management.commands.analyze in handle View Issue

Did you find this useful? React with a 👍 or 👎

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant