add tests to trajectory eval chain #8909
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Ignored Deployment
|
dosubot
bot
added
Ɑ: agent
| ) # Scan for first digit | ||
|
|
||
| if not 1 <= int(score_str) <= 5: | ||
| # Use regex to extract the score |
There was a problem hiding this comment.
nit: Would be useful to document the intended behavior to help verify implementation.
Is the reason for the change to change missing score into a None rather than a 0 ?
If so, could we keep the previous logic and replace like so:
score_str = next(
(char for char in score_str if char.isdigit()), None
) # Scan for first digit
There was a problem hiding this comment.
Thanks!
This change is intended to take care of cases where LLM makes invalid score like 1.1 or 20.
next((char for char in score_str if char.isdigit()), None) will return "1" for 1.1 and "2" for 20 which is invalid score.
My change will validate these case as error.
There was a problem hiding this comment.
Added explanation on the code comment.
|
Thanks for the change and the tests. Do you mind including somewhere an explanation of what behavior this PR modifies? |
What