End-to-end evaluation demo #32
Conversation
|
|
jaidhyani
force-pushed
the
13-first-end2end-eval-demo
branch
2 times, most recently
from
addecee to
47baf30
Compare
jaidhyani
force-pushed
the
13-first-end2end-eval-demo
branch
2 times, most recently
from
9b28687 to
bfb67da
Compare
|
|
|
This ended up being an embarrassing quantity of iteration and written-then-deleted code for what was ultimately a pretty small PR. On the upside, I feel like I gained a better understanding of a lot of the codebase in the process. |
data/README.md
Outdated
There was a problem hiding this comment.
Yup, now that the static dir PR is in
data/model_group_stats.pkl
Outdated
There was a problem hiding this comment.
Should be removed and added to static
data/token_map.pkl
Outdated
There was a problem hiding this comment.
Should be removed and added to static
jaidhyani
force-pushed
the
13-first-end2end-eval-demo
branch
from
5a2c6af to
ff64510
Compare
|
|
||
| def load_logprob_datasets(split: str = "validation") -> dict[str, list[list[float]]]: | ||
| return { | ||
| model: cast(dict, load_logprob_dataset(model)[split])["logprobs"] |
There was a problem hiding this comment.
I'm worried how much we have to use casting with Dataset objects :/
There was a problem hiding this comment.
Yeah, the dataset library clearly wasn't designed with type-guarantees in mind. Building a library of functions that take care of the casting and other stuff we don't want to worry about in our day-to-day might be a good idea. Ticket would be something like "identify repetitive casts in the codebase and replace them with library functions".
| protobuf==4.25.2 | ||
| plotly==5.18.0 | ||
| spacy-transformers==1.3.4 |
There was a problem hiding this comment.
Sorry, meant to reply to this but the vscode inline thingy is kind of buggy and it ended up as a separate comment. Anyway:
spacy-transformers is a dependency I needed to get the token labeling code to work that I think just got missed by accident earlier (it's a hidden dependency that only shows up at runtime).
WIP