first end2end eval demo

[jettjaniak]

• just loss over groups of tokens on a few models
• interactive plotly line plot (you can choose tokens group with a drop down)
• models jbrinkma/delphi-llama2-{100,200,400}k
• dataset: validation split of jbrinkma/tinystories-v2-clean-tokenized

• depends on: categorize tokens #12 inference script #10 map tokens to prompts and positions #8 load validation dataset only #4 interactive performance plot #5
• notebook using some library functions is fine
  • prioritize making a functional plot above clean code, tests etc
  • when it works we can make it pretty