dataset tokenization script

[jettjaniak]

• similar to training script for mamba & llama #3 I imagine this will import some utils from llama2.c and build around them
• the short script itself should be in scripts/
• waiting for @woog97 to confirm what should we do with BOS/EOS
• I would like it to pull our v2 text dataset from HF, instead of building it on the fly
  • reduces chance of errors
• either save to file or upload to hf (depending on cli args)

[woog97]

Checked with eleuther. We should train with BOS to start every sample. In practice this would look like:

<BOS>tinystory1<EOS>tinystory2<EOS>tiny
<BOS>story3<EOS>tinystory4<EOS>tinystor
<BOS>y5<EOS>tinystory6<EOS>...```

[jettjaniak]

duplicate, taken care of in #31