Add function to tokenize text stories and split into batches

[siwei-li]

The test is right now two cases for text input and tokenized input, lmk if there's anything to add to it

[joshuawe]

• add docstrings
• update test to not download the validation dataset from huggingface

[joshuawe]

Please please a docstring 😄
text_stories is a list of stories? Each story is of arbitrary length and is not tokenized? E.g.: "There once was a dog called Barky. Barky lived .... "?
And in the end we split each of the stories into chunks that are of max size context_size? Is that the main task of the get_tokenized_batches function?

[joshuawe]

@siwei-li , a reply to my questions would have been much appreciated, instead of just resolving them. They might be simple or even stupid questions, but it would have helped me understand your code.

[jettjaniak]

there are no batches, so I think the function name should be changed? to something like "tokenize_dataset"

[siwei-li]

@joshuawe I added the docstrings to the functions,
the first two functions extend_deque() and make_new_samples() serve as two helper functions;

I noticed that the failing check is for the pytest in

tests/eval/test_token_labelling.py:2: in
import spacy
E ModuleNotFoundError: No module named 'spacy'

[joshuawe]

Hi @siwei-li , will check your updated code.
The error that we currently see (Link) is not because of the missing spacy module (I assume that your error message was from your local machine).
Following the link we see that beartype raised a type error in the extend_deque(). It seems the parameter text_stories expected the type list[str] but got something like list[int].

E   beartype.roar.BeartypeCallHintParamViolation: Function delphi.train.dataset_tokenization.extend_deque() parameter 
text_stories=[[1581, 1327, 61, 300, 3003, 1427, 1029, 2570, 1736, 3552, 3170, 635, 3551, 91, 3052, 2560,...]] violates type hint 
list[str] under non-default configuration BeartypeConf(claw_is_pep526=True, is_color=None, is_debug=False, 
is_pep484_tower=False, strategy=<BeartypeStrategy.O1: 2>, warning_cls_on_decorator_exception=<class 
'beartype.roar._roarwarn.BeartypeClawDecorWarning'>), as list index 11 item list [3638, 2316, 723, 505, 3953, 325, 742, 2799, 
3320, 4063, 159, 3931, 3300, 3072, 3636, 2544, ...] not instance of str.```

[jettjaniak]

We need a script in scripts/ that will take input dataset name, output dataset name, tokenizer name and HF credentials and tokenize and push to HF. The column name in the output dataset should be "tokens". The column name in the input dataset should be an optional argument, that defaults to the only column that is available or fails if there are more than one column. Please check (on a subset of the dataset) how slow the tokenization is. If it takes more than a few minutes, we might want to use tokenizer.batch_encode

[siwei-li]

The script for HF is added @jettjaniak
(Uploaded the tokenized 'stories' dataset to https://huggingface.co/datasets/delphi-suite/batched-tokenized-stories)

[joshuawe]

can be merged, if no rebase is necessary

[joshuawe]

Why is this done one hundred times?

[joshuawe]

I find the explanation a bit confusing (or I am confused).
This function does not generate entirely new content, correct?
It reduces the length pre-existing samples by clipping it to context_size, correct?
A different wording would make it easier to understand. You could also consider renaming the function make_new_samples to something such as clip_samples or similar. But this is up to you.

[joshuawe]

I see, my previous assumption was wrong. This function does not clip the samples, but rather splits them, prepending a BOS token as well as adding the final token from the previous split as well.