add token labelling

[joshuawe]

No description provided.

[joshuawe]

Are the files that have been added stored in a good location?
I think not.

[menamerai]

@joshuawe I think you need to add spacy to the requirements.txt

[jettjaniak]

Let's make a pickle with dict[int, dict[str, bool]] that labels every single token in the tokenizer.

[jettjaniak]

I would like to have function that just takes a str token label. Should we return this as a dict?

[joshuawe]

Have a look at the notebook, where function calling as well as the creation of the pickle object which is basically Look Up Table containing a dict with keys token_id and value token_labels.

Note that token_labels is itself a dict with all keys label_name and values bool

For example:

token_ids_lablled = {
  0: {'Starts with space': False, 'Capitalized': True, 'Is Noun': True, 'Is Pronoun': False, 'Is Adjective': False, 'Is Verb': False, 'Is Adverb': False, 'Is Preposition': False, 'Is Conjunction': False, 'Is Interjunction': False, 'Is Named Entity': False},
  1: {'Starts with space': False, 'Capitalized': True, 'Is Noun': True, 'Is Pronoun': False, 'Is Adjective': False, 'Is Verb': False, 'Is Adverb': False, 'Is Preposition': False, 'Is Conjunction': False, 'Is Interjunction': False, 'Is Named Entity': False},
  2: ...
...
}

The question that remains is

1. where to store the pickle.
2. How to make it accessible. Should it always be loaded into the memory, when the module is imported?

[joshuawe]

Another interesting thing is the following,
the vocab size of the tokenizer is larger than 4000 by far.

>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("roneneldan/TinyStories-1M")

>>> vocab_size = tokenizer.vocab_size
>>> print("The vocab size is:", vocab_size)
50247

So, am I using the correct tokenizer? And are there multiple tokenizers?

[jettjaniak]

The tokenizer is wrong, it's from the original tiny stories models. You need to pass model name from any of the models on delphi-suite Hugging Face to use our small tokenizer that was trained on tiny stories dataset.

[jettjaniak]

why?

[joshuawe]

I still require this so I can import everything under delphi. For WHATEVER reason an empty __init__.py file will not allow me to import it. (of course, I used pip install -e .)

[jettjaniak]

My preferred docstring template is

"""Short one-line description

Optional longer description
"""

If you want to list and describe all arguments that's fine, but specifying their type and optionality is redundant, you can see all of that in function definition.

[jettjaniak]

you need to rebase on top of main

[jettjaniak]

The notebook is good as an explanation, but generating the labels shouldn't require running it. Let's add something in scripts/ for that.

[jettjaniak]

Is there any reason we don't have a function like label_token(token: str) -> dict[str, bool]? Combining random tokens into sentences seems opaque and bug prone.

[jettjaniak]

for now just store the pickle in eval directory

[jettjaniak]

We should debug this. What is your exact setup? System, python interpreter, how you make your venv etc. Also, can you `import sys; print(sys.path)`

…

On Fri, 9 Feb 2024, 08:39 Joshua We, ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In src/delphi/__init__.py <#21 (comment)>: > +from . import eval + I still require this so my I can import everything. For WHATEVER reason an empty *init*.py file will not allow me to import it. (of course, I used pip install -e .) — Reply to this email directly, view it on GitHub <#21 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AECKP3YGYAEMHVD3A4UROPDYSZGLNAVCNFSM6AAAAABCW6SGQOVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTQNZSHAYDAMRVGU> . You are receiving this because you commented.Message ID: ***@***.***>

[joshuawe]

We should debug this. What is your exact setup? System, python interpreter, how you make your venv etc. Also, can you import sys; print(sys.path)

My setup:

• Windows
• conda for venv creation
• pip install -e .
• Python interpreter by running where Python, the first entry is :
  C:/Users/.../anaconda3/envs/delphi2/python.exe

Here the other outputs

>>> print(dir(delphi))
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'beartype_this_package', 'eval']

>>> import sys; print(sys.path)
['c:\\"path to \\delphi\\notebooks', 'c:\\Users\\...\\anaconda3\\envs\\delphi2\\python310.zip', 'c:\\Users\\...\\anaconda3\\envs\\delphi2\\DLLs', 'c:\\Users\\...\\anaconda3\\envs\\delphi2\\lib', 'c:\\Users\\...\\anaconda3\\envs\\delphi2', '', 'c:\\Users\\...\\anaconda3\\envs\\delphi2\\lib\\site-packages', 'c:\\users\\...\\joshua\\research\\2024_01_tinyevals\\delphi\\src', 'c:\\Users\\...\\anaconda3\\envs\\delphi2\\lib\\site-packages\\win32', 'c:\\Users\\...\\anaconda3\\envs\\delphi2\\lib\\site-packages\\win32\\lib', 'c:\\Users\\...\\anaconda3\\envs\\delphi2\\lib\\site-packages\\Pythonwin']

[jettjaniak]

The only thing that comes to mind is conda being weird. You could just download and install python for windows and make venv this way

[joshuawe]

• added tests
• added a script label_all_tokens.py that does some token labelling
• now two files are saved under src/delphi/eval/. using the tokenizer from a given huggingface language model repo
  1. the tokens and their ids are stored in all_tokens_{huggingface_model_name}.txt
  2. The labelled tokens stored as labelled_token_ids_dict_{huggingface_model_name}.pkl
• had to update requirements in order to extract/run the delphi tokenizer

Small todo's left for tomorrow:

• Clear up the __init__.py files again
• Implement the function that takes in a token as a str and checks for its labels in the token_label_dict
  @jettjaniak Should the tokenized dict be loaded in memory always, i.e. when the module is loaded it also loads the pickle?

[jettjaniak]

the tokens and their ids are stored in all_tokens_{huggingface_model_name}.txt
The labelled tokens stored as labelled_token_ids_dict_{huggingface_model_name}.pkl

Can we drop the _{huggingface_model_name}? All delphi models use the same tokenizer.

Implement the function that takes in a token as a str and checks for its labels in the token_label_dict
@jettjaniak Should the tokenized dict be loaded in memory always, i.e. when the module is loaded it also loads the pickle?

I assume this question only applies if you're implementing this function? I'd say don't implement it, at least for now. The pickle is enough for the demo. And in this case, you should never be loading the pickle in code included in this PR.

[jettjaniak]

branch is out-of-date with main, you need to git rebase main

[jettjaniak]

were you using git merge? something is wrong with "Files changed" tab in this PR and with the commit history