get new non-spacy token labels

[transcendingvictor]

My plan is to use the token_labelling.py module, redo the basic categories.
Also, be able to make build categories by interacting manually with on Sheets and exporting the .csv file.

[jettjaniak]

*non-spacy Could you be more specific? What are the TODOs, what's your next step? What do you mean by using the module? Keep in mind there is a PR open for spacy token labeling, but should be merged soon

…

On Tue, 5 Mar 2024, 11:41 transcendingvictor, ***@***.***> wrote: My plan is to use the token_labelling.py module, redo the basic categories. Also, be able to make build categories by interacting manually with on Sheets and exporting the .csv file. — Reply to this email directly, view it on GitHub <#53>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AECKP37T6VOIQEQ6QF4R4LDYWYNUZAVCNFSM6AAAAABEHX5IPCVHI2DSMVQWIX3LMV43ASLTON2WKOZSGE3TAMBRGMZDCMY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[jettjaniak]

ACTUALLY - instead of "is in the dataset" column, let's have "how many times appeared in the dataset column". Then we can sort by this and only focus on first 500-1000 tokens for manual labeling

[transcendingvictor]

TO-DOs.

1. Implement the new token categories:

• is a word
• instances on dataset

2. Implement a way to retrieve the labels from a .csv file so that we can manually edit categories.

[jettjaniak]

also

• starts with space
• capitalized

[jettjaniak]

but first we need to merge #40 - can you work on that?