Error when no tokenizer.json file is present in BERT models

[lordofthejars]

I am using this model, https://huggingface.co/unitary/toxic-bert, which doesn't have the tokenizer.json. Some projects use this model, and they can infer it, so I am wondering if they are providing the tokenizer.json file or if, for example, there is a way to avoid having the file.

In the case of JLama, an exception is thrown Caused by: java.lang.IllegalArgumentException: No tokenizer.json found in

[lordofthejars]

I created the toknizer file and it works: https://huggingface.co/lordofthejars/toxic-bert

In case you want to add it as supported models

[tjake]

Cool, how did you create it?

[lordofthejars]

There is a tool called Autotkenizer

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("unitary/toxic-bert")
assert tokenizer.is_fast
tokenizer.save_pretrained("...")