You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi,
Is there any way to disable all or some of normalization rules in parsbert tokenizer?
For example do not convert "آ" to "ا" or "ئ" to "ی".
Also the tokenizer removes all half-spaces and concatenate the words.
Setting the do_lower_case and srip_accents parameters to false does not work.
I would be so grateful if you let me know whether there is any solution to my problem.
The text was updated successfully, but these errors were encountered:
Hi,
Is there any way to disable all or some of normalization rules in parsbert tokenizer?
For example do not convert "آ" to "ا" or "ئ" to "ی".
Also the tokenizer removes all half-spaces and concatenate the words.
Setting the do_lower_case and srip_accents parameters to false does not work.
I would be so grateful if you let me know whether there is any solution to my problem.
The text was updated successfully, but these errors were encountered: