Fix bug huggingface#35447 Tokenizer does not split text according to …

…newly added input tokens

The root reason is Trie.split method didn't ignore partial match that should be removed

src/transformers/tokenization_utils.py

@@ -175,7 +175,10 @@ def split(self, text: str) -> List[str]:
                     # matches
                     # "[CLS]", "L", we need to match CLS even if L is special
                     for lookstart, looktrie_pointer in states.items():
-                        if lookstart > start:
+                        if lookstart in to_remove:
+                            # This partial match should be removed
+                            continue
+                        elif lookstart > start:
                             # This partial match is later, we can stop looking
                             break
                         elif lookstart < start: