-
Notifications
You must be signed in to change notification settings - Fork 234
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
3 changed files
with
3 additions
and
3 deletions.
There are no files selected for viewing
Submodule legion
updated
from 24e8c4 to 0d32b3
Submodule tokenizers-cpp
updated
19 files
+3 −0 | .gitmodules | |
+28 −4 | CMakeLists.txt | |
+0 −1 | example/CMakeLists.txt | |
+5 −1 | example/build_and_run.sh | |
+69 −18 | example/example.cc | |
+17 −2 | include/tokenizers_c.h | |
+38 −0 | include/tokenizers_cpp.h | |
+1 −0 | msgpack | |
+1 −1 | rust/Cargo.toml | |
+100 −16 | rust/src/lib.rs | |
+64 −10 | src/huggingface_tokenizer.cc | |
+143 −0 | src/rwkv_world_tokenizer.cc | |
+33 −0 | src/rwkv_world_tokenizer.h | |
+20 −1 | src/sentencepiece_tokenizer.cc | |
+1 −1 | web/build.sh | |
+2 −2 | web/package.json | |
+31 −10 | web/src/tokenizers.ts | |
+3 −1 | web/src/tokenizers_binding.cc | |
+22 −1 | web/tests/src/index.ts |