feat: Add Rust implementation of Tokenizer

This commit adds the Rust implementation of the Tokenizer class. The Tokenizer struct is defined in the `node-napi/src/lib.rs` file. It includes methods for encoding and decoding text using the tokenizer. The implementation is based on the `RustTokenizer` struct from the `tokenizers` crate. This change enables the use of the Tokenizer in Rust-based projects.
flexpilot-ai · Nov 1, 2024 · d11d082 · d11d082
1 parent 6c15458
commit d11d082
Show file tree

Hide file tree

Showing 36 changed files with 96,946 additions and 939 deletions.
diff --git a/.github/conda/bld.bat b/.github/conda/bld.bat
diff --git a/.github/conda/build.sh b/.github/conda/build.sh
diff --git a/.github/conda/meta.yaml b/.github/conda/meta.yaml
diff --git a/.github/stale.yml b/.github/stale.yml