This PR drops all intermediate python and related rust structures and builds an Index from regex purely in rust. Also stabilizing python bindings for Vocabulary, Index and Guide and rust interfaces for Vocabulary and Index.

• Closes Build index from regex #97
• Closes Change vocabulary interface #92
• Closes Add doc-strings to ported code #79
• Supersedes PR Port necessary interegular functionality #38 and closes Port necessary interegular functionality #8
• Makes irrelevant and closes Create Rust-only index construction function #9
• Makes irrelevant and closes Handle regex-to-FSM conversion in Rust #10
• Makes irrelevant and closes Port walk_fsm tests to Rust #68
• Makes irrelevant and closes Port create_fsm_index_end_to_end tests to Rust #69
• Makes irrelevant and closes Move test_guide tests to test_regex #71

Rust

• Build Index from regex with regex-automata
• Change Token type from String to Vec<u8>
• Change type of Vocabulary's eos_token_id from Option<TokenId> to TokenId
• Stabilize Index interface
• Stabilize Index python binding interface
• Stabilize Vocabulary interface
• Stabilize Vocabulary python binding interface
• Drop all intermediary structures related to interregular workflow
• Introduce new Guide python binding
• Add and/or port tests
• Add docs

Python

• Drop all intermediary structures related to interregular workflow
• Fix existing tests to use different interface of Guide, Index & Vocabulary
• Adjust benchmarks to use new interface

Benchmarks

Benchmarks reveal x10+ times on average improvement for index compilation comparing to the current main branch.

On my machine to the degree, where a showcase with setting longer GIL switch interval for multithreaded case doesn't exhibit difference with one thread (because it's so fast in the first place, that there is a little sense to give it more time to execute without interruption), but it depends on machine.