Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Build Index from regex #125

Draft
wants to merge 10 commits into
base: main
Choose a base branch
from
Draft

Build Index from regex #125

wants to merge 10 commits into from

Conversation

torymur
Copy link
Contributor

@torymur torymur commented Dec 12, 2024 •
edited
Loading

Rust

  • Build Index from regex with regex-automata
  • Change Token type from String to Vec<u8>
  • Change type of Vocabulary's eos_token_id from Option<TokenId> to TokenId
  • Stabilize Index interface
  • Stabilize Index python binding interface
  • Stabilize Vocabulary python binding interface
  • Drop all intermediary structures related to interregular workflow
  • Introduce new Guide python binding
  • Add and/or port tests
  • Add docs

Python

  • Drop all intermediary structures related to interregular workflow
  • Fix existing tests to use different interface of Guide, Index & Vocabulary

@torymur torymur added the enhancement New feature or request label Dec 12, 2024
@rlouf
Copy link
Member

rlouf commented Dec 13, 2024

Supersedes #38?

@torymur
Copy link
Contributor Author

torymur commented Dec 13, 2024

Yes, it will

@torymur torymur force-pushed the index-from-regex-97 branch from 933d7b7 to 52b1093 Compare December 13, 2024 21:25
@torymur torymur force-pushed the index-from-regex-97 branch from fa36be9 to 5b8808d Compare December 18, 2024 15:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
2 participants
@torymur @rlouf