Code

backport small upstream improvements#16

Code

Merged

tmm1 merged 3 commits intomainanysphere/tiktoken-rs:mainfrom upstream-fixesanysphere/tiktoken-rs:upstream-fixesCopy head branch name to clipboardOct 17, 2024

+38-25

Commits

Commits on Oct 17, 2024

• Avoid calling byte_pair_encode for existing tokens

  Show description for 48e7eda
  

  Lőrinc

  authored and

  tmm1

  committed
  48e7eda

  Copy full SHA for 48e7eda

• Inline custom mapping function in _byte_pair_merge

  Show description for 0ecd650
  

  Lőrinc

  authored and

  tmm1

  committed
  0ecd650

  Copy full SHA for 0ecd650

• upstream fixups to byte_pair_encode

  Show description for 2692606
  tmm1

  committed
  2692606

  Copy full SHA for 2692606