Code

Extend Vocabulary#88

Code

Merged

torymur merged 14 commits intomaindottxt-ai/outlines-core:mainfrom extend-vocabularydottxt-ai/outlines-core:extend-vocabularyCopy head branch name to clipboardNov 19, 2024

+1,198-162

Commits

Commits on Nov 5, 2024

• Introduce eos token locator

  torymur

  committed
  1446c76

Commits on Nov 7, 2024

• Introduce token processor

  torymur

  committed
  c3b4430

Commits on Nov 8, 2024

• Extend vocabulary with eos token id & pretrained models

  torymur

  committed
  e85ce90

Commits on Nov 11, 2024

• Move vocabulary to a dedicated module

  torymur

  committed
  5eb350d

• Add more tests

  torymur

  committed
  9b42797

• Improve documentation and visibilities

  torymur

  committed
  d09ea69

Commits on Nov 12, 2024

• Separate and simplify errors

  torymur

  committed
  c7af981

• Python: ignore deprecations warnings

  Show description for fb833ae
  torymur

  committed
  fb833ae

Commits on Nov 13, 2024

• Apply suggestions from CR

  torymur

  committed
  334dab0

• Add test rust coverage cmd to Makefile

  torymur

  committed
  527f952

• Improve test coverage

  torymur

  committed
  2420742

Commits on Nov 14, 2024

• Locator as a trait

  torymur

  committed
  5e0177a

Commits on Nov 19, 2024

• Separate tokenizers errors, test supported pretrained models

  torymur

  committed
  b85523e

• Apply CR suggestions

  torymur

  committed
  741f59c