Code

Ported token mapping function#22

Code

Merged

menamerai merged 10 commits intomaindelphi-suite/delphi:mainfrom 8-map-tokens-to-prompts-and-positionsdelphi-suite/delphi:8-map-tokens-to-prompts-and-positionsCopy head branch name to clipboardFeb 14, 2024

+88

Commits

Commits on Feb 14, 2024

• add token mapping

  menamerai

  committed
  48e56c5

• use cast for typing

  menamerai

  committed
  210a28a

• moved token_map function to library

  menamerai

  committed
  ebd0314

• added test case for token_map

  menamerai

  committed
  293f2db

• added test cases for token_map

  menamerai

  committed
  9ba18dd

• review changes

  menamerai

  committed
  fa83336

• add load_hf_dataset function

  menamerai

  committed
  399c9e3

• shorten docstring

  menamerai

  committed
  3dedb4c

• revert pickling changes

  menamerai

  committed
  99b4d88

• added token mapping script

  menamerai

  committed
  40f83d2