Update README to include chat templating #1372
Conversation
The existing README has underwhelming or incorrect results (Example is underwhelming dottxt-ai#1347) due to lack of templating for instruct models. This adds special tokens for each instruct model call, as well as provide comments on how to obtain/produce special tokens.
| @@ -107,7 +129,7 @@ generator = outlines.generate.choice(model, Sentiment) | |||
| answer = generator(prompt) | |||
| ```` | |||
There was a problem hiding this comment.
Seems this part also needs to be adjusted, but maybe we can show only the difference:
from enum import Enum
class Food(str, Enum):
pizza = "Pizza"
pasta = "Pasta"
salad = "Salad"
dessert = "Dessert"
...
generator = outlines.generate.choice(model, Food)There was a problem hiding this comment.
IMO every example in a README should be completely copy-able with no slice-and-dice on the user's part. This is of course personal preference, so up to y'all.
There was a problem hiding this comment.
If one day we generate documentation with mdbook, it offers a feature for hiding code lines from the user. I realized there isn't a similar feature in GitHub Flavored Markdown (yet?). The closest thing I'm aware of is collapsed sections...
There was a problem hiding this comment.
@yvan-sraka yeah, sadly collapse sections doesn't work in the code snippets. Would be nice if visually repetitive code would be hidden by collapsing, but then it would copy everything including the hidden code, but without tags. This way "copy-paste and it works" magic would not be lost and "what you see is what you get" predictability will also be served.
In mdbook hiding code lines nicely works for running doc tests for example, but on "copying the code" side it still copies just what's visible, which might not work as it is without hidden code parts. Kind of on the same side of things there is also html comments, but also completely not helpful in copying the code snippets.
@cpfiffer fair point!
But this enum example still needs to be updated, considering that we're just showing different ways of multiple choice, maybe we can just extend original section with Enum in the first place and list as an alternative, wdyt?:
import outlines
from enum import Enum
model_name = "HuggingFaceTB/SmolLM2-360M-Instruct"
model = outlines.models.transformers(model_name)
# You must apply the chat template tokens to the prompt!
# See below for an example.
prompt = """
<|im_start|>system
You extract information from text.
<|im_end|>
<|im_start|>user
What food does the following text describe?
Text: I really really really want pizza.
<|im_end|>
<|im_start|>assistant
"""
class Food(str, Enum):
pizza = "Pizza"
pasta = "Pasta"
salad = "Salad"
dessert = "Dessert"
generator = outlines.generate.choice(model, Food)
# You can also pass these choices simply as list:
# generator = outlines.generate.choice(model, ["Pizza", "Pasta", "Salad", "Dessert"])
answer = generator(prompt)
# Likely answer: Pizza
There was a problem hiding this comment.
Awesome, I like the updated example. I'll add it. In general it sounds like we'll need to just overhaul the docs, which I suspect is probably more fruitful after the 1.0 release.
Co-authored-by: Victoria Terenina <[email protected]>
|
Great, appreciate the comments! |
There was a problem hiding this comment.
This overall LGTM. I like that we provide a lot of examples in the README.md and understand why the Positive/Negative one felt counterintuitive... That said, I found most of the examples a bit abstract/useless and would prefer more concrete, real-world use cases!
The example that probably fits this category best is character generation, though it would be even more interesting if one of the character generation fields was an unconstrained (or less constrained) string, something like a description field (eventually with a limit, e.g., less than 500 characters of lore)! Otherwise, I think it could be nice to hide internet culture jokes in example because, hey, we’re cool kids! Ofc, these comments are for future improvement, the PR is already great and ready to be merged as is!
| You can find the chat template tokens in the model's HuggingFace repo or documentation. As an example, the SmolLM2-360M-Instruct special tokens can be found [here](https://huggingface.co/HuggingFaceTB/SmolLM2-360M-Instruct/blob/main/special_tokens_map.json). | ||
|
|
||
| A convenient way to do this is to use the `tokenizer` from the `transformers` library: | ||
|
|
||
| ```python | ||
| from transformers import AutoTokenizer | ||
|
|
||
| tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM2-360M-Instruct") | ||
| prompt = tokenizer.apply_chat_template( | ||
| [ | ||
| {"role": "system", "content": "You extract information from text."}, | ||
| {"role": "user", "content": "What food does the following text describe?"}, | ||
| ], | ||
| tokenize=False, | ||
| add_bos=True, | ||
| add_generation_prompt=True, | ||
| ) | ||
| ``` | ||
|
|
||
| yields | ||
|
|
||
| ``` | ||
| <|im_start|>system | ||
| You extract information from text.<|im_end|> | ||
| <|im_start|>user | ||
| What food does the following text describe?<|im_end|> | ||
| <|im_start|>assistant | ||
| ``` | ||
|
|
There was a problem hiding this comment.
We can leave the warning but this section should be in the documentation and we can add a link from here.
There was a problem hiding this comment.
I would go as far as having this warning before the first example instead of having a separate section.
There was a problem hiding this comment.
Agreed. I'm not sure what the best place is for it. We could add it to the prompting reference page or a new page. Not sure which is better. It wouldn't be too hard to cook up a new page for the chat templating issue, as we can provide lots of little bits of context.
The existing README has underwhelming or incorrect results (Example is underwhelming #1347) due to lack of templating for instruct models.
This adds special tokens for each instruct model call, as well as provide comments on how to obtain/produce special tokens.