-
-
Notifications
You must be signed in to change notification settings - Fork 5.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Frontend] Add Command-R and Llama-3 chat template #10496
Open
ccs96307
wants to merge
1
commit into
vllm-project:main
Choose a base branch
from
ccs96307:add-command-r-jinja
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| {{ bos_token -}} | ||
| {% if messages[0]['role'] == 'system' -%} | ||
| {% set loop_messages = messages[1:] -%} | ||
| {% set system_message = messages[0]['content'] -%} | ||
| {% elif false == true -%} | ||
| {% set loop_messages = messages -%} | ||
| {% set system_message = 'You are a large language model called Command R built by the company Cohere. You act as a brilliant, sophisticated, AI-assistant chatbot trained to assist human users by providing thorough responses.' -%} | ||
| {% else -%} | ||
| {% set loop_messages = messages -%} | ||
| {% set system_message = false -%} | ||
| {% endif -%} | ||
|
|
||
| {% if system_message != false -%} | ||
| {{ '<|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|>' + system_message + '<|END_OF_TURN_TOKEN|>' -}} | ||
| {% endif -%} | ||
|
|
||
| {% for message in loop_messages -%} | ||
| {% if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%} | ||
| {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') -}} | ||
| {% endif -%} | ||
| {% set content = message['content'] -%} | ||
| {% if message['role'] == 'user' -%} | ||
| {{ '<|START_OF_TURN_TOKEN|><|USER_TOKEN|>' + content.strip() + '<|END_OF_TURN_TOKEN|>' -}} | ||
| {% elif message['role'] == 'assistant' -%} | ||
| {{ '<|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>' + content.strip() + '<|END_OF_TURN_TOKEN|>' -}} | ||
| {% endif -%} | ||
| {% endfor -%} | ||
|
|
||
| {% if add_generation_prompt -%} | ||
| {{ '<|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>' -}} | ||
| {% endif -%} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| {{ bos_token }}{% if messages[0]['role'] == 'system' -%} | ||
| {% set loop_messages = messages[1:] -%} | ||
| {% set system_message = messages[0]['content'] -%} | ||
| {% else -%} | ||
| {% set loop_messages = messages -%} | ||
| {% set system_message = '## Task and Context\nYou help people answer their questions and other requests interactively. You will be asked a very wide array of requests on all kinds of topics. You will be equipped with a number of search engines or similar tools to help you, which you use to research your answer. You should focus on serving the user\'s needs as best you can, which will be wide-ranging.\n\n## Style Guide\nUnless the user asks for a different style of answer, you should answer in full sentences, using proper grammar and spelling.' -%} | ||
| {% endif -%} | ||
| {{ '<|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|>' -}} | ||
| {{ '# Safety Preamble\nThe instructions in this section override those in the task description and style guide sections. Don\'t answer questions that are harmful or immoral.' -}} | ||
| {{ '\n\n# System Preamble\n## Basic Rules\nYou are a powerful conversational AI trained by Cohere to help people. You are augmented by a number of tools, and your job is to use and consume the output of these tools to best help the user. You will see a conversation history between yourself and a user, ending with an utterance from the user. You will then see a specific instruction instructing you what kind of response to generate. When you answer the user\'s requests, you cite your sources in your answers, according to those instructions.' -}} | ||
| {{ '\n\n# User Preamble\n' + system_message -}} | ||
| {{ '<|END_OF_TURN_TOKEN|>' -}} | ||
|
|
||
| {% for message in loop_messages -%} | ||
| {% set content = message['content'] -%} | ||
| {% if message['role'] == 'user' -%} | ||
| {{ '<|START_OF_TURN_TOKEN|><|USER_TOKEN|>' + content.strip() + '<|END_OF_TURN_TOKEN|>' -}} | ||
| {% elif message['role'] == 'assistant' -%} | ||
| {{ '<|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>' + content.strip() + '<|END_OF_TURN_TOKEN|>' -}} | ||
| {% endif -%} | ||
| {% endfor -%} | ||
|
|
||
| {{ '<|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|><results>' -}} | ||
| {% for document in documents -%} | ||
| {{ '\nDocument: ' + loop.index0|string -}} | ||
| {% for key, value in document.items() -%} | ||
| {{ '\n' + key + ': ' + value|string -}} | ||
| {% endfor -%} | ||
| {% endfor -%} | ||
| {{ '</results>' -}} | ||
| {{ '<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|>' -}} | ||
| {{ 'Carefully perform the following instructions, in order, starting each with a new line.\nFirstly, Decide which of the retrieved documents are relevant to the user\'s last input by writing \'Relevant Documents:\' followed by comma-separated list of document numbers. If none are relevant, you should instead write \'None\'.\nSecondly, Decide which of the retrieved documents contain facts that should be cited in a good answer to the user\'s last input by writing \'Cited Documents:\' followed a comma-separated list of document numbers. If you dont want to cite any of them, you should instead write \'None\'.' -}} | ||
| {% if citation_mode == 'accurate' -%} | ||
| {{ '\nThirdly, Write \'Answer:\' followed by a response to the user\'s last input in high quality natural english. Use the retrieved documents to help you. Do not insert any citations or grounding markup.' -}} | ||
| {% endif -%} | ||
| {{ '\nFinally, Write \'Grounded answer:\' followed by a response to the user\'s last input in high quality natural english. Use the symbols <co: doc> and </co: doc> to indicate when a fact comes from a document in the search result, e.g <co: 0>my fact</co: 0> for a fact from document 0.' -}} | ||
| {{ '<|END_OF_TURN_TOKEN|>' -}} | ||
| {% if add_generation_prompt -%} | ||
| {{ '<|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>' -}} | ||
| {% endif -%} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,179 @@ | ||
| {%- macro json_to_python_type(json_spec) %} | ||
| {%- set basic_type_map = { | ||
| "string": "str", | ||
| "number": "float", | ||
| "integer": "int", | ||
| "boolean": "bool" | ||
| } %} | ||
|
|
||
| {%- if basic_type_map[json_spec.type] is defined %} | ||
| {{- basic_type_map[json_spec.type] }} | ||
| {%- elif json_spec.type == "array" %} | ||
| {{- "List[" + json_to_python_type(json_spec.items) + "]" }} | ||
| {%- elif json_spec.type == "object" %} | ||
| {{- "Dict[str, " + json_to_python_type(json_spec.additionalProperties) + "]" }} | ||
| {%- elif json_spec.type is iterable %} | ||
| {{- "Union[" }} | ||
| {%- for t in json_spec.type %} | ||
| {{- json_to_python_type({"type": t}) }} | ||
| {%- if not loop.last %} | ||
| {{- "," }} | ||
| {%- endif %} | ||
| {%- endfor %} | ||
| {{- "]" }} | ||
| {%- else %} | ||
| {{- "Any" }} | ||
| {%- endif %} | ||
| {%- endmacro %} | ||
|
|
||
| {%- macro old_tool_parser(tools) %} | ||
| {%- for tool in tools %} | ||
| {%- if loop.index0 != 0 %} | ||
| {{- '\n\n' }} | ||
| {%- endif %} | ||
| {{- '```python\ndef ' + tool.name + '(' }} | ||
| {%- for param_name, param_fields in tool.parameter_definitions|items %} | ||
| {%- if loop.index0 != 0 %} | ||
| {{- ', ' }} | ||
| {%- endif %} | ||
| {{- param_name + ': ' }} | ||
| {%- if not param_fields.required %} | ||
| {{- 'Optional[' + param_fields.type + '] = None' }} | ||
| {%- else %} | ||
| {{- param_fields.type }} | ||
| {%- endif %} | ||
| {%- endfor %} | ||
| {{- ') -> List[Dict]:\n """' }} | ||
| {{- tool.description }} | ||
| {%- if tool.parameter_definitions|length != 0 %} | ||
| {{- '\n\n Args:\n ' }} | ||
| {%- for param_name, param_fields in tool.parameter_definitions|items %} | ||
| {%- if loop.index0 != 0 %} | ||
| {{- '\n ' }} | ||
| {%- endif %} | ||
| {{- param_name + ' (' }} | ||
| {%- if not param_fields.required %} | ||
| {{- 'Optional[' + param_fields.type + ']' }} | ||
| {%- else %} | ||
| {{- param_fields.type }} | ||
| {%- endif %} | ||
| {{- '): ' + param_fields.description }} | ||
| {%- endfor %} | ||
| {%- endif %} | ||
| {{- '\n """\n pass\n```' }} | ||
| {%- endfor %} | ||
| {%- endmacro %} | ||
|
|
||
| {%- macro new_tool_parser(tools) %} | ||
| {%- for tool in tools %} | ||
| {%- if loop.index0 != 0 %} | ||
| {{- '\n\n' }} | ||
| {%- endif %} | ||
| {%- if tool.function is defined %} | ||
| {%- set tool = tool.function %} | ||
| {%- endif %} | ||
| {{- '```python\ndef ' + tool.name + '(' }} | ||
| {%- for param_name, param_fields in tool.parameters.properties|items %} | ||
| {%- if loop.index0 != 0 %} | ||
| {{- ', ' }} | ||
| {%- endif %} | ||
| {{- param_name + ': ' }} | ||
| {%- if not param_name in tool.parameters.required %} | ||
| {{- 'Optional[' + json_to_python_type(param_fields) + '] = None' }} | ||
| {%- else %} | ||
| {{- json_to_python_type(param_fields) }} | ||
| {%- endif %} | ||
| {%- endfor %} | ||
| {{- ') -> List[Dict]:\n """' }} | ||
| {{- tool.description }} | ||
| {%- if tool.parameters.properties|length != 0 %} | ||
| {{- '\n\n Args:\n ' }} | ||
| {%- for param_name, param_fields in tool.parameters.properties|items %} | ||
| {%- if loop.index0 != 0 %} | ||
| {{- '\n ' }} | ||
| {%- endif %} | ||
| {{- param_name + ' (' }} | ||
| {%- if not param_name in tool.parameters.required %} | ||
| {{- 'Optional[' + json_to_python_type(param_fields) + ']' }} | ||
| {%- else %} | ||
| {{- json_to_python_type(param_fields) }} | ||
| {%- endif %} | ||
| {{- '): ' + param_fields.description }} | ||
| {%- endfor %} | ||
| {%- endif %} | ||
| {{- '\n """\n pass\n```' }} | ||
| {%- endfor %} | ||
| {%- endmacro %} | ||
|
|
||
| {{- bos_token }} | ||
| {%- if messages[0]['role'] == 'system' %} | ||
| {%- set loop_messages = messages[1:] %} | ||
| {%- set system_message = messages[0]['content'] %} | ||
| {%- else %} | ||
| {%- set loop_messages = messages %} | ||
| {%- set system_message = '## Task and Context\nYou help people answer their questions and other requests interactively. You will be asked a very wide array of requests on all kinds of topics. You will be equipped with a wide range of search engines or similar tools to help you, which you use to research your answer. You should focus on serving the user\'s needs as best you can, which will be wide-ranging.\n\n## Style Guide\nUnless the user asks for a different style of answer, you should answer in full sentences, using proper grammar and spelling.' %} | ||
| {%- endif %} | ||
| {{- '<|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|>' }} | ||
| {{- '# Safety Preamble' }} | ||
| {{- '\nThe instructions in this section override those in the task description and style guide sections. Don\'t answer questions that are harmful or immoral.' }} | ||
| {{- '\n\n# System Preamble' }} | ||
| {{- '\n## Basic Rules' }} | ||
| {{- '\nYou are a powerful conversational AI trained by Cohere to help people. You are augmented by a number of tools, and your job is to use and consume the output of these tools to best help the user. You will see a conversation history between yourself and a user, ending with an utterance from the user. You will then see a specific instruction instructing you what kind of response to generate. When you answer the user\'s requests, you cite your sources in your answers, according to those instructions.' }} | ||
| {{- '\n\n# User Preamble' }} | ||
| {{- '\n' + system_message }} | ||
| {{- '\n\n## Available Tools\nHere is a list of tools that you have available to you:\n\n' }} | ||
| {%- set ns = namespace(new_tools=true) %} | ||
| {%- for tool in tools %} | ||
| {%- if tool.parameter_definitions is defined %} | ||
| {%- set ns.new_tools = false %} | ||
| {%- endif %} | ||
| {%- endfor %} | ||
| {%- if ns.new_tools %} | ||
| {{- new_tool_parser(tools) }} | ||
| {%- else %} | ||
| {{- old_tool_parser(tools) }} | ||
| {%- endif %} | ||
| {{- '<|END_OF_TURN_TOKEN|>' }} | ||
| {%- for message in loop_messages %} | ||
| {%- set content = message['content'] %} | ||
| {%- if message.role == 'user' %} | ||
| {{- '<|START_OF_TURN_TOKEN|><|USER_TOKEN|>' + content|trim + '<|END_OF_TURN_TOKEN|>' }} | ||
| {%- elif message.role == 'system' %} | ||
| {{- '<|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|>' + content|trim + '<|END_OF_TURN_TOKEN|>' }} | ||
| {%- elif message.role == 'assistant' and message.tool_calls is defined %} | ||
| {{- '<|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>' }} | ||
| {%- if message.content is defined %} | ||
| {{- message.content|trim }} | ||
| {%- endif %} | ||
| {{- '\nAction:\n```json\n[\n' }} | ||
| {%- for tool_call in message.tool_calls %} | ||
| {%- if tool_call.function is defined %} | ||
| {%- set tool_call = tool_call.function %} | ||
| {%- endif %} | ||
| {{- '{\n'|indent(4, first=true) }} | ||
| {{- '"tool_name": "'|indent(8, first=true) + tool_call.name + '",\n' }} | ||
| {{- '"parameters": '|indent(8, first=true) }} | ||
| {%- if tool_call.arguments is defined and tool_call.arguments|length > 0 %} | ||
| {{- tool_call.arguments|tojson(indent=4)|indent(8) }} | ||
| {{- '\n' }} | ||
| {%- else %} | ||
| {{- '{}\n' }} | ||
| {%- endif %} | ||
| {{- '}'|indent(4, first=true) }} | ||
| {%- if not loop.last %} | ||
| {{- ',\n' }} | ||
| {%- endif %} | ||
| {%- endfor %} | ||
| {{- '\n]```\n' }} | ||
| {%- elif message.role == 'assistant' %} | ||
| {{- '<|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>' + content|trim + '<|END_OF_TURN_TOKEN|>' }} | ||
| {%- elif message.role == 'tool' %} | ||
| {{- '<|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|><results>\n' }} | ||
| {{- message.content|trim }} | ||
| {{- '</results><|END_OF_TURN_TOKEN|>' }} | ||
| {%- endif %} | ||
| {%- endfor %} | ||
| {{- '<|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|>Write \'Action:\' followed by a json-formatted list of actions that you want to perform in order to produce a good response to the user\'s last input. You can use any of the supplied tools any number of times, but you should aim to execute the minimum number of necessary actions for the input. You should use the `directly-answer` tool if calling the other tools is unnecessary. The list of actions you want to call should be formatted as a list of json objects, for example:\n```json\n[\n {\n "tool_name": title of the tool in the specification,\n "parameters": a dict of parameters to input into the tool as they are defined in the specs, or {} if it takes no parameters\n }\n]```<|END_OF_TURN_TOKEN|>' }} | ||
| {%- if add_generation_prompt %} | ||
| {{- '<|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>' }} | ||
| {%- endif %} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| {% if messages[0]['role'] == 'system' %} | ||
| {% set offset = 1 %} | ||
| {% else %} | ||
| {% set offset = 0 %} | ||
| {% endif %} | ||
|
|
||
| {{ bos_token }} | ||
| {% for message in messages %} | ||
| {% if (message['role'] == 'user') != (loop.index0 % 2 == offset) %} | ||
| {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }} | ||
| {% endif %} | ||
|
|
||
| {{ '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n' + message['content'] | trim + '<|eot_id|>' }} | ||
| {% endfor %} | ||
|
|
||
| {% if add_generation_prompt %} | ||
| {{ '<|start_header_id|>' + 'assistant' + '<|end_header_id|>\n\n' }} | ||
| {% endif %} |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't understand why this is necessary. Llama 3's chat template is already included in the model's
tokenizer_config.jsonand this omits functionality like tool calls. What is the use case for this?