Skip to content

Commit

Permalink
docs: note on toolcalls in implementing engines
Browse files Browse the repository at this point in the history
  • Loading branch information
zhudotexe committed Nov 8, 2023
1 parent 5dce06b commit 6a27a3a
Show file tree
Hide file tree
Showing 2 changed files with 11 additions and 1 deletion.
9 changes: 8 additions & 1 deletion docs/engines/implementing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,10 @@ You'll need to implement two methods: :meth:`.BaseEngine.predict` and :meth:`.Ba
build such a prompt. :meth:`.BaseEngine.function_token_reserve` tells kani how many tokens that prompt takes, so the
context window management can ensure it never sends too many tokens.

You'll also need to add previous function calls into the prompt (e.g. in the few-shot function calling example).
When you're building the prompt, you'll need to iterate over :attr:`.ChatMessage.tool_calls` if it exists, and add
your model's appropriate function calling prompt.

To parse the model's requests to call a function, you also do this in :meth:`.BaseEngine.predict`. After generating the
model's completion (usually a string, or a list of token IDs that decodes into a string), separate the model's
conversational content from the structured function call:
Expand All @@ -56,4 +60,7 @@ conversational content from the structured function call:
:align: center

Finally, return a :class:`.Completion` with the ``.message`` attribute set to a :class:`.ChatMessage` with the
appropriate :attr:`.ChatMessage.content` and :attr:`.ChatMessage.function_call`.
appropriate :attr:`.ChatMessage.content` and :attr:`.ChatMessage.tool_calls`.

.. note::
See :ref:`functioncall_v_toolcall` for more information about ToolCalls vs FunctionCalls.
3 changes: 3 additions & 0 deletions docs/function_calling.rst
Original file line number Diff line number Diff line change
Expand Up @@ -317,3 +317,6 @@ following internal representation:

:attr:`.ChatMessage.function_call` is actually an alias for ``ChatMessage.tool_calls[0].function``. If there is more
than one tool call in the message, kani will raise an exception.

A ToolCall is effectively a named wrapper around a :class:`.FunctionCall`, associating the request with a generated
ID so that its response can be linked to the request in future rounds of prompting.

0 comments on commit 6a27a3a

Please sign in to comment.