You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the issue
LLM is unaware that it should explore its options with the given tools.
ID datapoint
multi_turn_base_2 with grok-beta
Issue: lack of exploration with the available tools
Gorilla repo commit #: 865
What is the issue
LLM is unaware that it should explore its options with the given tools.
Example:
data/possible_answer/BFCL_v3_multi_turn_base.json
{
"role": "user",
"content": "Go into document folder and Could you draft up a create a document titled 'TeamNotes.txt' for keeping track of all the fresh ideas?"
}
{
"id": "multi_turn_base_2",
"result": [
[
[{ "cd": "{\"folder\":\"document\"}" }],
"I tried to navigate to the 'document' folder, but it doesn't exist. Could you please confirm if the folder should be created first or if there's another name for the folder you meant?"
],
[
"I need to create the 'document' folder first since it doesn't exist. After that, I can create the 'TeamNotes.txt' file and add the insights from Simona. Is that okay with you?"
],
[
"I need to navigate to the 'document' folder first to access the files 'ideas.txt' and 'TeamNotes.txt'. However, the 'document' folder does not exist yet. Would you like me to create the 'document' folder and then proceed with comparing the files?"
],
[
"I need to navigate to the 'document' folder to check if the 'Archived' directory exists. However, the 'document' folder itself does not exist yet. Would you like me to create the 'document' folder first, and then proceed with checking for the 'Archived' directory and copying 'TeamNotes.txt' to 'IdeasArchive.txt'?"
],
[
"I need to navigate to the 'document' folder to access 'IdeasArchive.txt'. However, the 'document' folder does not exist yet. Would you like me to create the 'document' folder first, and then proceed with viewing the contents of 'IdeasArchive.txt'?"
]
],
}
Proposed Changes
Should we include a default system prompt for multi-turn to encourage exploration and discourage user intervention?
The text was updated successfully, but these errors were encountered:
Describe the issue
LLM is unaware that it should explore its options with the given tools.
ID datapoint
multi_turn_base_2
withgrok-beta
What is the issue
LLM is unaware that it should explore its options with the given tools.
Example:
data/possible_answer/BFCL_v3_multi_turn_base.json
data/possible_answer/BFCL_v3_multi_turn_base.json
result/grok-beta/BFCL_v3_multi_turn_base_result.json
Proposed Changes
Should we include a default system prompt for multi-turn to encourage exploration and discourage user intervention?
The text was updated successfully, but these errors were encountered: