Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BFCL] Improve multi-turn performance with prompting #874

Open
rolandgvc opened this issue Jan 9, 2025 · 0 comments
Open

[BFCL] Improve multi-turn performance with prompting #874

rolandgvc opened this issue Jan 9, 2025 · 0 comments

Comments

@rolandgvc
Copy link

rolandgvc commented Jan 9, 2025

Describe the issue
LLM is unaware that it should explore its options with the given tools.

ID datapoint

  1. multi_turn_base_2 with grok-beta
  2. Issue: lack of exploration with the available tools
  3. Gorilla repo commit #: 865

What is the issue
LLM is unaware that it should explore its options with the given tools.

Example:

data/possible_answer/BFCL_v3_multi_turn_base.json

{
"role": "user",
"content": "Go into document folder and Could you draft up a create a document titled 'TeamNotes.txt' for keeping track of all the fresh ideas?"
}

data/possible_answer/BFCL_v3_multi_turn_base.json

{
"id": "multi_turn_base_2",
"ground_truth": ["cd(folder='documents')", "touch(file_name='TeamNotes.txt')"]
}

result/grok-beta/BFCL_v3_multi_turn_base_result.json

{
"id": "multi_turn_base_2",
"result": [
      [
        [{ "cd": "{\"folder\":\"document\"}" }],
        "I tried to navigate to the 'document' folder, but it doesn't exist. Could you please confirm if the folder should be created first or if there's another name for the folder you meant?"
      ],
      [
        "I need to create the 'document' folder first since it doesn't exist. After that, I can create the 'TeamNotes.txt' file and add the insights from Simona. Is that okay with you?"
      ],
      [
        "I need to navigate to the 'document' folder first to access the files 'ideas.txt' and 'TeamNotes.txt'. However, the 'document' folder does not exist yet. Would you like me to create the 'document' folder and then proceed with comparing the files?"
      ],
      [
        "I need to navigate to the 'document' folder to check if the 'Archived' directory exists. However, the 'document' folder itself does not exist yet. Would you like me to create the 'document' folder first, and then proceed with checking for the 'Archived' directory and copying 'TeamNotes.txt' to 'IdeasArchive.txt'?"
      ],
      [
        "I need to navigate to the 'document' folder to access 'IdeasArchive.txt'. However, the 'document' folder does not exist yet. Would you like me to create the 'document' folder first, and then proceed with viewing the contents of 'IdeasArchive.txt'?"
      ]
    ],
}

Proposed Changes
Should we include a default system prompt for multi-turn to encourage exploration and discourage user intervention?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant