Chain-of-Verification - Prompt Engineering

[Ankush-lastmile]

Chain-of-Verification - Prompt Engineering

Chain-of-Verification is a prompt engineering technique to reduce hallucinations from LLMs!

Research Paper (Meta AI): https://arxiv.org/pdf/2309.11495.pdf

This recipe uses AIConfig - an open-source config-based framework for building generative AI applications.

https://github.com/lastmile-ai/aiconfig/tree/main

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[Ankush-lastmile]

Hi @MKhalusova! Would love a review for this recipe. Thanks!

[MKhalusova]

Line #2.    import openai

As this is an open source ai cookbook, let's illustrate this technique with an open source model. From the readme, I can see that these models should be supported:

• https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1
• https://huggingface.co/chat?model=meta-llama/Llama-2-70b-chat-hf

Reply via ReviewNB

[Ankush-lastmile]

I've updated the model usage to use The Mixtral-8x7B-Instruct-v0.1, thanks for the suggestion!

[MKhalusova]

It would be helpful to explain the config a little bit here. At least its essential components. Maybe add some advice on the structure/content of the prompts?

Reply via ReviewNB

[Ankush-lastmile]

Added a description and links

[MKhalusova]

Line #3.    verification_question =  "Where was this coding language developed: {{entity}}?"

How do we choose the verification question?

Reply via ReviewNB

[MKhalusova]

A really interesting prompting technique! Thanks for adding! I left some feedback on the notebook, mainly - let's switch to an open source model (I saw that Mistral and Llama are supported), and remove unnecessary TODOs.

In order to publish the notebook, a few more steps are needed:

• Add the notebook to the _toctree.yml
• Add the notebook to the list of the latest notebooks in the index.md. As this is the most recent addition, put it at the top of the list.
• Right after the first header (notebook title), add yourself as an author, like this: _Authored by: [Your Name](https://huggingface.co/your_profile)_ Feel free to use either your Hugging Face profile, or GitHub profile, it's up to you which one to link.

[MKhalusova]

Once you address this feedback, I'll be able to check if the notebook renders and runs correctly, and if it does, we should be able to merge.

[Ankush-lastmile]

• switched model usage to Mixtral
• rebase to main
• removed Todos
• added descriptions
• updated index.md & _toctree.yml

[Ankush-lastmile]

Hi @MKhalusova! Thanks for the review and the detailed comments. I've gone ahead and addressed your comments, as well as made the necessary changes you've requested. It would be great if you could take another look.

Thanks!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[MKhalusova]

Minor nit: GitHub (H should be capitalized)

Reply via ReviewNB

[MKhalusova]

This paragraph can be much shorter:

The next code cell sets up a 'CoVe prompt template' within a structure known as AIConfig. AIConfig is a data format to organize prompt templates and specific model settings. To create a config you can either:

* Generate it using the AIConfig Vscode Extension, or

* Download the config here and load it with:

config = AIConfigRuntime.load('cove_template_config.json').

Reply via ReviewNB

[MKhalusova]

Line #7.          "baseline_prompt": "Name 20 programming languages that were developed in the United States. Include the developer name in parantheses.",

minor, typo: it should be "parentheses"

Reply via ReviewNB

[MKhalusova]

Line #1.    # verification_question = "Where was {{entity}} born?"

I think you can remove this comment

Reply via ReviewNB

[MKhalusova]

Line #10.        verification_completion = await config.run("verification", params, options=inference_options)

I'm getting an error on this line:

UnwrapError: Called Result.unwrap() on an Err value: "Missing API key 'OPENAI_API_KEY' in environment"

Reply via ReviewNB

[MKhalusova]

Line #5.    # Display with Markdown

There seems to be an issue rendering this output. Let's switch to a simple print.

Reply via ReviewNB

[MKhalusova]

Thank you for iterating on the cookbook notebook! I found a few typos and minor issues, but there's also an important one - the notebook throws an error:
UnwrapError: Called Result.unwrap()on anErr value: "Missing API key 'OPENAI_API_KEY' in environment"

Let's make sure it runs smoothly without errors.