RAG Evaluations Primary Discussion

[jalling97]

RAG Evaluations

This is where discussions related to RAG evaluations for LFAI will be had.

Epic

Please see here for the current status of the RAG Evals MVP epic.

Design Docs

Please see the current draft for the Design Doc here (INTERNAL ONLY).

Current Repos

These are the current repositories related to RAG evals experiments. Once portions are mature enough to incorporate into the main LFAI repository, they will be ported over.

• RAG qa dataset: https://huggingface.co/datasets/jalling/LFAI_RAG_qa_v1
• RAG qa dataset generation: https://github.com/jalling97/rag_eval_dataset_generation
• RAG niah dataset: https://huggingface.co/datasets/jalling/LFAI_RAG_niah_v1
• RAG niah dataset generation: https://github.com/jalling97/rag_niah_dataset_generation

[alekst23]

I put together a draft of a little visual:

https://lucid.app/lucidchart/80b6a515-1be0-4bb4-a5c2-a67dfbd12b3b/edit?viewport_loc=639%2C-491%2C2625%2C1587%2C0_0&invitationId=inv_b3e56fe4-71ab-4f8d-a618-5f7eaebe75e7