Long Form Question and Answer generates answers not in docs saved to document store

[heyt0pe]

Using the LFQA tutorial here, I noticed that when I make queries, it often gives me answers that are not in any of the documents I saved to the document store. I even asked it off-topic questions like "Who is barrack Obama?" and "What is a prime minister?" and it still generated fairly accurate answers. I set up my LFQA exactly as is in the tutorial so for the retriever query embedding model is vblagoje/dpr-question_encoder-single-lfqa-wiki and passage embedding model is vblagoje/dpr-ctx_encoder-single-lfqa-wiki, while for the generator, the model is vblagoje/bart_lfqa.

I think the data it uses to generate answers for me where part of the data the model was trained with, but I'm not so sure, regardless I wanted to know if it is possible for me to have a LFQA form where the answers generated to my queries are only gotten from the documents within my document store. Thank you.

[anakin87]

Hello @heyt0pe!

As you can read in the documentation, Answer Generator systems (such as LFQA) draw upon both the knowledge gained during language model pretraining (parametric memory) and the passages provided to it with a Retriever (non-parametric memory).

So it is expected that these systems can also answer questions that are not covered in your collection of documents.

---

I wanted to know if it is possible for me to have a LFQA form where the answers generated to my queries are only gotten from the documents within my document store

Not sure if this is feasible with these types of models, so I tag @vblagoje who is a real expert.

[julian-risch]

Hi @heyt0pe I think if you want to limit LFQA as you describe, what you are looking for instead is RAG (Retrieval Augmented Generation). We have a tutorial on that here: https://github.com/deepset-ai/haystack-tutorials/blob/main/tutorials/07_RAG_Generator.ipynb
With RAG, the generated answer is based on documents retrieved from your collection of documents. The model can't make up an answer that is not contained in you collection of documents in that case.

[felixriehm]

@julian-risch when i followed the RAG and LFQA tutorials, it was the other way around for me... for some reason the RAG tutorial gave me answers that were not found in my documents while LFQA only returned answers in my documents. wtf 🤔

[ZanSara]

@felixriehm can you share something more? It's quite impossible to tell why it could have happened, with this amount of information.

[felixriehm]

Sorry for the inconvenience. I was in a rush. My issue is not relevant for me at the moment. I just stumbled on the dissussion and was wondering.

[ZanSara]

No problem! Just curious how you managed to get such outcomes 🙂