Best llama index tool or implementation for multi document comparison

[sumitbindra]

Hello,

I am trying to setup an example tat requires me to compare multiple documents with each other. I am looking for opinions on what is the best way to do this using llama index.

Some of the considerations are:

1. I want to keep both documents separate and want reference from each when answering questions
2. I have a preset list of queries queries that i want to run over both of the documents and get answers for
3. the documents have hierarchy i.e. one of the document is ground truth and others are the ones that we are comparing against this ground truth

Do you have suggestions on what examples I should look at and how to go about doing this?

Thanks.

[dosubot[bot]]

The best tool for your needs is the Multidoc Autoretrieval Pack. It supports:

1. Document Separation: Keeps documents separate while allowing references.
2. Preset Queries: You can run preset queries using the pack.run() method.
3. Hierarchical Structures: Handles hierarchical document structures effectively.

Here's a brief example of how you might use it:

response = pack.run("Tell me about some issues on 12/11")
print(str(response))

response = pack.run("Tell me about some open issues related to agents")
print(str(response))

This setup is ideal for structured and efficient document comparison tasks ^[1].

To continue talking to Dosu, mention @dosu.

[sumitbindra]

No I dont want to use a pack but what to implement this from scratch so I understand it well using llama index.

Can you suggest how?

[dosubot[bot]]

To implement multi-document comparison from scratch using LlamaIndex, follow these steps:

1. Document Separation and Referencing:

   • Use Documents and Nodes to keep documents separate and maintain references. Each Document acts as a container, and Nodes represent chunks with metadata for referencing ^[1].

2. Preset Queries:

   • Use the run() function to execute preset queries on your documents. This function is a wrapper around query_engine.query() and allows you to run queries directly on your documents ^[2].

3. Hierarchical Structures:

   • Represent hierarchical document structures using the IndexGraph class with IndexStructType.TREE to manage relationships between documents ^[3].

4. Custom Document Comparison:

   • Create a custom retriever by subclassing CustomPGRetriever. Implement methods to retrieve and compare documents based on your criteria. Use multiple retrieval strategies and reranking if needed ^[4].

5. Metadata and Tagging:

   • Use metadata to tag documents and control their visibility during retrieval. Customize metadata format and use filters to refine queries ^[5].

Here's a basic outline to get you started:

from llama_index.core import Document
from llama_index.core.retrievers import CustomPGRetriever

# Define your documents with metadata
doc1 = Document(text="Document 1 content", metadata={"type": "ground_truth"})
doc2 = Document(text="Document 2 content", metadata={"type": "comparison"})

# Implement a custom retriever
class MyCustomRetriever(CustomPGRetriever):
    def custom_retrieve(self, query_str: str) -> str:
        # Custom retrieval logic
        pass

# Use the retriever to run queries
retriever = MyCustomRetriever()
response = retriever.custom_retrieve("Your query here")

This approach gives you full control over the document comparison process, allowing you to understand and customize each step.

To continue talking to Dosu, mention @dosu.

[akashmathur-2212]

Hello Sumit,
I am also working on a similar usecase. Can you connect with me and we can discuss our learnings?