refreshing documents in index generated form text nodes

[RalfGraefeIntel]

Hi,
I'm still trying to understand how to refresh the underlying documents in an Index if the index was created by nodes and not directly from documents. In my application I first manually create a list of documents. I then want to further split the content of these documents into nodes using something like SentenceWindowNodeParser or SimpleNodeParser. I hand over the documents to those parsers using the get_nodes_from_documents(). I then create an index from the results of the NodeParsers. In this situation what can I do if only some documents change or are added? If I try to apply the refresh_ref_docs() function to my index (generated from nodes) I get the error that text nodes don't have document IDs.

Thanks

[dosubot[bot]]

Based on your description, it seems like you're trying to refresh an index that was created from nodes, but the nodes don't have document IDs. This is causing an issue because the refresh_ref_docs() function relies on document IDs to check if a document already exists in the index and whether it needs to be updated or inserted.

Here's a step-by-step guide on how you can resolve this issue:

1. Ensure Documents Have IDs: Before calling refresh_ref_docs(), make sure each Document object has a unique identifier (doc_id). This is crucial because the refresh_ref_docs() function relies on document IDs to check if a document already exists in the index and whether it needs to be updated or inserted.

2. Use from_documents Method for Initial Index Creation: If you initially created the index from nodes without document IDs, consider reconstructing the index using the from_documents class method. This method is designed to take a sequence of Document objects, which should have document IDs, and create an index from them. This approach ensures that each document (and consequently, each node derived from these documents) is associated with a document ID.

3. Implement Document ID Assignment: If your documents do not have IDs, you need to assign unique IDs to them before indexing. This can be done by modifying the documents or by extending the Document class to generate IDs automatically.

4. Call refresh_ref_docs() with Updated Documents: Once you have an index built from documents with IDs, you can call refresh_ref_docs() with a sequence of updated Document objects. The function will check each document's hash against the stored hash in the document store. If a document does not exist in the store (based on its ID) or if its hash has changed (indicating a change in content or metadata), the document will be inserted or updated in the index.

Here's a simplified example of how you might prepare and call refresh_ref_docs():

# Assuming `documents` is a list of Document objects with unique IDs
updated_documents = [...]  # Your updated documents with IDs

# Create or update the index with updated documents
refreshed_docs_flags = index.refresh_ref_docs(updated_documents)

# `refreshed_docs_flags` will be a list of booleans indicating which documents were refreshed

Remember, the key to using refresh_ref_docs() effectively lies in ensuring that all documents have unique IDs and that the index is initially created with these document IDs in mind. If your workflow starts with nodes that lack direct association with document IDs, you'll need to adapt it to ensure document IDs are assigned and used throughout the indexing and refreshing process.

For more details, you can refer to the source code of the refresh_ref_docs() function in the LlamaIndex codebase.

Sources

• llama-index-legacy/llama_index/legacy/indices/base.py
• llama-index-legacy/llama_index/legacy/node_parser/interface.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[git-hamza]

I have run into the same issue, do we have any answer on the above question?

[bkuss]

I have this issue as well.