feat: return page number of pdf documents upon retrieval

[jasonkang14]

Checklist:

Important

Please review the checklist below before submitting your pull request.

• Please open an issue before creating a PR or link to an existing issue
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods

Description

• Return the original page number of an uploaded pdf file when retrieving knowledge.
  • the value will be None if page is not in the knowledge base
• Closes Include Original Document Page Numbers in Retrieval Results #7738
• Closes Add page number to the segments #7944

Fixes

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update, included: Dify Document
• Improvement, including but not limited to code refactoring, performance optimization, and UI/UX improvement
• Dependency upgrade

Testing Instructions

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

• Tested on different local files besides pdf and it works
• page number is returned when querying a pdf file

[crazywoola]

I tried several PDFs but always get the page number as null.

[jasonkang14]

@crazywoola i will check again.

[jasonkang14]

@crazywoola i think its because not all vector stores save page numbers. can i change those in this PR as well?

[jasonkang14]

@crazywoola it's a bug with weaviate. i have tried other vector stores and it retrieves page numbers. and i have found a bug with milvus so will be making another PR for each

[crazywoola]

@crazywoola it's a bug with weaviate. i have tried other vector stores and it retrieves page numbers. and i have found a bug with milvus so will be making another PR for each

@jasonkang14 You can fix the weaviate db in this PR, and you can open another one to fix others. :)

[jasonkang14]

@crazywoola I have found a better way to resolve this. It turned out that other vector stores were actually returning page which was not included in the attributes in vector_factory.py now the "weaviate bug" has been fixed

[crazywoola]

Cool, I got the page number from the doc. :)

[ifsheldon]

@crazywoola @jasonkang14 This silently breaks my existing Dify that is built from the older source code. My Dify was in use prior to this commit, so the existing weaviate DB does not have page field.

When I updated the source code which includes this commit and built and run it again, I got a weird error stating something like

werkzeug.exceptions.InternalServerError: 500 Internal Server Error: Error during query: [{'locations': [{'column': 155, 'line': 1}], 'message': 'Cannot query field "page" on type "Vector_index_50fde6d5_97eb_4201_8048_c9311f9b8183_Node".', 'path': None}];
Error during query: [{'locations': [{'column': 17082, 'line': 1}], 'message': 'Cannot query field "page" on type "Vector_index_50fde6d5_97eb_4201_8048_c9311f9b8183_Node".', 'path': None}]

The error is caught high up in the stack, so it took me a day to locate the exception.

The exception is caught here:

dify/api/controllers/console/datasets/hit_testing.py

Line 81 in 86f7f24

except Exception as e:

The breaking change is here:

dify/api/core/rag/datasource/vdb/vector_factory.py

Line 33 in 86f7f24

attributes = ['doc_id', 'dataset_id', 'document_id', 'doc_hash', 'page']

The exception is raised here:

dify/api/core/rag/datasource/vdb/weaviate/weaviate_vector.py

Line 197 in 86f7f24

result = (

I think this is actually a breaking change. Probably you should advise on how to do a DB migration?

[jasonkang14]

@ifsheldon On my end, a database migration was not necessary. Dify automatically stores the extracted text from PDFs into the vector store. The recent update simply ensures that the page is now included in the retrieval process. check out the source code below

dify/api/core/rag/extractor/pdf_extractor.py

Line 69 in af92f19

metadata = {"source": blob.source, "page": page_number}

[ifsheldon]

@jasonkang14 The problem, I guess, is if the content of a knowledgebase comes from pure text, then the metadata will not contain page field. However, you changed the default attributes to search, adding page into attributes. Then, the knowledgebase backed by weaviate will throw an exception when performing a search for these attributes(including page). You can try simply creating a knowledgebase from a txt file and making a hit test.