Chunk and Embedding Management in LanceDB

[Pipboyguy]

Feature description

Prevent orphaned records during updates and deletions, especially with parent-child table relationships to maintain referential integrity. This will enable chunking mechanisms to work seamlessly once documents are dropped or updated.

Are you a dlt user?

Yes, I run dlt in production.

Use case

When working with hierarchical data structures in LanceDB, it's crucial to maintain referential integrity between parent and child records. This is particularly important for scenarios like document chunking, where a large document (parent) is split into smaller chunks (children) for embedding and similarity searches. The main challenge is to update or delete parent records without orphaning their associated child records.

Proposed solution

• Enhance merge operations to handle parent-child relationships effectively.
• Implement automatic removal of orphaned child records when parent records are updated or deleted.

Related issues

• Efficient Update Strategy for Chunked Documents in Vector Databases #1533
• User-Provided Text Splitters for Document Chunking #1615