ValueError: Vector column vector has variable length vectors Set on_bad_vectors='drop' to remove them, or set on_bad_vectors='fill' and fill_value=<value> to replace them.

[AhmadShaik]

Code:

txt_path = "/media/ahamad/ontent_large_pdf"
# # html_paths = get_sds_paths(pdf_dir, extn=".html")  
# chunking_strategy=SemanticChunking()

# vector_db=PgVector(
#     table_name="autozone_documents",
#     db_url="postgresql+psycopg://ai:ai@localhost:5432/ai",
#     embedder=OllamaEmbedder(),
# ),

vector_db = LanceDb(
    table_name="auto_zone",
    uri="/tmp/lancedb",
    embedder=OllamaEmbedder()
)

knowledge_base = PDFKnowledgeBase(
    path=txt_path,
    vector_db=vector_db,     
    # vector_db=ChromaDb(collection="auto_zone", embedder=OllamaEmbedder()),
    # chunking_strategy=SemanticChunking()
    chunking_strategy=FixedSizeChunking()
)

knowledge_base.load(recreate=False)

Error:

Exception has occurred: ValueError
Vector column vector has variable length vectors Set on_bad_vectors='drop' to remove them, or set on_bad_vectors='fill' and fill_value=<value> to replace them.
  File "/media/ahamad/work/projects/auto_zone/Autozone_experiments/knowledge_base.py", line 47, in <module>
    knowledge_base.load(recreate=False)
ValueError: Vector column vector has variable length vectors Set on_bad_vectors='drop' to remove them, or set on_bad_vectors='fill' and fill_value=<value> to replace them.