Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Core] Omni-Modal Embedding, Vector Index and Retriever #13551

Open
wants to merge 42 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
5f2aadf
Enable multi-modal RAG on arbitrary modality combinations
DarkLight1337 May 16, 2024
c5493dc
Add demo of `OmniModalVectorStoreIndex` heavily based on `multi_modal…
DarkLight1337 May 16, 2024
c625b4c
Move inner classes outside
DarkLight1337 May 17, 2024
9ca072c
Fix using pydantic field instead of dataclass field
DarkLight1337 May 17, 2024
2951c29
Add missing `wikipedia` dependency
DarkLight1337 May 17, 2024
a56c48d
Fix some issues in `GenericTransformComponent`
DarkLight1337 May 18, 2024
1654df6
Fix self equality
DarkLight1337 May 18, 2024
74a0963
Add tests for `OmniModalEmbedding`
DarkLight1337 May 18, 2024
311bf26
Add tests for `OmniModalEmbeddingBundle`
DarkLight1337 May 18, 2024
2630e7a
Code cleanup
DarkLight1337 May 18, 2024
9c4938d
Fix incompatible override
DarkLight1337 May 21, 2024
3370419
Enable `const` inference of key type parameter
DarkLight1337 May 23, 2024
5567ea2
Merge branch 'main' into omni-modal
DarkLight1337 May 23, 2024
7fc7594
Improve error message
DarkLight1337 May 27, 2024
3ccce97
Fix unexpected error in `from_documents` due to the default behaviour…
DarkLight1337 May 27, 2024
22e4d60
Avoid expensive logging operation for large data objects
DarkLight1337 May 27, 2024
5b61e90
Merge branch 'main' into omni-modal
DarkLight1337 May 27, 2024
ab3b32e
Apply #13352
DarkLight1337 May 27, 2024
ca5236c
Apply #13712
DarkLight1337 May 27, 2024
60513d1
Avoid expensive validation of embeddings
DarkLight1337 May 27, 2024
92acc6d
Add sanity check for multi-modal persist/load cycle
DarkLight1337 Jun 3, 2024
5bb6331
Cleanup
DarkLight1337 Jun 6, 2024
fb04e0c
Add retrieval evaluator for any modality
DarkLight1337 Jun 7, 2024
1e6d158
Add expected docs to result
DarkLight1337 Jun 12, 2024
43fb821
Merge branch 'main' into omni-modal
DarkLight1337 Jun 12, 2024
0d7ce4f
Apply #13973
DarkLight1337 Jun 12, 2024
83d30d0
Apply #14003
DarkLight1337 Jun 12, 2024
761c816
Apply #13700
DarkLight1337 Jun 12, 2024
9331e9a
Skip test when CLIP is not available
DarkLight1337 Jun 12, 2024
e69fdb8
Fix circular import (follow-up to "Apply #13973")
DarkLight1337 Jun 12, 2024
6c8bdf6
Add description for tqdm
DarkLight1337 Jun 12, 2024
f6b7ef8
Merge branch 'main' into omni-modal
DarkLight1337 Jun 14, 2024
1be9920
Add generic parameters
DarkLight1337 Jun 15, 2024
8712914
Merge branch 'main' into omni-modal
DarkLight1337 Jun 16, 2024
13420d9
Merge branch 'main' into omni-modal
DarkLight1337 Jun 18, 2024
ba0199d
Merge branch 'main' into omni-modal
DarkLight1337 Jun 18, 2024
73e1527
Merge branch 'main' into omni-modal
DarkLight1337 Jun 20, 2024
aa2833f
Fix incomplete serialization
DarkLight1337 Jul 18, 2024
ffe9750
Apply #14383
DarkLight1337 Jul 18, 2024
6eb4583
Merge branch 'main' into omni-modal
DarkLight1337 Jul 18, 2024
ba7c68f
Factor out validation logic
DarkLight1337 Jul 18, 2024
b5840eb
More simplification
DarkLight1337 Jul 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions docs/docs/examples/multi_modal/multi_modal_retrieval.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -210,9 +210,8 @@
],
"source": [
"import qdrant_client\n",
"from llama_index.core import SimpleDirectoryReader\n",
"from llama_index.core import SimpleDirectoryReader, StorageContext\n",
"from llama_index.vector_stores.qdrant import QdrantVectorStore\n",
"from llama_index.core import VectorStoreIndex, StorageContext\n",
"from llama_index.core.indices import MultiModalVectorStoreIndex\n",
"\n",
"# Create a local Qdrant vector store\n",
Expand Down
745 changes: 745 additions & 0 deletions docs/docs/examples/multi_modal/omni_modal_retrieval.ipynb

Large diffs are not rendered by default.

Loading