Skip to content

Commit

Permalink
Add details to visual search section
Browse files Browse the repository at this point in the history
  • Loading branch information
kkraune committed Dec 19, 2024
1 parent b4b08b9 commit 224acd9
Showing 1 changed file with 21 additions and 4 deletions.
25 changes: 21 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,10 +166,27 @@ with a Vespa app using streaming mode to retrieve personal data.


## Visual search
* [PDF-Retrieval using ColQWen2 (ColPali) with Vespa](https://pyvespa.readthedocs.io/en/latest/examples/pdf-retrieval-with-ColQwen2-vlm_Vespa-cloud.html)
* [ColPali Ranking Experiments on DocVQA](https://pyvespa.readthedocs.io/en/latest/examples/colpali-benchmark-vqa-vlm_Vespa-cloud.html)
* [Vespa 🤝 ColPali: Efficient Document Retrieval with Vision Language Models](https://pyvespa.readthedocs.io/en/latest/examples/colpali-document-retrieval-vision-language-models-cloud.html)
* [Scaling ColPALI (VLM) Retrieval](https://pyvespa.readthedocs.io/en/latest/examples/simplified-retrieval-with-colpali-vlm_Vespa-cloud.html)
[Pyvespa: Vespa 🤝 ColPali: Efficient Document Retrieval with Vision Language Models](https://pyvespa.readthedocs.io/en/latest/examples/colpali-document-retrieval-vision-language-models-cloud.html).
This notebook demonstrates how to represent [ColPali](https://huggingface.co/vidore/colpali) in Vespa.
ColPali is a powerful visual language model that can generate embeddings for images and text.
In this notebook, we will use ColPali to generate embeddings for images of PDF _pages_ and store them in Vespa.
We will also store the base64 encoded image of the PDF page and some meta data like title and url.
We will then demonstrate how to retrieve the PDF pages using the embeddings generated by ColPali.

[Pyvespa: Scaling ColPALI (VLM) Retrieval](https://pyvespa.readthedocs.io/en/latest/examples/simplified-retrieval-with-colpali-vlm_Vespa-cloud.html)
This notebook demonstrates how to represent [ColPali](https://huggingface.co/vidore/colpali) in Vespa and to scale to large collections.
Also see the [Scaling ColPali to billions of PDFs with Vespa](https://blog.vespa.ai/scaling-colpali-to-billions/) blog post.

[Pyvespa: ColPali Ranking Experiments on DocVQA](https://pyvespa.readthedocs.io/en/latest/examples/colpali-benchmark-vqa-vlm_Vespa-cloud.html).
This notebook demonstrates how to reproduce the ColPali results on [DocVQA](https://huggingface.co/datasets/vidore/docvqa_test_subsampled) with Vespa.
The dataset consists of PDF documents with questions and answers.
We demonstrate how we can binarize the patch embeddings and replace the float MaxSim scoring with a hamming based MaxSim
without much loss in ranking accuracy but with a significant speedup (close to 4x) and reducing the memory (and storage) requirements by 32x.

[Pyvespa: PDF-Retrieval using ColQWen2 (ColPali) with Vespa](https://pyvespa.readthedocs.io/en/latest/examples/pdf-retrieval-with-ColQwen2-vlm_Vespa-cloud.html).
This notebook is a continuation of our notebooks related to the ColPali models for complex document retrieval.
This notebook demonstrates using the new ColQWen2 model checkpoint.




Expand Down

0 comments on commit 224acd9

Please sign in to comment.