diff --git a/README.md b/README.md index 51ae18b8b..10be61614 100644 --- a/README.md +++ b/README.md @@ -166,10 +166,27 @@ with a Vespa app using streaming mode to retrieve personal data. ## Visual search -* [PDF-Retrieval using ColQWen2 (ColPali) with Vespa](https://pyvespa.readthedocs.io/en/latest/examples/pdf-retrieval-with-ColQwen2-vlm_Vespa-cloud.html) -* [ColPali Ranking Experiments on DocVQA](https://pyvespa.readthedocs.io/en/latest/examples/colpali-benchmark-vqa-vlm_Vespa-cloud.html) -* [Vespa 🤝 ColPali: Efficient Document Retrieval with Vision Language Models](https://pyvespa.readthedocs.io/en/latest/examples/colpali-document-retrieval-vision-language-models-cloud.html) -* [Scaling ColPALI (VLM) Retrieval](https://pyvespa.readthedocs.io/en/latest/examples/simplified-retrieval-with-colpali-vlm_Vespa-cloud.html) +[Pyvespa: Vespa 🤝 ColPali: Efficient Document Retrieval with Vision Language Models](https://pyvespa.readthedocs.io/en/latest/examples/colpali-document-retrieval-vision-language-models-cloud.html). +This notebook demonstrates how to represent [ColPali](https://huggingface.co/vidore/colpali) in Vespa. +ColPali is a powerful visual language model that can generate embeddings for images and text. +In this notebook, we will use ColPali to generate embeddings for images of PDF _pages_ and store them in Vespa. +We will also store the base64 encoded image of the PDF page and some meta data like title and url. +We will then demonstrate how to retrieve the PDF pages using the embeddings generated by ColPali. + +[Pyvespa: Scaling ColPALI (VLM) Retrieval](https://pyvespa.readthedocs.io/en/latest/examples/simplified-retrieval-with-colpali-vlm_Vespa-cloud.html) +This notebook demonstrates how to represent [ColPali](https://huggingface.co/vidore/colpali) in Vespa and to scale to large collections. +Also see the [Scaling ColPali to billions of PDFs with Vespa](https://blog.vespa.ai/scaling-colpali-to-billions/) blog post. + +[Pyvespa: ColPali Ranking Experiments on DocVQA](https://pyvespa.readthedocs.io/en/latest/examples/colpali-benchmark-vqa-vlm_Vespa-cloud.html). +This notebook demonstrates how to reproduce the ColPali results on [DocVQA](https://huggingface.co/datasets/vidore/docvqa_test_subsampled) with Vespa. +The dataset consists of PDF documents with questions and answers. +We demonstrate how we can binarize the patch embeddings and replace the float MaxSim scoring with a hamming based MaxSim +without much loss in ranking accuracy but with a significant speedup (close to 4x) and reducing the memory (and storage) requirements by 32x. + +[Pyvespa: PDF-Retrieval using ColQWen2 (ColPali) with Vespa](https://pyvespa.readthedocs.io/en/latest/examples/pdf-retrieval-with-ColQwen2-vlm_Vespa-cloud.html). +This notebook is a continuation of our notebooks related to the ColPali models for complex document retrieval. +This notebook demonstrates using the new ColQWen2 model checkpoint. +