Sign in to continue learning
Multi-vector Image Retrieval
Intermediate
Topics
AI Coding
Search and Retrieval
You now know how ColPali works, how its memory footprint can be optimized, and how MUVERA speeds up multi-vector search. Let's put all these different pieces together and build a functioning multi-modal RAG system based on ColPali. Let's build something. In previous lessons, we've seen how ColPali enables visual document retrieval through multi-vector embeddings. Now, you will build a complete RAG system that combines ColPali's retrieval with OpenAI's vision models for generation. This approach lets you answer questions about documents by retrieving relevant pages and having a vision capable language model read them directly. No OCR needed. You will start by loading the OpenAI API key from the environmental file and creating an OpenAI client. You will recreate the collection from lesson three that contains ColPali embeddings with various optimizations. This ensures you can run this lesson independently without completing lesson three first. The collection includes multiple optimization strategies. original embeddings, binary and scalar quantization, and different kinds of pooling. In the previous lesson, you explored MUVERA, which compresses ColPali's variable length multi-vector embeddings into fixed dimensional encodings. This enables HNSW indexing for faster search at scale. You will add MUVERA embeddings to the same collection to demonstrate two-stage retrieval, using MUVERA for fast candidate retrieval, then reranking with ColPali for maximum accuracy. Let's create a MUVERA instance with the same configuration like in the previous lesson. As a next step, you will recreate the Qdrant collection with all the memory optimizations, including MUVERA embeddings. You have another helper function to achieve exactly this. The recreate_colpali_optimizations_collection helper function imported all the memory optimizations we discussed in one of the previous lessons, and also created MUVERA representations for the original ColPali multi-vectors. It's time for queries. This helper function will retrieve precomputed vector embeddings for a list of interesting queries that you can use to test your RAG system. You can now test the helper function with a sample question about the documents in the collection. Instead of embedding queries separately and passing them to search functions, you will create a unified retriever that handles query embedding internally and returns a list of image path and score tuples. The retrieve function takes a textual query and sets using parameter of the query_points method of Qdrant's SDK, to a specific vector name that we want to use for retrieval in that particular case. Because some of the optimization methods we discussed earlier are quantization methods, and Qdrant has a rescoring function built in. We specifically say that we don't want to use the rescoring mechanism because otherwise we could get better results than just with quantized vectors. Another helper function will run search with two-stage retrieval. It will use MUVERA embeddings as an initial prefetch mechanism, and then it will rerank with the vectors that you want to use at that stage. Obviously, we could rescore with the original ColPali embeddings. However, if we are able to use float vectors, we could also save the memory and still get decent results out of this two-stage retrieval. So we are actually going to test both MUVERA for faster retrieval. and different kinds of memory optimizations to get the best out of both worlds. Let's test our retriever function with a sample query. The retrieval will handle the query embedding generation internally, and will return the top results. Here are the top PDF pages for the query, How does the human brain work? Now let's focus on the generation part. You will send the retrieved document pages to GPT-4o along with the user's question. GPT-4o can read text, diagrams, and formulas directly from the images to generate accurate answers. There are different parts of that process. First of all, we are sending a system prompt. telling the model to answer our questions based only on the provided document images. We also want to have an answer generated as a Markdown so we can just render that properly. Then, we'll pass the original prompt sent to the system. Finally, you will attach all the images provided to this particular function. At the very end, this entire interaction will be sent to the OpenAI service to get the LLM output. Let's test the generation with the previously retrieved documents and see how GPT-4o answers our question. Nice. You've just implemented a retrieval augmented generation that works directly on visual documents. Let's test our complete RAG workflow with several different questions. First of all, you will retrieve documents for all the queries using the original ColPali retriever, extract the image paths and generate the answer using GPT-4o's vision capabilities. This two-step approach, retrieve and generate, using separate functions makes the process explicit and give us full control over each stage. You are not only displaying the answer as generated by the LLM, but also the input images that we sent along with the prompt. The same goes for the second query. from our dataset. And the last query. At this point, it shouldn't be surprising that original ColPali vectors did a pretty good job. However, in practice, you will typically try to optimize that process using all the techniques that we learned. And for that, you store multiple optimized vectors in the same collection so you can just test different optimization methods. Let's test some of the techniques we used in one of the previous lessons. You will specifically use binary quantized vectors and hierarchical token pooling with pool factors set to two. If you now test the same retriever but using different optimization techniques, you should see some different outputs because the input documents will be obviously slightly different. Here's the answer generated for the Binary Quantized version of our embeddings. Even though it didn't use the same set of images. It still provided the retrieval response to the input prompt. Hierarchical token pooling was doing even a better job in our experiments. And not surprisingly, it also did a good job. However, memory optimization is typically not enough. As we saw in the previous lesson, MUVERA alone can have lower precision compared to the original ColPali multi-vectors, but it offers significantly faster search through HNSW indexing. The optimal production approach combines both. Use MUVERA to quickly retrieve a broader set of candidates, and then re rank those candidates with any optimization method for the final results. That's why you created a retrieve with two-stage function that implements this pattern using Qdrant's prefetch mechanism with a flexible reranking strategy to control it. Let's see it in action. Let's test the two-stage retrieval and compare it with the original ColPali retriever. For this comparison, you will use the default reranking strategy with original full ColPali embeddings. after MUVERA's fast prefetch. That should hopefully maintain high accuracy while benefiting from MUVERA's HNSW search, which is especially important when scaling to larger document collections. Let's see how the MUVERA two-stage retrieval results look like. The original ColPali retriever does a really good job, but this is our baseline. And two-stage retrieval with MUVERA and ColPali reranking is actually using a very good set of input images, so the output of the LLM is grounded in the knowledge we have. And while MUVERA handles fast prefetch with HNSW indexing, the reranking stage can use any optimization method, not necessarily the original vectors. As a last step, let's compare how this different reranking strategies affect retrieval quality and performance. When we run this experiment, the documents retrieved by MUVERA will be reranked using Original ColPali vectors, Binary Quantized ones, and hierarchical token pooling. Each reranking strategy offers different trade-offs. Original ColPali provides the highest accuracy with full max computation across all multi-vectors. However, binary quantization offers faster reranking by using compressed representation and it also significantly reduces the amount of memory needed to keep all the vectors. Though, with slightly reduced precision. hierarchical token pooling reduces the number of vectors to compare with during reranking, providing another speed accuracy balance point. In production, you may choose binary quantization for cost-sensitive applications or stick with original ColPali when maximum accuracy is critical. In this lesson, we built a complete multimodal RAG system combining ColPali's visual retrieval with OpenAI's vision models. The two-stage retrieval pattern using MUVERA is particularly powerful for production systems. It combines the speed of HNSW indexes while reranking is quite flexible as we can use any optimization method. Throughout the entire course, you learned how to build a retrieval system working directly on scanned documents, PDFs, or images. There are two key issues related to multi-vectors. The first is high memory usage thanks to all those vectors. You can mitigate this issue by using quantization or pooling techniques or even by combining multiple methods together. The second problem is the fact that multi-vector representations are not compatible with the most popular approximate nearest neighbors techniques such as HNSW. So, by default, they require a full scan that doesn't scale. MUVERA is a promising method of converting variable length sequences of vectors into fixed dimensional representations, enabling approximate search. Typically, original vectors are still used for reranking to achieve both speed and high quality. Putting all these tools together, it's possible to build a flexible and high-performing multimodal RAG system, as you just saw.
