The OCR-free document stack 2026: ColPali, ColQwen, ModernVBERT, and Qdrant
Visual retrieval models search the page as an image instead of via OCR. A sober overview.
This translation was produced automatically using AI. The German version is the editorially reviewed original.
For document RAG, OCR is no longer the obvious first step in 2026. A set of visual retrieval models searches the page directly as an image, layout, tables, and diagrams included, without breaking it down into text first. That changes where errors arise in a RAG architecture and which building blocks you even still need. Four names keep coming up: ColPali, ColQwen, ModernVBERT, and Qdrant. This piece places them without hype, with the points where the switch pays off and the ones where it does not.
01. Why OCR was the shaky part of the pipeline
OCR was long the point where silent errors crept into the answer. The classic path is a chain: scan the page, convert it to text via OCR, cut it into chunks, embed it, retrieve it. Every stage loses something. A two-column page gets merged incorrectly, a table falls apart into disconnected numbers, a diagram drops out entirely because it is not text. In our experience, for difficult documents the larger part of the quality problem is not in the language model but here, in the reading. Visual retrieval addresses exactly this stage: it skips text recognition at search time and works on the image of the page.
02. The four building blocks, briefly explained
ColPali. The reference, introduced in July 2024 (arXiv). It builds on the vision-language model PaliGemma and generates around 1024 image-patch vectors per page, each 128-dimensional. Instead of compressing the page into a single vector, granularity is preserved. Matching happens via late interaction, a technique adopted from ColBERT, which we break down in more detail here.
ColQwen. The same recipe, a different base: ColQwen2.5 builds on Qwen2.5-VL instead of PaliGemma and mostly leads on the ViDoRe benchmark. If you are starting fresh today, this is a sensible place to begin.
ModernVBERT. The efficiency lever. ColModernVBERT has 250M parameters, around ten times fewer than ColPali, and according to the paper still lags only 0.6 nDCG@5 behind. This is the building block that makes visual retrieval affordable on-prem.
Qdrant. The infrastructure underneath. Qdrant natively stores the multi-vectors of these models and computes the late-interaction score at search time (docs). Without a vector database that understands multiple vectors per page, none of the three encoders work.
03. When the switch pays off, and when it does not
Visual retrieval wins where layout carries the information. Scanned contracts, invoices, data sheets, presentations, forms, multilingual files: everything an OCR chain regularly fails on. It also saves the entire maintenance burden of that chain.
But it is not a blanket win. Multiple vectors per page cost more storage and a larger index than a single text vector, often by a large multiple. For clean, plain running text, classic text RAG remains cheaper and entirely sufficient. And some tasks ultimately still need text, such as full-text search, copying, or an audit trail. There are still good reasons for OCR, in a hybrid alongside visual retrieval.
04. A sober roadmap
- Test with your own documents, not the benchmark. ViDoRe is a good reference point, but it is not your document set. Take the twenty pages where your current search fails today.
- Start with ColQwen or ColModernVBERT, depending on your hardware. On a tight GPU, the small model is often the more honest choice.
- Set up Qdrant as a multi-vector store, using late interaction only at the re-ranking stage to keep the index small.
- Calculate the storage cost up front. Index size is the thing that hurts later, not the model choice.
If you are looking for the bigger picture, namely why local models and in-house retrieval belong together at all: that is covered in On-Premise versus Cloud. We go deeper into the individual building blocks in dedicated articles, starting with the Mittelstand view.
