How AI reads documents: OCR, layout and visual search

01

What does it mean for AI to understand a document?

AI understands a document when it does three things: recognise the characters, map them onto the page's structure, such as a table cell, heading or form field, and connect both to a question or a target schema. Each layer builds on the one beneath it, so errors at the bottom propagate upward.

A digitally created PDF already contains a text layer, whereas a scan is only pixels. An invoice table is unambiguous to the eye, but to a machine it starts out as a set of characters with coordinates. Whether the number 240 is a quantity or an amount is not written in the digits themselves, but in the column they sit in.

All three layers keep coming up. How retrieved passages become an answer with a citation is covered in Retrieval-augmented generation explained. Where documents fit into the bigger picture of capturing, finding and maintaining knowledge is covered in Knowledge management with AI.

02

How does a classic OCR pipeline work?

A classic OCR pipeline first prepares the image: deskewing, rotation and contrast. A layout analysis step then finds text blocks and lines, a recognition model reads each line, and post-processing checks the result against dictionaries or formats. The output is text with coordinates that can be searched, copied and processed further.

Modern OCR tools no longer recognise letter by letter but read whole lines with neural networks. The open-source engine Tesseract, for example, includes a recognition system based on recurrent networks (LSTM) that, according to the project documentation, works as a line recogniser and needs roughly ten times the compute of the older recogniser (Tesseract documentation). The gain in accuracy comes at the cost of compute.

The pipeline's weak point is rarely character recognition itself, but the step before it. If layout analysis misses two columns, recognition reads straight across the column boundary, and every line produces a grammatically plausible but factually wrong sentence. The result looks clean and is still useless. These silent errors are exactly what makes OCR the most common source of error in document projects, as the glossary entry on OCR also notes.

03

What can layout-aware models do when they read text and position together?

Layout-aware models process each word's position on the page as well, often together with the image itself. That lets them learn that a value to the right of "Date" is a date, and a number in the third table column is an amount. They handle form and table tasks noticeably better than models that only see the character sequence.

LayoutLM (Xu et al., KDD 2020) laid the groundwork: the model was the first to be pre-trained jointly on text and layout positions, and according to the paper improved the form-understanding score from 70.72 to 79.27. Successors such as LayoutLMv3 (Huang et al., ACM Multimedia 2022) process text and image patches in a single transformer and learn which parts of the image belong to which words.

A second branch skips upstream OCR entirely. Donut (Kim et al., ECCV 2022) reads the page image directly and produces structured output. The authors justify this with three drawbacks of classic pipelines: the compute cost of OCR, poor flexibility across languages and document types, and OCR errors propagating into every downstream step. Today's multimodal language models, which process image and text together, continue that line (multimodal model).

Layout analysis as a task in its own right

How well a system recognises blocks depends heavily on the documents it learned from. The DocLayNet dataset (Pfitzmann et al., KDD 2022) was annotated by humans, distinguishes 11 layout classes and deliberately covers a range of document types. The authors show that models trained on it generalise better across document types than models that have only seen scientific papers.

04

How does visual document search work with models like ColPali?

Visual document search splits every page into many small image patches and computes a vector for each one. The search query is likewise broken into vectors, one per token. During matching, every query token looks for its best-matching patch, and the top scores are summed. That lets the search find pages whose information sits in the layout.

ColPali (Faysse et al., arXiv 2024, published at ICLR 2025) embeds document page images directly, according to the paper, and matches them using the late interaction principle. The authors introduced the ViDoRe benchmark alongside it, because existing search systems mostly score extracted text and ignore visual cues such as tables, charts and font sizes. By their measurements, the approach outperforms classic pipelines while also being simpler to build.

1. Capture pages as images

   Every page is rendered or scanned. No OCR is needed for the search itself.

2. Embed the patches

   The model divides the page into a grid of small patches and produces a vector for each one, so a single page ends up described by many vectors.

3. Embed the query

   The question is broken into tokens, and each one gets its own vector.

4. Compare late

   For every query token, only the most similar page patch counts. The sum of these best scores becomes the page's score. The method is called MaxSim and comes from ColBERT (Khattab and Zaharia, SIGIR 2020).

5. Re-rank the candidates

   In practice, a faster search often narrows down the candidates first, and the expensive comparison only runs over those pages.

The main drawback is storage: many vectors per page take up a multiple of the index a single text vector would need. Smaller models shift that calculation. ModernVBERT (Teiletche et al., arXiv 2025) needs only 250 million parameters and, according to the paper, outperforms models up to ten times its size once fine-tuned for document search. The OCR-free document stack lays out the building blocks, and Late interaction explained goes deeper on the late interaction principle.

05

Extraction or search: which task should document AI actually solve?

Extraction and search need different things. Extraction transfers values into a fixed schema, such as a date, case number or reading, and needs exact characters to do it. Search must find the right page for a question and tolerates small character errors as long as the page turns up. Naming the task first picks the right technique.

Many projects need both. A proven pattern is to split the work: visual search finds the handful of relevant pages, and only those get a thorough OCR pass whose result is cited or processed further. That keeps the expensive stage small and stops OCR errors from spoiling the search in the first place. Classic OCR alongside visual search describes this split using a concrete model.

06

Why do tables, scans and handwriting fail so often?

Tables, scans and handwriting fail because their information does not sit in the characters alone. In tables it sits in rows and columns, in scans the image itself is already degraded, and in handwriting the characters themselves vary. Public benchmarks show that no single approach leads across all document types, so testing on your own collection is what decides it.

The authors of OmniDocBench (Ouyang et al., CVPR 2025) find that classic pipelines and end-to-end vision-language models each have their own strengths and weaknesses depending on document type. That is an argument against picking an approach purely by leaderboard rank.

07

When is classic OCR still the better choice?

Classic OCR is the better choice whenever you need the actual characters: for full-text search across an archive, for quoting and copying, as an audit trail, for handing exact values to other systems, and for documents whose information sits almost entirely in running text. Visual search delivers a page, not text, and does not substitute for any of these tasks.

• Full text and quotes: A result that a person needs to copy or quote verbatim requires recognised text.
• Audit trail: Anyone who has to prove which word at which position a value came from needs characters with coordinates.
• Running text with simple layout: For reports, minutes or contracts with plain formatting, text search is cheaper and good enough.
• Handing off to systems: Databases and line-of-business applications expect values, not page images.

08

How do you check the quality of document AI?

You check document AI quality layer by layer: character error rate for OCR, structural fidelity for tables, share of correct fields for extraction, and recall for search. The basis is a test set of your own, difficult documents with verified expected values. A single overall score hides which layer the error actually comes from.

Checking AI answers goes deeper on how to judge a system's answers against the documents it retrieved, that is, faithfulness to the source and relevance.

09

How do you pick the right approach for your own document collection?

Three questions decide the right approach: does the task need exact characters or the right page? Does the layout carry the information, as in tables and forms? And are the documents allowed to leave your own network? The answers determine whether OCR, visual search or a combination takes centre stage, and what gets tested first.

The third question, whether documents may leave the building, changes the choice more than any leaderboard. OCR, layout models and compact visual search models all run on your own hardware today. Local language models explains what running models locally requires.

iiterate Technologies GmbH builds AI applications and custom software from architecture through implementation to operation, on client infrastructure or in EU hosting, with a focus on RAG, local language models and integration with existing systems (Services).