Retrieval-augmented generation (RAG) explained: pipeline, variants, limits

01

What is retrieval-augmented generation, and where does the term come from?

Retrieval-augmented generation refers to systems that retrieve matching passages from an index before formulating an answer. The term goes back to Lewis et al. (NeurIPS, 2020). Today it stands generally for any architecture in which search and language model are separate, individually testable components.

A large language model (LLM) generates text by calculating the most probable continuation. What it knows about the world sits in its weights, frozen at the point of training. It knows nothing about an organisation's manuals, contracts or tickets, and it cannot say where a statement comes from. Retrieval-augmented generation therefore splits the task in two: a search step (retrieval) finds supporting passages, and only then does the model formulate an answer.

The Lewis et al. paper (NeurIPS, 2020) coupled a generation model with a dense vector index and trained both together. In practice, search and model today usually stay separate components, connected through the prompt and therefore individually swappable and measurable. What is RAG? offers an introduction, and Knowledge management with AI places it in the wider picture.

02

What problem does RAG solve that a language model alone cannot?

A language model on its own knows no internal documents, goes stale along with its training cutoff, and cannot back up its statements. RAG supplies current sources at runtime, makes every answer traceable to its source, and allows access rights to be enforced at the index. New knowledge only needs to be ingested, not retrained.

RAG is usually weighed against three alternatives: the model with no data of its own, a very long context window, and fine-tuning, retraining on an organisation's own examples. RAG or fine-tuning goes deeper into the choice against fine-tuning.

The entry on long contexts rests on Lost in the Middle (Liu et al., 2023): information in the middle of long inputs is used markedly worse than information at the beginning or end.

03

How does a RAG pipeline work, step by step?

A RAG pipeline has two phases. At index time, documents are ingested, split into chunks, embedded as vectors and written to an index. At query time, the question is embedded the same way, matching chunks are retrieved, reranked and passed to the model together with the question, which then answers and names its sources.

The first four stages run whenever the underlying material changes, the last three run for every question. Mistakes made at index time, such as a table torn apart, cannot be fixed at query time any more.

What happens during ingestion

1. Reading text and structure

   Extract text along with headings, lists and tables. Scans need OCR or visual retrieval, see Understanding documents with AI.

2. Carrying metadata along

   Attach title, source, version and validity to every chunk, otherwise neither filtering nor clean citation is possible.

3. Carrying over access rights

   Store read permissions from the source system on each chunk, see Access control in AI knowledge systems.

4. Detecting changes

   Track new, changed and deleted documents so that no outdated version gets cited.

04

How should documents be chunked for RAG?

Good chunking follows the structure of the document: splits at headings, paragraphs and table boundaries, large enough to hold a complete thought, small enough for a precise match. Fixed character lengths are the simplest starting point, but they tear apart tables and lists. Contextual enrichment and parent-child chunks help when individual chunks make no sense without their surroundings.

Chunking decides which units can be found at all. Chunks that are too small lose their context: The deadline is four weeks is worthless without its heading. Chunks that are too large dilute the vector and fill the prompt with text that has nothing to do with the question.

05

What happens during embedding and indexing?

An embedding model translates every chunk into a numerical vector, whose closeness to other vectors represents content similarity. These vectors sit in an index that finds approximate nearest neighbours quickly. A classic keyword index alongside it is worth having, because vectors are poor at matching exact identifiers such as part numbers.

An embedding is only comparable to vectors from the same model. The question and the chunks must therefore be embedded with the same model. The vectors are stored in a vector database or a database extension, which usually works with approximation methods such as HNSW graphs (Malkov and Yashunin, 2016): these very likely, but not necessarily exactly, find the nearest neighbours.

Semantic search in the enterprise goes deeper into vector and full-text search, the particulars of German, and measuring relevance.

06

Why is pure vector search usually not enough for RAG?

Vector search finds meaning but misses exact terms such as standard designations, part numbers and proper names. Hybrid search therefore combines it with BM25 keyword search and merges the two rankings. A reranking step downstream scores the best candidates more precisely, so less irrelevant text ends up in the prompt and the model answers more accurately.

Karpukhin et al. (EMNLP, 2020) measured dense vector search at a top-20 hit rate 9 to 19 percentage points higher than a strong BM25 system. That holds for the data used there. For abbreviations, model designations and rare technical terms, keyword search is often just as good or better.

Hybrid search runs both methods in parallel and merges the rankings, usually with Reciprocal Rank Fusion (Cormack et al., SIGIR 2009). A reranking model then scores the question and candidate together, more precisely than two separately computed vectors.

The 67% figure comes from Anthropic's measurement of Contextual Retrieval (2024): the share of relevant chunks outside the top 20 fell from 5.7% to 1.9%. The order of magnitude transfers, not the exact value.

07

How does the model answer only from the sources and cite them?

The model receives the retrieved chunks numbered and with their provenance, along with an instruction to answer only from them, to back every statement with its number, and to say openly when the chunks contain no answer. The system then checks that the cited numbers actually exist and links the sources.

Three decisions shape generation. Quantity: a few well-ordered chunks beat many mediocre ones. Order: following Liu et al. (2023), the strongest matches belong at the start, not in the middle. Citation discipline: a statement with no number counts as an error in the system.

Answer the question using only the excerpts below.
Support every statement with the number of the excerpt in square brackets, for example [2].
If the excerpts contain no answer, say so and state which information is missing.

[1] Source: maintenance manual, section 4.2, version 2026-03
<excerpt text>

[2] Source: test bench work instruction, section 1
<excerpt text>

Question: <the user's question>

08

Where do RAG systems fail in practice?

RAG systems mostly fail at a handful of recurring points: the right chunk is not found, it is found but not used, outdated versions compete with valid ones, questions span multiple sources, or access rights are mapped incorrectly. Each failure mode has its own diagnostic step, and most causes sit upstream of the language model.

09

What RAG variants exist, and when are they needed?

Gao et al. (2023) distinguish naive, advanced and modular RAG systems. Beyond that, there are variants for specific failure modes: query decomposition for questions spanning multiple sources, graph approaches for connections and overview questions, agentic retrieval for multi-step research, and visual retrieval for layout-heavy documents. Every variant costs extra computation and should solve a measured problem.

Naive RAG is the simple chain of index, retrieval and generation, advanced RAG adds steps such as rewriting the query and reranking, and modular RAG wires together interchangeable building blocks depending on the query. In practice, what matters is which observed failure an extension actually fixes.

• Query decomposition (Query Decomposition) splits a compound question into sub-questions. Query decomposition and the advanced RAG toolkit covers the measurements and the cost.
• RAG-Fusion puts several rephrasings of a question and merges the rankings, see F-RAG (RAG-Fusion).
• Graph-based RAG builds a knowledge graph from the store. Edge et al. (Microsoft Research, 2024) show that plain RAG cannot answer global questions across an entire store. Knowledge graphs and GraphRAG covers when this pays off.
• Agentic retrieval lets a model decide for itself whether, where and how often to search. AI agents in knowledge work explains the loop, the tools and the approval points.
• Visual retrieval embeds document pages as images, as in ColPali (Faysse et al., 2024). Understanding documents with AI covers when this beats OCR.

10

When is RAG the right choice, and when is it not?

RAG fits when answers live in written documents, those documents change, sources are needed, and access rights apply. It fits poorly for exact calculations from databases, for knowledge that exists nowhere in writing, and for contradictory material with no owner. In those cases a database query, a knowledge-elicitation effort, or cleaning up the material first, helps more.

If the knowledge sits only in people's heads, the work starts before the index: Capturing knowledge before it retires. For running on an organisation's own network: Local language models. Self-test: RAG readiness check, implementation: RAG implementation.