What is RAG? Retrieval-augmented generation explained for the Mittelstand
RAG couples a language model to your own documents, so answers stay verifiable and current.
This translation was produced automatically using AI. The German version is the editorially reviewed original.
Retrieval-augmented generation, RAG for short, connects a language model to a searchable collection of your own documents. Instead of generating the answer solely from training knowledge, the system first searches for the matching passages in your data and formulates the answer from them. For the Mittelstand, this is the most practical way to apply a language model to your own knowledge, without retraining a model and without internal documents leaving your own infrastructure.
This piece explains what RAG is, how a pipeline is structured, why the retriever usually matters more than model size, and when the effort pays off.
01. The problem RAG solves
A general-purpose language model does not know your internal documents. It was trained on public data and ends at a cutoff date. If an employee asks about the current supply agreement with a particular vendor, the model has two options: it says it does not know the answer, or it invents one that sounds plausible. The second option is the dangerous one.
RAG closes this gap. Before every answer, the system searches your own sources, such as contracts, manuals, tickets or a wiki, and passes the relevant passages to the model. The model then formulates an answer that is grounded in these specific passages and can cite them. A model that has to guess becomes a system that looks things up. We described the further business benefit in Discover the Potential of AI in Business.
02. How a RAG pipeline works
A RAG pipeline consists of five steps that separate cleanly from one another.
- Indexing. Your documents are broken down into small chunks and translated into numerical vectors that capture their meaning. These vectors are stored in a vector database.
- Retrieval. For a given question, the system computes the matching vector and fetches the most similar chunks from the database. This is where the quality of the later answer is decided.
- Augmentation. The retrieved chunks are placed together with the question into the model's prompt.
- Generation. The model formulates the answer from the supplied context, not from its memory.
- Source citation. The system names the documents the answer came from, so a human can check them.
The important point of this separation is the third one: you can swap out the model without reprocessing your data, and you can update your data without touching the model.
03. Why the retriever often matters more than model size
In practice, most energy goes into choosing the model. In our experience, the lever more often sits with retrieval. We ran the same domain question against a large hosted model and against a smaller local model with a carefully tuned retriever. On clear-cut questions, both performed equally well. On the ambiguous cases, the minority that really counts, retrieval quality decided whether the answer was usable, not model size.
The reason is simple: a larger model just formulates a wrong answer more eloquently if it is missing the right passage. A well-tuned retriever, on the other hand, delivers the matching passage, and then a smaller model is enough. We described this same observation in more detail in our earlier article on retrieval-augmented generation.
04. RAG, data protection, and the on-premise case
RAG fits well with a data-protection-conscious architecture, because every building block can run on your own premises. Retriever, vector database and model can all run on-prem, so that neither the question nor the retrieved documents go to an external service. For companies working with personal data, contracts or engineering documents, this is often the precondition for a project to start at all.
Which model suits local operation, and with what hardware, we cover in Local LLM in the enterprise: hardware, cost, reality. The fundamental trade-off between local operation and the cloud is set out in On-premise vs. cloud LLM: when local AI is the right choice.
05. When RAG pays off and when it does not
RAG pays off when your knowledge sits in documents, changes regularly, and is queried by many people. Typical cases are internal knowledge bases, support, proposal review and searching technical documentation.
RAG pays off less when the answer is an exact calculation from a database, since a classic query is more precise for that. It also does not replace clean data maintenance: if the source documents are contradictory or outdated, even good retrieval will deliver contradictory answers. The first step of a RAG project is therefore usually an honest review of the existing knowledge, not the choice of model.
