Why AI features fail in practice: seven stages where errors arise

From the outside, two things are visible in an AI feature: the question and the answer. Everything in between looks like a closed box, so complaints land on the language model, because it is the only component whose name everyone knows. The obvious reaction is a different model, a longer instruction or a more expensive plan.

In a built system the box is a chain. Documents are collected, split, indexed and searched for every question; only then does a model write, and ideally a further stage checks the result before it is output. Break the technician's complaint down along this chain and a different picture appears: the manual was in the index, but the correct section only came sixth in the result list, and only the first four were passed to the model.

In this case the repair lies in retrieval. Nobody would need to change anything in the language model, and switching models would at best have covered up the fault.

The breakdown that follows is built for troubleshooting. How the processing works technically, from splitting through embeddings to reranking the hits, is explained in the knowledge article Retrieval-Augmented Generation. For troubleshooting, all that counts at each stage is what goes wrong there and how you notice it.

1. Intake. Documents, tickets and records are collected, together with their version and read permissions. Typical faults are an incomplete collection, outdated versions and scans without a text layer. You notice it when answers cite a superseded version or certain sets of files never show up.

2. Preparation. Texts are split into sections that can be found individually. If splitting tears a table apart or separates a heading from its content, retrieval finds fragments without context. The symptom is an answer that gives a number without its unit or condition.

3. Index. The sections are stored in searchable form, as a keyword index, a vector index or both. Faults arise when changed documents are not re-indexed or permissions are lost on import. The assistant then cites documents that have been deleted or that the person asking may not open.

4. Retrieval. For each question the system looks for matching sections and passes a limited number to the model. If the correct passage sits too far down, or the question uses different words from the document, it never arrives. The symptom is a generic, half-right answer although the knowledge is in the collection.

5. Model. The language model writes an answer from the question, the instruction and the sections passed on. It can weight sections wrongly, drop conditions or fill gaps with plausible general knowledge. You recognise this case when the correct passage was demonstrably passed on and the answer still deviates.

6. Check. A check stage compares claims with their evidence, verifies format and permissions and, when in doubt, holds an answer back. If it is missing or set too leniently, unsupported claims get through. If it is too strict, users conspicuously often receive no answer at all.

7. Output. The answer reaches a target system, such as an interface, a ticket or a record. Footnotes get lost there, numbers are reformatted or fields truncated. The symptom is an answer that was correct in the log and arrives wrong for the user.

For every question, a language model reads only what is in its context window: the instruction, the question and the sections passed on. It is like a colleague who is handed four pages from a binder: he can write well, but only about those four pages. How many sections are passed on is a setting, and every setting cuts the result list off somewhere.

On top of that comes a vocabulary problem. The technician writes "leak rate", the manual speaks of "leakage", and a pure keyword search sees two different terms. Vector search and hybrid methods soften this, yet with technical terms, type designations and compound words the gap remains a frequent reason for misses. Which methods exist and how they can be combined is described in the knowledge article Semantic search in the enterprise.

Both effects are quiet. The answer reads fluently because the model makes something fitting out of the sections it received, and nobody sees the result list in which the correct passage sits just below the cut-off.

The demonstration works with thirteen invented sections from a maintenance manual, a standard extract, a data sheet and maintenance logs. The technician's question is preset, and scoring uses a simple keyword search that weights rarer words more heavily. Watch the highlighted row with manual section 4.5 Leakage and the indicator of whether it reaches the model.

Set the context window to six sections and check whether the manual section arrives. Then switch back to four sections and turn on the synonym list.

Even at six sections, section 4.5 Leakage does not arrive: a few short maintenance-log entries and the standard extract crowd it out, because they repeat the same query words in far less text. Only the synonym list changes that. It reads "leak" as "leaks" and "leakage" too, and the section jumps straight to rank 1, from rank 7 of 13.

The two corrections come at different prices. More sections in the context window make every request longer and give the model more material to weight wrongly. A synonym list acts precisely but has to be maintained by someone who knows both the language of the documents and the language of the people asking.

Also try the question "When is the pump leaking?" Without the synonym list the correct section already lands at rank 2, and with it at rank 1, with the standard extract's definition of tightness directly behind it at rank 2. That is a good result, because the two passages together make up the complete answer.

The demonstration has deliberately tight limits. It calculates without embeddings, without reranking and without a language model, and its collection is small enough for every row to be traced. The ranks are results of the demonstration, not measurements from a system, yet the pattern behind them holds in large collections too: a correct answer requires the correct passage to arrive.

Individual complaints show that something goes wrong, but not how often. For that you need a fixed collection of real questions, each noted in advance with the source passage a correct answer would have to cite. A few dozen such everyday questions are enough for a first baseline.

The test set separates two questions that a complaint mixes together. The first is whether the correct passage arrived in the context window; if not, the fault lies in stages 1 to 4. The second is whether the answer was written correctly and with evidence from the passage that arrived; if not, it lies in the model, the instruction or the check.

After every change to splitting, index, retrieval, model or instruction, the same test set runs again. That makes visible whether a change helped or made something worse elsewhere. How answers can be tied to sources and assessed, including human spot checks, is covered in the knowledge article Checking AI answers. Why this evaluation is an ongoing cost item and not a project phase is set out in What a RAG system really costs to run.

For a single complaint a fixed order pays off, because each step rules out whole stages. Starting with the model means checking first the place where the fault is least often visible.

1. Record the question and the expected source passage. Without the passage a correct answer would have to cite, no fault can be demonstrated.
2. Check whether the passage is in the index, in its valid version and with the right permissions. If it is missing, the cause lies in stages 1 to 3.
3. Look at the rank of the passage in the result list. If it sits beyond the context window cut-off, the cause lies in stage 4.
4. If the passage was passed on, compare instruction and answer. If the answer still deviates, the cause lies in stage 5.
5. Finally check the check stage and the output, that is, whether a supported answer was held back, shortened or reformatted.

This requires a retrieval log that records, for every answer, what was found, what was passed on and what was used. Without that log only guessing remains, and troubleshooting ends up at the model again.

Not every question from operations needs seven stages. If a question can be phrased as a fixed query, such as a date or a master data value, a database is more precise and cheaper than any language model.

Two further cases fail before the technology does. If three versions of the same manual are in circulation, the first step is deciding which one applies. And without a person who adds new versions and withdraws old ones, the answers go stale without anyone noticing.

If a single error triggers payments, safety risks or downtime, a person belongs at the decision. Whether pump P-40 starts up again is the technician's call, and the assistant gives him the source passage for it. How well your own collection is prepared for such a system can be assessed with the RAG readiness check, and the building blocks an implementation involves are described on the page AI development.