Knowledge: Answer Quality
Checking AI answers: hallucinations, source grounding and evaluation
An AI assistant that answers fluently looks competent, even when it is wrong. The error types that occur, how a system technically grounds answers in sources, and how test questions, metrics, human spot checks and monitoring in operation combine into a reliable picture of answer quality are all covered here.
Short answer
AI answers are checked in three layers: technically, by having the system bind every statement to a checked source and verify citations in code; through measurement, with real test questions and metrics such as faithfulness to source; and humanly, with expert spot checks. No single layer is enough alone, because each one reveals errors the others miss.
Definition
Hallucination (artificial intelligence): A hallucination is an output from a language model that is phrased convincingly but is wrong or unsupported: an invented detail, a false connection, or a statement that contradicts the context it was given. It occurs because the model generates likely continuations instead of looking up facts. Grounding in sources reduces how often it happens; measurement makes it visible.
In the glossary: Hallucination, Gold-standard test set, Evaluation, Benchmark, Retrieval-augmented generation, Retrieval, Reranking, Temperature, Human in the loop, Observability, Large language model
01
What kinds of errors do AI systems make in their answers?
AI systems make two basic kinds of errors. Factuality errors contradict reality or invent details. Faithfulness errors deviate from the instruction, from the supplied context, or from the model's own reasoning. In systems with document retrieval, errors upstream of the model are added: wrong, outdated or missing sources. Each kind needs its own way of checking.
The term hallucination covers very different kinds of error. The survey by Huang and others (ACM Transactions on Information Systems, preprint 2023) separates factuality hallucination, which contradicts verifiable knowledge, from faithfulness hallucination, which does not match the instruction or the context. For a system that answers from an organisation's own documents, the second family usually matters most: does the statement actually appear in the approved source?
| Error type | Example | Check |
|---|---|---|
| Invented detail | A deadline that appears in no document | Citation check, faithfulness to source |
| Contradicts the context | The answer says four weeks, the passage says two | Faithfulness to source, per statement |
| Deviates from the instruction | Running text instead of the requested list | Format check in code |
| Outdated source | A superseded version gets cited | Context precision, a validity filter |
| Missing source | No answer, even though the store contains one | Context recall against known passages |
The National Institute of Standards and Technology calls the phenomenon confabulation in its Generative AI Profile (NIST AI 600-1, 2024): content presented with confidence but wrong, including content that contradicts the input. The user-facing view is covered in AI chatbot for organisations; the measurement goes a level deeper from here.
02
Why do language models invent answers in the first place?
Language models generate the statistically most likely continuation of a text, not a checked statement. When the knowledge is missing, a fluent sentence still comes out. Kalai and others (2025) also argue that training and common evaluations reward guessing and penalise admitting uncertainty. An enterprise system must therefore explicitly treat "no answer found" as a good outcome.
NIST AI 600-1 (2024) calls confabulation a natural consequence of how generative models are built: they predict the next word, and nothing in that prediction distinguishes a correct sentence from a merely plausible one. Kalai, Nachum, Vempala and Zhang (arXiv, 2025) add that most evaluations count only correct answers. A model that guesses scores better than one that admits uncertainty.
03
How do you technically bind AI answers to sources and enforce citations?
Source grounding means the model only receives retrieved passages with an identifier, and must assign every statement to one. This is not enforced by the instruction alone, but by code: the answer comes back structured, the system checks that every cited identifier exists and that the quotation appears verbatim in the passage, and it discards unsupported answers.
How a system retrieves passages and hands them over with a citation requirement is explained in Retrieval-augmented generation explained. What matters here is the step after that: the instruction "support every statement" is only a request. It only becomes binding once code checks the output, a technique known as grounding.
Liu, Zhang and Liang (Findings of EMNLP 2023) measure citation recall, the share of statements fully supported, and citation precision, the share of citations that support their statement. Gao and others (EMNLP 2023) assess citation quality in the ALCE benchmark. Both measures carry over to any system of your own.
Require a structured answer
Statements, with the identifiers of the supporting passages and a short verbatim quote for each, plus a field for whether the documents actually answer the question.
Check the identifiers
Every identifier must belong to a passage retrieved for this particular query.
Search for the quote
The short quote must appear verbatim in the passage. This catches invented citations, but not distorted statements.
Assess coverage
Whether the statement actually follows from the passage is checked by an evaluation model, using the faithfulness principle, initially on a sample.
Make failure visible
If a check fails, the system either shows only the passages found or states openly that no supported answer is available, and logs the reason.
04
How do you build a set of test questions for an AI system?
A set of test questions is built from real queries, such as tickets, emails and search logs, supplemented with deliberately difficult cases. Every question needs an expected key point, its source passage, and a question group. Questions with no answer in the store belong in there deliberately. A held-out portion shows whether improvements only fit familiar questions.
The gold-standard test set answers one question: is the new version better or worse than the old one, on real questions? Only with it does evaluation become a repeatable measurement.
Checklist
What a usable set of test questions contains
Tick what your set covers. The list stores nothing.
How large the test set needs to be is shown by a rough approximation for confidence intervals of proportions (NIST/SEMATECH e-Handbook of Statistical Methods): with 100 questions and 90 percent correct answers, the uncertainty is around plus or minus 5.9 percentage points (1.96 times the square root of 0.9 times 0.1 divided by 100), and around 2.9 with 400 questions. A three-point difference is therefore not demonstrable with 100 questions. For small samples, the handbook recommends more precise methods such as the Wilson interval. What matters more than the total count is that every question group stays separately assessable.
05
Which metrics measure the quality of RAG answers?
The Ragas paper (Es et al., 2023) defines three measures that need no reference answer: faithfulness, answer relevance and context relevance. The open-source library of the same name adds context precision and context recall, which do need a reference. There are also citation precision, fact-level precision on individual statements, and plain correctness against the expected key point.
The paper by Es, James, Espinosa-Anke and Schockaert (arXiv, 2023) computes each measure with the help of a language model. Faithfulness: the answer is broken into statements, each is checked against the context, and the score is the share of statements that are supported. Answer relevance: questions generated from the answer are compared with the original question by embedding. Context relevance: the share of context sentences that are actually needed to answer.
Context precision and context recall come from the Ragas library's documentation: one measures whether relevant passages sit near the top of the results list, the other whether everything needed was retrieved, which requires a reference. In the paper, faithfulness agreed with human judgements in 95 percent of cases, answer relevance in 78, and context relevance in 70 percent (Es et al., 2023).
Metrics compared
| Criterion | Measures | Needs a reference | Misses |
|---|---|---|---|
| Faithfulness | Statements that follow from the context | No | A faithful rendering of a wrong source |
| Answer relevance | Whether the answer addresses the question | No | Factual errors in an on-topic answer |
| Context precision | Relevant passages near the top of the list | Depends on the variant | Whether the answer ends up correct |
| Context recall | Whether everything needed was retrieved | Yes | Excess context |
| Fact precision (FActScore) | Supported atomic facts | A knowledge source | Missing important statements |
| Correctness against the key point | Expected statements present and correct | Yes | Coincidentally correct, unsupported statements |
FActScore (Min et al., EMNLP 2023) breaks long texts into atomic facts; a widely used commercial chat model scored only 58 percent on biographies. The metrics complement each other: faithfulness is worthless if retrieval delivers the wrong version.
06
Can a language model reliably judge the answers of another language model?
Partly. Zheng and others (NeurIPS 2023) found over 80 percent agreement between strong models and human preferences, but also biases: favouring one position, longer answers, and their own outputs. A language model as judge scales up checking, but needs fixed scoring rules, swapped orderings, and calibration against human judgements.
Almost all automatic metrics use a language model as a judge (LLM-as-a-judge). Zheng and others identify position bias (the answer shown first is favoured), length bias, and self-preference.
- Narrow questions instead of a grade: "Does statement 3 follow from passage D17-4?" instead of one overall score.
- Swap the order: comparisons run twice with the position swapped; diverging verdicts count as a tie.
- Calibrate: people score a subset; only once agreement is sufficient does the judge model's score count.
- Without a reference: SelfCheckGPT (Manakul et al., EMNLP 2023) treats contradictions between several answers to the same question as a warning sign, but does not replace checking against sources.
If the judge model is swapped, every score shifts; measurement series stay comparable only with a fixed scoring prompt. The ongoing effort involved is set out in What a RAG system costs to run.
07
How do you organise a human spot check of AI answers?
A workable sample mixes randomly drawn answers with deliberately chosen ones: negative feedback, unsupported answers, and low automatic scores. Domain experts score against a fixed scheme, without seeing the automatic score. Part of the sample is checked twice, to measure agreement between reviewers. Every confirmed error becomes a new test question.
Draw the sample
Random everyday answers, plus deliberately chosen negative feedback, rejected citations and low faithfulness scores.
Score against a scheme
Correct and supported, correct but unsupported, partly wrong, wrong, or rightly or wrongly no answer given.
Score blind
The automatic score stays hidden, otherwise it gets adopted unconsciously.
Check in duplicate
Part of the sample goes to two people; the agreement, measured for example with Cohen's kappa, shows whether the scheme is unambiguous.
Feed it back
Confirmed errors are logged with their cause (retrieval, source, model, instruction) and turned into a test question.
Checking should be done by the domain experts responsible for the sources, and they see only what they are also allowed to see in the source system (Access control in AI knowledge systems). The principle is described in the glossary entry Human in the loop.
08
How do you monitor answer quality in ongoing operation?
In operation, what matters are signals that need no reference answer: the share of answers with no source, failed citation checks, automatic faithfulness scoring on a sample, feedback, and repeated questions. Every change to the model, the instructions, chunking or the document store triggers a full test-set run before it applies to all users.
- 01QueryPassages and answer
- 02Automatic checkCitations, format, no source
- 03Judge modelFaithfulness on a sample
- 04Human spot checkScheme, blind
- 05Test setNew cases, run before release
This requires that it stays traceable, for every answer, which passages were retrieved and cited (observability), without storing question content unnecessarily. What is informative is change over time: if the share with no source rises after new documents are added, the cause is more likely in ingestion than in the model.
Decision path
May a change go into operation?
All questions and results as a list
- Has the full test set run against the new version, including the held-out portion?
- Yes, continue with: Are correctness and faithfulness at least at the previous level in every question group?
- No, Result: Measure first
- Are correctness and faithfulness at least at the previous level in every question group?
- Yes, continue with: Does the system still answer "not covered" for questions with no answer in the store?
- No, Result: Clarify the regression
- Does the system still answer "not covered" for questions with no answer in the store?
- Yes, Result: Release with monitoring
- No, Result: Investigate guessing
- Result: Measure firstWorth discussing: how the test set runs automatically before every change, and who reads the result.
- Result: Clarify the regressionWorth discussing: which group got worse, and whether the difference is larger than the measurement uncertainty.
- Result: Investigate guessingWorth discussing: thresholds for "no answer", and whether the model fills gaps with general knowledge.
- Result: Release with monitoringWorth discussing: which domain experts review the strongest deviations, and how a rollback would work.
09
What framework does the NIST AI Risk Management Framework offer for checking AI answers?
The AI Risk Management Framework from the United States National Institute of Standards and Technology is a voluntary framework with four functions: Govern, Map, Measure and Manage. Measuring and monitoring in operation belong to Measure. The Generative AI Profile, NIST AI 600-1, lists invented content as its own risk, under the name confabulation.
Version 1.0 (NIST AI 100-1) was released for voluntary use in January 2023. Relevant here are Measure 2.4, monitoring function and behaviour in production, and Measure 2.5, demonstrating validity and reliability along with documented limits (NIST, 2023). The Generative AI Profile followed in July 2024.
| Function | In practice, for answer quality |
|---|---|
| Govern | Ownership of sources, the test set and approvals |
| Map | Which questions the system should answer, and which error causes which harm |
| Measure | Test set, metrics, judge model, spot checks, operational signals |
| Manage | Prioritising findings, maintaining sources, rolling back changes |
The framework does not provide a legal assessment, and none is offered here either. How measurement and upkeep fit into a knowledge system is described in Knowledge management with AI.
10
Which misconceptions about hallucinations and their measurement are common?
Five assumptions are common: that a cited source proves the answer, that document retrieval eliminates hallucination, that a high metric means correct answers, that a judge model replaces people, and that a temperature of zero stops invention. Each contains a grain of truth, and taken uncritically, each leads to systems that look better than they are.
Five assumptions, fairly tested
No
Only 74.5 percent of citations actually supported their sentence (Liu, Zhang, Liang, 2023). A citation proves that a passage exists, not that it carries the statement.
It reduces them, it does not remove them
Legal research tools with document retrieval hallucinated in 17 to 33 percent of cases, less often than a general chat model (Magesh et al., 2024).
Only if the source is correct
A faithful rendering of an outdated work instruction still scores perfectly. Correctness needs test questions with a confirmed key point.
No
Judge models show position, length and self-preference bias (Zheng et al., 2023), and are only as good as their calibration against human judgements.
No
Temperature controls the randomness of word choice. At zero, the most likely continuation is chosen, and that can still be wrong.
With agents, errors in intermediate steps are added on top, see AI agents in knowledge work.
Read more on iiterate.de
Foundations in the Knowledge section
- Knowledge Knowledge management with AI Measurement and upkeep across the lifecycle.
- Knowledge Retrieval-augmented generation explained The pipeline behind the answers.
- Knowledge Semantic search in the enterprise Measuring the relevance of retrieval.
- Knowledge Access control in AI knowledge systems Rights in the index and the logs.
- Knowledge AI agents in knowledge work Failure modes of multi-step agents.
Signals for further reading
- Signal What a RAG system costs to run Evaluation as an ongoing cost.
- Signal No-code agent builders for mid-sized companies Missing evaluation as a limit.
- Signal GPT-6 Astra versus Claude Fable 5.1 Why benchmark tables often do not hold up.
- Signal Context engineering Context poisoning from the wrong passages.
Implementation, tools and terms
- Consulting AI chatbot for organisations Sources, handover and measuring success.
- Consulting RAG implementation Implementing RAG on company data.
- Consulting AI security Security tests alongside quality checking.
- Tool RAG readiness check A self-test for a retrieval system.
- Glossary Gold-standard test set in the glossary A short definition of the test set.
Frequently asked questions
How many test questions does a first AI knowledge system need?
There is no fixed number. With 100 questions and 90 percent correct answers, the measurement uncertainty is roughly six percentage points, and just under three with 400 questions. A few dozen carefully confirmed questions per question group is often enough to start, growing with every finding from operation.
What good is faithfulness to source if the source itself is wrong?
None, in that case, and that is by design. The metric only checks whether the model renders the passage correctly. Whether the passage itself is right is shown by test questions with a confirmed key point and validity metadata in the index. If such cases pile up, maintaining the store matters more than any technical change.
Should an AI assistant show how confident it is?
A confidence statement formulated by the model itself is not a measured probability, and can make wrong answers more convincing. Verifiable signals are more useful: a source for every statement, a clear indication when no supported answer is available, and a simple way to report a wrong answer.
Can answers be checked when there is no reference answer?
Partly. Faithfulness, answer relevance and context relevance from the Ragas paper need no reference, because they check the answer, the question and the context against each other. SelfCheckGPT treats contradictions between several answers as a warning sign. But without a reference, only a human check shows whether an answer is factually correct.
How do you check answers that an AI agent produces over several steps?
Alongside the final answer, the intermediate steps are assessed: whether the agent called suitable tools with sensible inputs, correctly carried over intermediate results, and stopped at approval points. An error in the third step can easily produce a well-phrased, wrong final answer. More on this in AI agents in knowledge work.
Does checking answer quality replace a security test?
No. Quality checking asks whether answers are correct and supported under normal use. A security test asks whether the system can be misused through crafted inputs or documents. Both use fixed test cases that run on every change, but entirely different ones. Attack scenarios are covered in AI security.
Read on
Related topics
Sources
- 01 A Survey on Hallucination in Large Language Models (Huang et al.), ACM TOIS arXiv, ACM, 2023-11-09 · arxiv.org
- 02 Ragas: Automated Evaluation of Retrieval Augmented Generation (Es et al.) arXiv, 2023-09-26 · arxiv.org
- 03 Context Precision, Ragas documentation Ragas, 2026 · docs.ragas.io
- 04 Context Recall, Ragas documentation Ragas, 2026 · docs.ragas.io
- 05 Evaluating Verifiability in Generative Search Engines (Liu, Zhang, Liang) arXiv, Findings of EMNLP, 2023-04-19 · arxiv.org
- 06 Enabling Large Language Models to Generate Text with Citations (Gao et al.) arXiv, EMNLP, 2023-05-24 · arxiv.org
- 07 FActScore (Min et al.) arXiv, EMNLP, 2023-05-23 · arxiv.org
- 08 Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al.) arXiv, NeurIPS, 2023-06-09 · arxiv.org
- 09 SelfCheckGPT (Manakul, Liusie, Gales) arXiv, EMNLP, 2023-03-15 · arxiv.org
- 10 Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools (Magesh et al.) arXiv, 2024-05-30 · arxiv.org
- 11 Why Language Models Hallucinate (Kalai et al.) arXiv, 2025-09-04 · arxiv.org
- 12 AI Risk Management Framework 1.0, NIST AI 100-1 NIST, 2023-01-26 · nvlpubs.nist.gov
- 13 AI RMF Core NIST AIRC, 2023 · airc.nist.gov
- 14 Generative Artificial Intelligence Profile, NIST AI 600-1 NIST, 2024-07-26 · nvlpubs.nist.gov
- 15 NIST/SEMATECH e-Handbook, Confidence intervals NIST, 2012 · itl.nist.gov
Let us talk about your project
Whether it is a prototype, an internal tool or an AI application: describe briefly what you are building or want to take into production.