Hallucination
Confabulation · Unsupported claim
Generated text that reads fluently but is not supported by, or contradicts, the source material or the facts.
Background conceptIllustrated explainer
On this page
The problem: three sentences that look equally true
A document chat answers “Are we on track to beat our FY2027 earnings?” with:
FY2027 revenue is projected at $48.2M, up 31% [1]. Operating margin lands at 20% in FY2027 [2]. The Larkspur deal adds $5M in FY2027 revenue.
All three sentences are fluent, specific and confident. Now open the sources. growth-plan-2027.pdf p. 12 does say $48.2M, up 31%. Page 13 says operating margin lands at 24%, against a 20% target: the model blended two numbers from one sentence. And nothing retrieved mentions Larkspur revenue or $5M at all.
One sentence is right, one contradicts the page it cites, and one has no source. Nothing in the text tells you which is which. That is a hallucination: output that reads like a fact but isn’t supported by the evidence, or by reality.
Why models do it
A large language model generates the most plausible next token, one at a time. Plausible and true overlap most of the time, which is why models are useful. They come apart in predictable places:
- Numbers, dates and names. “20%” and “24%” are both plausible after “operating margin lands at”. Near-identical phrasings with different facts are exactly where next-token prediction slips.
- Gaps in the evidence. Asked about the Larkspur deal’s revenue with no passage on it, the model fills in what such documents usually say. Training rewards producing an answer, and confident guessing scores better than “I don’t know” on most benchmarks.
- Blending. One passage says a 24% projection and a 20% target; the summary fuses them.
- Over-generalizing. A passage about one office’s bookings becomes a statement about every market.
It helps to name two kinds:
| Kind | Meaning | In the example |
|---|---|---|
| Intrinsic | Contradicts the supplied source | “20%”, citing a page that says 24% |
| Extrinsic | Can’t be verified from the source at all | “adds $5M in FY2027 revenue” |
Extrinsic claims might even be true. In a grounded system they are still a defect, because the reader can’t check them and the system promised to answer from the documents.
How to reduce it
No setting removes hallucination. Each layer below narrows it.
- Ground the model. Supply the passages and instruct it to answer only from them. This is what RAG is for, and it removes the biggest source: answering from memory.
- Retrieve the right evidence. Many hallucinations are retrieval failures in disguise: the passage that would have answered never reached the prompt, so the model filled the gap.
- Make abstaining easy. Tell the model to say when the excerpts don’t cover the question, and skip generation entirely when retrieval finds nothing relevant. See abstention.
- Require citations per claim. A claim with no marker is immediately suspicious, and a marker tells the reader where to check.
- Check claims after generation. Compare each sentence with the passage it cites, by exact match for numbers and names, or with a natural-language-inference model or an LLM-as-judge for paraphrases. Flag or drop what fails.
Where it breaks
- Citations can launder errors. The “20% [2]” sentence is the dangerous one: it carries a citation, so readers trust it more than the uncited sentence, yet it contradicts its source. A marker shows where to look, not that the claim is right.
- Verification has its own error rate. A judge model can miss a subtle numeric change or reject a correct paraphrase. Spot-check the checker against human labels.
- Low temperature doesn’t fix it. Temperature 0 makes the same wrong answer repeatable. See temperature.
- “Grounded” can still mean wrong. If the source document itself is outdated, a perfectly faithful answer is still false. Faithfulness to the sources and truth in the world are different properties.
How to measure it
Count supported claims, not good-looking answers. For a sample of answers, split each into claims and label every claim as supported, contradicted or unsupported by the retrieved passages. The share of supported claims is the answer’s faithfulness. In the example above, 1 of 3 claims is supported. Track it over time and alongside retrieval metrics, so you can tell a generation problem from a retrieval one.
In DocSlurp
Background concept
DocSlurp reduces the risk with grounding rules, cited evidence and a no-model fallback when nothing relevant is retrieved (core/server/src/services/chat-instructions.ts, core/server/src/services/project-chat.ts). It does not check claims against sources at runtime; support is measured offline by the retrieval-quality harness (core/scripts/retrieval-quality/).