Skip to content

LearnGenerate & answer

Hallucination

Confabulation · Unsupported claim

Generated text that reads fluently but is not supported by, or contradicts, the source material or the facts.

Background conceptIllustrated explainer

On this page
A supported claim, a contradicted claim and an unsupported claimAn answer with three sentences next to the three retrieved passages. Sentence one, "FY2027 revenue is projected at $48.2M, up 31% [1]", is supported by growth-plan-2027.pdf page 12. Sentence two, "Operating margin lands at 20% in FY2027 [2]", cites growth-plan-2027.pdf page 13, which says 24% against a 20% target, so it contradicts its own citation. Sentence three, "The Larkspur deal adds $5M in FY2027 revenue", has no citation and no passage supports it.MODEL ANSWERRETRIEVED PASSAGES[1] growth-plan-2027.pdf · p.12FY2027 revenue is projected at$48.2M, up 31% year over year.[2] growth-plan-2027.pdf · p.13Operating margin lands at 24%,against a 20% target.[3] q3-earnings-memo.pdf · p.2Q3 gross margin came in at 71%,six points above plan.✓FY2027 revenue is projected at$48.2M, up 31% [1]supported: [1] says $48.2M, up 31%≠Operating margin lands at 20%in FY2027 [2].cited, but [2] says 24%?The Larkspur deal adds $5Min FY2027 revenue.no retrieved passage says this∅
Fluency doesn’t reveal which claims are true. All three sentences read with the same confidence. Checked against the passages, one is supported, one contradicts the page it cites (the 20% target became the projection), and one has no source at all. The cited error is the most dangerous, because the marker makes it look verified.

The problem: three sentences that look equally true

A document chat answers “Are we on track to beat our FY2027 earnings?” with:

FY2027 revenue is projected at $48.2M, up 31% [1]. Operating margin lands at 20% in FY2027 [2]. The Larkspur deal adds $5M in FY2027 revenue.

All three sentences are fluent, specific and confident. Now open the sources. growth-plan-2027.pdf p. 12 does say $48.2M, up 31%. Page 13 says operating margin lands at 24%, against a 20% target: the model blended two numbers from one sentence. And nothing retrieved mentions Larkspur revenue or $5M at all.

One sentence is right, one contradicts the page it cites, and one has no source. Nothing in the text tells you which is which. That is a hallucination: output that reads like a fact but isn’t supported by the evidence, or by reality.

Why models do it

A large language model generates the most plausible next token, one at a time. Plausible and true overlap most of the time, which is why models are useful. They come apart in predictable places:

  • Numbers, dates and names. “20%” and “24%” are both plausible after “operating margin lands at”. Near-identical phrasings with different facts are exactly where next-token prediction slips.
  • Gaps in the evidence. Asked about the Larkspur deal’s revenue with no passage on it, the model fills in what such documents usually say. Training rewards producing an answer, and confident guessing scores better than “I don’t know” on most benchmarks.
  • Blending. One passage says a 24% projection and a 20% target; the summary fuses them.
  • Over-generalizing. A passage about one office’s bookings becomes a statement about every market.

It helps to name two kinds:

Kind Meaning In the example
Intrinsic Contradicts the supplied source “20%”, citing a page that says 24%
Extrinsic Can’t be verified from the source at all “adds $5M in FY2027 revenue”

Extrinsic claims might even be true. In a grounded system they are still a defect, because the reader can’t check them and the system promised to answer from the documents.

How to reduce it

No setting removes hallucination. Each layer below narrows it.

  • Ground the model. Supply the passages and instruct it to answer only from them. This is what RAG is for, and it removes the biggest source: answering from memory.
  • Retrieve the right evidence. Many hallucinations are retrieval failures in disguise: the passage that would have answered never reached the prompt, so the model filled the gap.
  • Make abstaining easy. Tell the model to say when the excerpts don’t cover the question, and skip generation entirely when retrieval finds nothing relevant. See abstention.
  • Require citations per claim. A claim with no marker is immediately suspicious, and a marker tells the reader where to check.
  • Check claims after generation. Compare each sentence with the passage it cites, by exact match for numbers and names, or with a natural-language-inference model or an LLM-as-judge for paraphrases. Flag or drop what fails.

Where it breaks

  • Citations can launder errors. The “20% [2]” sentence is the dangerous one: it carries a citation, so readers trust it more than the uncited sentence, yet it contradicts its source. A marker shows where to look, not that the claim is right.
  • Verification has its own error rate. A judge model can miss a subtle numeric change or reject a correct paraphrase. Spot-check the checker against human labels.
  • Low temperature doesn’t fix it. Temperature 0 makes the same wrong answer repeatable. See temperature.
  • “Grounded” can still mean wrong. If the source document itself is outdated, a perfectly faithful answer is still false. Faithfulness to the sources and truth in the world are different properties.

How to measure it

Count supported claims, not good-looking answers. For a sample of answers, split each into claims and label every claim as supported, contradicted or unsupported by the retrieved passages. The share of supported claims is the answer’s faithfulness. In the example above, 1 of 3 claims is supported. Track it over time and alongside retrieval metrics, so you can tell a generation problem from a retrieval one.

In DocSlurp

Background concept

DocSlurp reduces the risk with grounding rules, cited evidence and a no-model fallback when nothing relevant is retrieved (core/server/src/services/chat-instructions.ts, core/server/src/services/project-chat.ts). It does not check claims against sources at runtime; support is measured offline by the retrieval-quality harness (core/scripts/retrieval-quality/).

Sources

Jump to