Skip to content

LearnGenerate & answer

Abstention

Answering "I don’t know"

A grounded system declining to answer because the retrieved evidence does not support one, instead of guessing.

In DocSlurpGlossary entry

On this page

Abstention is when a question-answering system says it can’t answer instead of producing one. In a grounded system the test is simple: if the retrieved passages don’t contain the answer, the right output is “the documents don’t say”, not a plausible guess.

Ask the travel corpus “Does the charter cover an onward leg from London to Paris?” If charter-agreement.pdf covers the Learjet 75 and its 3-hour readiness window but never mentions onward legs, a well-behaved system replies that the agreement doesn’t address it, and cites what it does say. A badly behaved one invents an onward-leg clause, because a language model is trained to continue text, and “I don’t know” is rarely the most likely continuation.

There are two places to abstain:

  • Before generation. If retrieval returns nothing above a relevance floor, skip the model entirely. This is cheap and cannot hallucinate, but it depends on the floor being set well.
  • During generation. Instruct the model to say when the excerpts are insufficient. This catches near misses retrieval let through, but it is a request the model may ignore.

The gotcha

Abstention is easy to over-reward. A system that refuses half its questions never hallucinates, and is useless. Score correct answers, wrong answers and abstentions separately, and keep unanswerable questions in your test set on purpose so you can tell a justified “I don’t know” from a retrieval miss. See faithfulness.

In DocSlurp

In DocSlurp

When the relevance floor removes every passage, the direct chat path returns a fixed "I couldn’t find anything relevant" answer without calling a model (buildFallbackAnswer in core/server/src/services/project-chat.ts). Otherwise abstention is prompt guidance: "If the excerpts are insufficient, say so plainly" (core/server/src/services/chat-instructions.ts).

Sources

Jump to