Skip to content

LearnEvaluate

Faithfulness

Groundedness · Answer faithfulness

The share of claims in a generated answer that the retrieved passages actually support. An answer can be true and still unfaithful.

Background conceptIllustrated explainer

On this page
Checking each claim of an answer against the retrieved passagesAn answer to "How fast can my assistant get the Learjet to London?" is split into three claims. Claim 1, "The Learjet is wheels-up within three hours of the request", cites passage 1, charter-agreement.pdf page 3, which supports it. Claim 2, "The flight to London takes about 7h 10m", cites passage 1, which does not say this; passage 2, travel-desk-guide.pdf page 12, does support it. Claim 3, "No approval is needed for a London trip", has no citation and is contradicted by passage 3, travel-desk-guide.pdf page 7: "The CFO pre-approves every trip over $40k, including all London charters." Faithfulness is 2 supported claims out of 3, or 0.67. Citation accuracy is separate: 1 of 2 citations points to a supporting passage, 0.50.Answer, split into claimsRetrieved passagesclaim 1The Learjet is wheels-up within threehours of the request. [1]✓ supportedclaim 2The flight to London takes about7h 10m. [1]✓ supported, but by [2], not the cited [1]claim 3No approval is needed for aLondon trip.✕ contradicted by [3][1] charter-agreement.pdf · p.3The Learjet 75 is ready for wheels-upwithin three hours of the assistant'srequest.[2] travel-desk-guide.pdf · p.12Teterboro to Farnborough takes about7h 10m, with catering on board.[3] travel-desk-guide.pdf · p.7The CFO pre-approves every trip over$40k, including all London charters.supportssupportscites [1]contradictsfaithfulness = 2 supported / 3 claims = 0.67citation accuracy, a separate number = 1 of 2 citations points at support = 0.50
Faithfulness asks, claim by claim, whether some passage says it. Claim 2 is faithful even though its citation points at the wrong passage; claim 3 might even be true in practice, but the travel desk guide says the opposite, so it fails. A sentence with a citation marker is not automatically supported, and a supported sentence is not automatically cited correctly.

The problem: a fluent answer with a bad claim inside

The chat answers “How fast can my assistant get the Learjet to London?” with:

The Learjet is wheels-up within three hours of the request [1]. The flight to London takes about 7h 10m [1]. No approval is needed for a London trip.

It reads well, it has citations, and two of its three sentences are right. The third is wrong in a way that matters: travel-desk-guide.pdf p. 7 says “The CFO pre-approves every trip over $40k, including all London charters.” An assistant who trusts this answer books the jet and then waits on a sign-off nobody requested.

Grading the whole answer as “good” or “bad” hides this. Faithfulness grades it claim by claim.

The definition

Faithfulness (also called groundedness) measures how much of an answer the retrieved context supports:

faithfulness = claims supported by the retrieved passages / total claims

Computing it takes three steps:

  1. Split the answer into claims. Each claim is one checkable statement. “The flight to London takes about 7h 10m” is one; a compound sentence may hold two.
  2. Check each claim against the passages. Is there a passage that states or directly implies it? The usual verdicts are supported, not supported (nothing says it), and contradicted (a passage says the opposite).
  3. Count. In the figure, claim 1 is supported by the charter agreement, claim 2 by the travel desk guide’s routing page, and claim 3 is contradicted by its approval page. Faithfulness = 2 / 3 = 0.67.

RAGAS, a common open-source framework, implements this with a language model doing both the splitting and the checking. Other approaches use natural-language-inference models, which classify each (passage, claim) pair as entailment, neutral or contradiction.

Faithful is not the same as true

Faithfulness checks the answer against the retrieved context, not against the world. That is deliberate. A RAG system’s job is to answer from your documents, and a claim the documents don’t support is a hallucination in that setting, even if it happens to be correct.

Suppose the CFO waived pre-approval for London trips last week. Claim 3 would be true, and still unfaithful: nothing the model was given says so. The right fix is to update the travel desk guide, not to reward the model for guessing. Maynez et al. (2020) drew the same line for summarization, separating unfaithful content (not supported by the source) from non-factual content (false); some unsupported additions they found were true, and they still counted as hallucinations.

The reverse also holds. A faithful answer can be false if the passage is outdated or wrong. Faithfulness tells you the model did its job; whether the documents are right is a separate question.

Faithfulness versus citation accuracy

Look at claim 2 again. It is supported, but by passage [2], while the answer cites [1]. That claim is faithful and its citation is wrong. A reader who clicks [1] finds a page about jet readiness and may conclude the answer made something up.

These are two measurements:

  • Faithfulness: does some retrieved passage support the claim?
  • Citation accuracy: does the cited passage support it?

The figure scores them separately: faithfulness 0.67, citation accuracy 0.50 (one of two citations points at support). A system can score well on one and badly on the other, and users experience both. See citation.

Where it breaks

  • Claim splitting is a judgment call. Split coarsely and one bad clause sinks a long, mostly correct claim; split finely and trivial claims (“this is about the jet”) inflate the score.
  • The checker is a model too. An LLM judge can accept paraphrases that change meaning (“within three hours” versus “in about three hours”) or reject valid inferences. Check its verdicts against human labels before trusting the number.
  • It rewards saying less. An answer that only repeats one supported sentence scores 1.0. Pair faithfulness with answer relevance or completeness, or a system can win by being unhelpful.
  • It can’t see what retrieval missed. If the approval page was never retrieved, claim 3 is merely “not supported”, not “contradicted”, and the score can’t tell a careless model from a missing passage. Look at recall@k too.

When to use it

Measure faithfulness on every RAG change that touches prompts, models or context assembly, using a fixed set of questions. Track contradicted claims separately from unsupported ones; contradictions are rarer and much more damaging. For high-stakes answers, a runtime check can flag unsupported claims before the user sees them, at the cost of extra model calls.

DocSlurp does not score faithfulness at answer time. Its chat responses attach evidence spans that link each cited sentence to the text of the passage it cites (core/server/src/services/project-chat.ts), which makes manual checking quick but does not verify support. Offline, its agent scorer checks citation structure: that every cited marker resolves to a real document page and that each claim cites exactly its own sources (evals/langfuse/scorers.json).

In DocSlurp

Background concept

DocSlurp does not compute a faithfulness score at answer time. Chat responses link each cited sentence to a source span (buildProjectChatEvidenceSpans in core/server/src/services/project-chat.ts), and the offline agent scorer checks that each claim cites exactly its own sources (claimAttribution in evals/langfuse/scorers.json).

Sources

Jump to