Faithfulness
Groundedness · Answer faithfulness
The share of claims in a generated answer that the retrieved passages actually support. An answer can be true and still unfaithful.
Background conceptIllustrated explainer
On this page
The problem: a fluent answer with a bad claim inside
The chat answers “How fast can my assistant get the Learjet to London?” with:
The Learjet is wheels-up within three hours of the request [1]. The flight to London takes about 7h 10m [1]. No approval is needed for a London trip.
It reads well, it has citations, and two of its three sentences are right. The third is wrong in a way that matters: travel-desk-guide.pdf p. 7 says “The CFO pre-approves every trip over $40k, including all London charters.” An assistant who trusts this answer books the jet and then waits on a sign-off nobody requested.
Grading the whole answer as “good” or “bad” hides this. Faithfulness grades it claim by claim.
The definition
Faithfulness (also called groundedness) measures how much of an answer the retrieved context supports:
faithfulness = claims supported by the retrieved passages / total claims
Computing it takes three steps:
- Split the answer into claims. Each claim is one checkable statement. “The flight to London takes about 7h 10m” is one; a compound sentence may hold two.
- Check each claim against the passages. Is there a passage that states or directly implies it? The usual verdicts are supported, not supported (nothing says it), and contradicted (a passage says the opposite).
- Count. In the figure, claim 1 is supported by the charter agreement, claim 2 by the travel desk guide’s routing page, and claim 3 is contradicted by its approval page. Faithfulness = 2 / 3 = 0.67.
RAGAS, a common open-source framework, implements this with a language model doing both the splitting and the checking. Other approaches use natural-language-inference models, which classify each (passage, claim) pair as entailment, neutral or contradiction.
Faithful is not the same as true
Faithfulness checks the answer against the retrieved context, not against the world. That is deliberate. A RAG system’s job is to answer from your documents, and a claim the documents don’t support is a hallucination in that setting, even if it happens to be correct.
Suppose the CFO waived pre-approval for London trips last week. Claim 3 would be true, and still unfaithful: nothing the model was given says so. The right fix is to update the travel desk guide, not to reward the model for guessing. Maynez et al. (2020) drew the same line for summarization, separating unfaithful content (not supported by the source) from non-factual content (false); some unsupported additions they found were true, and they still counted as hallucinations.
The reverse also holds. A faithful answer can be false if the passage is outdated or wrong. Faithfulness tells you the model did its job; whether the documents are right is a separate question.
Faithfulness versus citation accuracy
Look at claim 2 again. It is supported, but by passage [2], while the answer cites [1]. That claim is faithful and its citation is wrong. A reader who clicks [1] finds a page about jet readiness and may conclude the answer made something up.
These are two measurements:
- Faithfulness: does some retrieved passage support the claim?
- Citation accuracy: does the cited passage support it?
The figure scores them separately: faithfulness 0.67, citation accuracy 0.50 (one of two citations points at support). A system can score well on one and badly on the other, and users experience both. See citation.
Where it breaks
- Claim splitting is a judgment call. Split coarsely and one bad clause sinks a long, mostly correct claim; split finely and trivial claims (“this is about the jet”) inflate the score.
- The checker is a model too. An LLM judge can accept paraphrases that change meaning (“within three hours” versus “in about three hours”) or reject valid inferences. Check its verdicts against human labels before trusting the number.
- It rewards saying less. An answer that only repeats one supported sentence scores 1.0. Pair faithfulness with answer relevance or completeness, or a system can win by being unhelpful.
- It can’t see what retrieval missed. If the approval page was never retrieved, claim 3 is merely “not supported”, not “contradicted”, and the score can’t tell a careless model from a missing passage. Look at recall@k too.
When to use it
Measure faithfulness on every RAG change that touches prompts, models or context assembly, using a fixed set of questions. Track contradicted claims separately from unsupported ones; contradictions are rarer and much more damaging. For high-stakes answers, a runtime check can flag unsupported claims before the user sees them, at the cost of extra model calls.
DocSlurp does not score faithfulness at answer time. Its chat responses attach evidence spans that link each cited sentence to the text of the passage it cites (core/server/src/services/project-chat.ts), which makes manual checking quick but does not verify support. Offline, its agent scorer checks citation structure: that every cited marker resolves to a real document page and that each claim cites exactly its own sources (evals/langfuse/scorers.json).
In DocSlurp
Background concept
DocSlurp does not compute a faithfulness score at answer time. Chat responses link each cited sentence to a source span (buildProjectChatEvidenceSpans in core/server/src/services/project-chat.ts), and the offline agent scorer checks that each claim cites exactly its own sources (claimAttribution in evals/langfuse/scorers.json).