Answer relevance
Answer relevancy
Whether a generated answer addresses the question that was asked, regardless of whether its facts are true or supported.
Background conceptGlossary entry
On this page
Answer relevance asks one thing: does the answer address the question? It ignores whether the answer is true or supported. That is a separate measure, faithfulness.
Ask “How far above our margin target are we?” and get back “FY2027 revenue is projected at $48.2M, up 31% year over year [1].” Every word is supported by growth-plan-2027.pdf p. 12, so faithfulness is perfect. Relevance is poor: the question was about margin, not revenue. The retriever found a nearby passage, and the model answered what that passage could answer.
How it’s measured
There are two common approaches:
- Rubric grading. A human or an LLM judge rates how directly the answer responds, often on a 1 to 5 scale.
- Reverse questions. RAGAS asks a model to write the questions this answer would fit, embeds them, and averages their similarity to the real question. An answer about revenue generates “What is FY2027 revenue projected at?”, which sits far from the original.
The gotcha
Relevance rewards confident, on-topic answers, including wrong ones and including answers that should have been a refusal. “We are 9 points above our margin target” is perfectly relevant and false. A correct abstention (“the documents don’t say”) can score low on relevance even though it is the right behavior. Always read relevance next to faithfulness and an abstention check, never alone.
In DocSlurp
Background concept
DocSlurp does not score answer relevance at runtime. Its offline agent-capability scorer checks required and forbidden answer patterns, and its registry notes that semantic correctness needs separate review (evals/langfuse/scorers.json, core/scripts/multisearch-quality/score.mjs).