Mean reciprocal rank
MRR
The average, over queries, of 1 divided by the rank of the first relevant result, counting 0 when none is found.
Background conceptGlossary entry
On this page
Mean reciprocal rank (MRR) scores a ranking by where the first relevant result appears. For each query take 1 / rank of that result, or 0 if none appears in the top k, then average across queries.
| Query | First relevant rank | Reciprocal rank |
|---|---|---|
| “Are we on track to beat our FY2027 earnings?” | 1 | 1.000 |
| “How far above our margin target are we?” | 3 | 0.333 |
| “When does the Larkspur acquisition close?” | none | 0 |
MRR = (1 + 0.333 + 0) / 3 = 0.444.
The curve is steep at the top: rank 1 is worth 1, rank 2 is worth 0.5, rank 3 is worth 0.33, and rank 10 is worth 0.1. Moving the first good result from rank 2 to rank 1 (+0.5) gains more than moving it from rank 10 to rank 2 (+0.4). That matches how people read search results, and how a model weighs the first passage in its context.
The gotcha
MRR looks only at the first relevant result. A question that needs two passages scores 1.0 if the first one is at rank 1, even when the second never appears. DocSlurp’s retrieval scorer inverts this for multi-premise questions: its reciprocalRank is 1 / the worst premise rank, and 0 if any premise is missing (core/scripts/retrieval-quality/eval.mjs). Premises at ranks 2 and 5 give 0.2, not 0.5. For graded relevance across the whole list, use nDCG.
In DocSlurp
Background concept
DocSlurp’s retrieval scorer reports reciprocalRank as 1 / the worst premise rank, so a multi-premise question scores by its last-found piece, not its first (core/scripts/retrieval-quality/eval.mjs).