Skip to content

LearnEvaluate

Relevance judgments

Qrels · Relevance labels

Labels saying which documents or passages are relevant to each test query, and often how relevant, from which retrieval metrics are computed.

Background conceptGlossary entry

On this page

Relevance judgments are the answer key for retrieval evaluation: for each test query, which documents or passages are relevant, and often how relevant. In TREC tradition the file is called qrels (query relevance), one line per query and document:

q17  0  travel-desk-guide.pdf#p3   2
q17  0  charter-agreement.pdf#p2   1
q17  0  q3-earnings-memo.pdf#p2    0

The last column is the grade: here 2 = answers the question, 1 = useful context, 0 = judged not relevant. Binary labels feed recall@k and precision@k; graded labels feed nDCG.

Nobody can judge every passage for every query, so test collections judge a pool: the union of the top results from several systems. Anything outside the pool is unjudged.

The gotcha

Metrics treat unjudged as irrelevant. A new retriever that surfaces a good passage none of the pooled systems found gets no credit, and can score lower than the systems that built the pool. When a new method looks worse, read its top results before believing the number. Judges also disagree with each other: Voorhees (2000) found that substituting different assessors changed absolute scores while system rankings stayed fairly stable, so trust comparisons more than absolute values.

DocSlurp labels retrieval questions with premises rather than passage IDs: exact quotes plus the document and pages that support them (core/scripts/retrieval-quality/eval.mjs). Matching on quoted text rather than chunk IDs keeps the labels valid when chunking changes.

In DocSlurp

Background concept

DocSlurp’s retrieval datasets label each question with premises: exact quotes, a document and its supporting pages, plus provenance (core/scripts/retrieval-quality/eval.mjs). The eval sync accepts query files with optional qrels (core/scripts/eval-sync/README.md).

Sources

Jump to