Golden dataset
Golden set · Test collection · Eval set
A fixed set of questions with known correct answers or relevant passages, used to score a system the same way every time it changes.
Background conceptGlossary entry
On this page
A golden dataset is a fixed set of test questions, each paired with what a correct system should produce: the passages that answer it, the expected answer, or both. You run every version of your system against the same set, so a change in score means a change in the system, not in the test.
For a company corpus, one entry might be:
{
"question": "How far above our margin target are we?",
"premises": [
{ "document": "growth-plan-2027.pdf", "pages": [13],
"quote": "24% against a 20% target" }
]
}
With labels like this you can compute recall@k, nDCG and citation checks automatically on every change.
What makes one good
- Real questions. Pull them from logs or users, not from skimming the documents, which produces questions that copy the documents’ wording and make keyword search look great.
- Labels written first. Decide which pages answer each question before looking at results; otherwise the labels drift toward whatever the system already returns. See relevance judgments.
- Hard cases on purpose: questions with no answer in the corpus, answers spread across documents, scanned pages, tables.
- A held-out split that you score once per release and never tune on.
The gotcha
A golden set goes stale the moment you start optimizing against it. Every fix inspired by a failing case makes the score a little less honest. This is benchmark leakage in slow motion. DocSlurp keeps development and held-out splits as separate datasets, stores fixtures outside the repository with hash-checked labels, and blocks tuning tools from reading held-out data (core/docs/evals/langfuse-sync.md).
In DocSlurp
Background concept
DocSlurp keeps evaluation fixtures outside the repository, with hash-checked labels and provenance, split into development and held-out datasets that sync to its Langfuse instances (evals/langfuse/datasets/, core/docs/evals/langfuse-sync.md).