Skip to content

LearnEvaluate

Offline and online evaluation

A/B testing · Interleaving

Offline evaluation scores a system against fixed labeled data before release; online evaluation measures real users’ behavior, such as clicks or ratings, after release.

In DocSlurpGlossary entry

On this page

Offline evaluation runs a system against a fixed, labeled golden dataset and computes metrics like recall@k or faithfulness. It is repeatable, cheap, and safe: you can test a risky change without any user seeing it.

Online evaluation measures behavior in production: clicks on results, thumbs up or down on answers, follow-up questions, abandoned sessions. An A/B test sends a random share of traffic to the new version and compares outcomes. Interleaving merges results from two rankers into one list and credits whichever ranker’s results get clicked, which detects ranking differences with far less traffic than an A/B test.

A typical loop: a change must pass offline gates, then runs online, and online failures become new offline test cases.

The gotcha

Each one is blind where the other sees. Offline scores only cover questions someone thought to write down and label, and repeated tuning against them invites benchmark leakage. Online signals are noisy and biased: users click the first result partly because it is first, and a “thumbs down” can mean the answer was wrong or the documents were.

In DocSlurp

Offline evaluation is the established practice: development and held-out datasets with frozen, versioned scorers that sync to Langfuse (evals/langfuse/). Online signals are collected but not yet wired into experiments. Search sessions store an optional variant label and which result was clicked (core/server/src/routes/search.ts), and chat answers accept a rating and comment (core/server/src/services/chat-feedback.ts).

In DocSlurp

In DocSlurp

Offline: labeled retrieval and extraction datasets with frozen scorers (evals/langfuse/). Online signals exist without an experiment framework: search sessions record a variant label and clicked results (core/server/src/routes/search.ts), and chat answers take a thumbs up or down (core/server/src/services/chat-feedback.ts).

Sources

Jump to