Skip to content

LearnEvaluate

LLM-as-judge

LLM-as-a-judge · Model-graded evaluation

Using a language model to grade outputs, such as whether an answer is supported or relevant, in place of a human reviewer or an exact-match rule.

Background conceptGlossary entry

On this page

LLM-as-judge means asking a language model to grade another model’s output: “Is each claim in this answer supported by these passages?” or “Rate how directly this answer responds, 1 to 5.” It scales where human review can’t, and it handles open-ended answers that exact-match rules can’t score.

A faithfulness judge for the finance corpus might receive the answer “Q3 gross margin was 71%, six points above plan,” the passage from q3-earnings-memo.pdf p. 3, and an instruction to return {"supported": true|false, "reason": "..."} for each claim.

Known biases

The MT-Bench study (Zheng et al., 2023) found judges agree with humans about as often as humans agree with each other on its tasks, and also documented systematic biases:

  • Position bias: in pairwise comparisons, a preference for whichever answer comes first.
  • Verbosity bias: longer answers look more thorough.
  • Self-preference: a judge can favor output from its own model family.

Swapping answer order, using a different model family from the one being judged, and asking for a short reason before the verdict all help.

The gotcha

A judge is a model under test too. Before trusting its scores, have humans label a sample and measure agreement on your data. Pin the judge’s model, prompt and settings, because changing any of them changes the ruler. DocSlurp’s scorer registry requires exactly that for learned judges (model, prompt version, settings) and treats a changed scorer as a new version with new score names (core/scripts/eval-sync/README.md). The scorers registered today are all deterministic.

In DocSlurp

Background concept

Every scorer in DocSlurp’s registry is deterministic. A learned judge must record its model, prompt version and settings, and the eval sync never creates hosted judge evaluators (evals/langfuse/scorers.json, core/scripts/eval-sync/README.md).

Sources

Jump to