Skip to content

LearnGenerate & answer

Structured output

Structured extraction · Schema-guided extraction

Asking a model to return data in a fixed machine-readable shape, usually JSON matching a schema, instead of free text.

In DocSlurpIllustrated explainer

On this page
A page and a schema produce JSON whose fields point back to their source spansPage 2 of charter-agreement.pdf reads: "Northwind Air keeps the Learjet 75 ready within 3 hours of a booking request. Over 20 flight hours a quarter earns a 10% rebate. Catering and a ground car are confirmed within 30 minutes." A schema asks for operator, ready_hours, rebate_percent and confirm_minutes. The resulting JSON has operator "Northwind Air" at confidence 0.97, ready_hours 3 at 0.95, rebate_percent 10 at 0.90 and confirm_minutes 30 at 0.80, each wired to the highlighted span it came from. With an auto-accept threshold of 0.85, three fields pass and confirm_minutes goes to review, so the document needs review even though the mean confidence is 0.905.charter-agreement.pdf · p.2Northwind Air keeps theLearjet 75 ready within 3 hoursof a booking request.Over 20 flight hours a quarterearns a 10% rebate. Cateringand a ground car are confirmedwithin 30 minutes.OUTPUTSCHEMA · THE SHAPEoperator: stringready_hours: numberrebate_percent: numberconfirm_minutes: integerEXTRACTED · VALUE / CONFIDENCEoperator: "Northwind Air"0.97✓ready_hours: 30.95✓rebate_percent: 100.90✓confirm_minutes: 300.80!document: needs reviewmean 0.905each field keeps the span it came from
The schema fixes the shape; the evidence makes each value checkable. Every field carries the span it was read from, so a reviewer checks one highlight instead of rereading the contract. At a 0.85 threshold the mean confidence is 0.905, yet the document needs review: one field at 0.80 decides it. Move the threshold and watch the verdict follow the weakest field.

The problem: 400 contracts and a spreadsheet

The travel desk wants a table of every charter agreement: operator, how fast the aircraft is ready, loyalty rebate, confirmation time. You could ask a chat model “how fast is the jet ready under this agreement?” 400 times and paste the answers into a spreadsheet, but free text resists automation. One answer says “3 hours”, another “same morning”, a third adds a paragraph about crew rest. You need the same fields, in the same types, from every document.

Structured output means asking the model for data in a fixed shape, usually a JSON object matching a JSON Schema, instead of prose. Your code can then validate it, store it and compare it across documents.

How it works

1. Describe the fields. Write a schema with names, types, descriptions and constraints:

{
  "type": "object",
  "properties": {
    "operator": { "type": "string" },
    "ready_hours": { "type": "number", "minimum": 0, "maximum": 72 },
    "rebate_percent": { "type": "number" },
    "confirm_minutes": { "type": "integer", "description": "Catering and ground car confirmation time" }
  },
  "required": ["operator", "ready_hours"]
}

2. Make the model produce that shape. There are three levels of strictness:

  • Prompting: include the schema in the prompt and ask for JSON only. Usually works; sometimes returns extra prose, a trailing comma or a missing field.
  • JSON mode: the provider guarantees syntactically valid JSON, but not your fields.
  • Constrained decoding: the provider (or an inference server such as vLLM) compiles the schema into a grammar and, at each step, masks out every token that would break it. The output is guaranteed to parse and match the schema’s structure.

3. Validate anyway. Check required fields, types, enums, patterns and ranges in your own code. Providers’ constrained modes often support only part of JSON Schema, and prompted output can be anything.

4. Ask for the evidence. For every field, ask for the verbatim text it came from. Then map that text back to its page and position in the document. This is the step that turns structured output from “data” into “data you can check”.

A worked example

From charter-agreement.pdf, page 2:

Northwind Air keeps the Learjet 75 ready within 3 hours of a booking request. Over 20 flight hours a quarter earns a 10% rebate. Catering and a ground car are confirmed within 30 minutes.

The model returns each field with a value, a self-reported confidence and an evidence snippet:

Field Value Confidence Evidence
operator “Northwind Air” 0.97 “Northwind Air keeps”
ready_hours 3 0.95 “ready within 3 hours”
rebate_percent 10 0.90 “earns a 10% rebate”
confirm_minutes 30 0.80 “confirmed within 30 minutes”

Now apply a review rule: auto-accept a field only if it is present, passes validation, and has confidence of at least 0.85. Three fields pass. confirm_minutes at 0.80 does not, so the document as a whole goes to review. The average confidence is (0.97 + 0.95 + 0.90 + 0.80) / 4 = 0.905, comfortably above 0.85, yet the right decision is still “needs review”. Gate on the weakest field, not the average.

Because each value carries its evidence, the reviewer doesn’t reread the contract. They look at one highlighted span on page 2 and confirm “30 minutes” means confirmation, not delivery.

Where it breaks

  • Valid isn’t true. Constrained decoding guarantees ready_hours is a number between 0 and 72. It doesn’t stop 2 when the page says 3. Every format guarantee is silent about correctness.
  • Confidence is self-reported. A model’s confidence number is another generated token, not a calibrated probability. Use it to sort the review queue, and measure how well it predicts errors on labelled data before trusting a threshold.
  • Evidence can be invented too. A model can quote a snippet that isn’t in the document. Match each snippet against the source text; one that doesn’t match is a red flag, not provenance.
  • Absent is not the same as null. If the agreement has no termination clause, the right value is null with no evidence, not a guess. Make fields nullable and tell the model so, or it will fill them.
  • Constraints can hurt. Forcing a strict format can degrade reasoning on hard extractions. Keep schemas flat and descriptive, and give the model room, for example a short evidence field before the value.
  • Long documents. A 200-page contract may not fit the context window; text past the cut-off is never read, and fields found only there come back empty.

When to use it

Use structured output whenever a program, not a person, consumes the result: building tables from document sets, routing documents by type, filling forms, feeding metadata filters, or calling tools. Pair it with per-field provenance and a review rule, so the cheap fields flow straight through and the doubtful ones reach a human with the source already highlighted.

In DocSlurp

In DocSlurp

A project outputSchema produces one JSON artifact per document with a value, confidence, verbatim evidence and page and box provenance per field (core/worker/src/stages/structured-extract.ts). The schema is sent as response_format only to private model endpoints; hosted models get it in the prompt, and their output is parsed leniently, then validated. See structured extraction.

Sources

Jump to