Skip to content

LearnGenerate & answer

Context window

Context length · Token budget

The maximum number of tokens a language model can handle in one call. Instructions, history, retrieved passages and the answer all share it.

In DocSlurpIllustrated explainer

On this page
A context window as a fixed tray of tokensAn illustrative 8,000-token window holds a 500-token system prompt, 1,500 tokens of history, and 1,000 tokens reserved for the answer, leaving 5,000 tokens for retrieved passages of 400 tokens each. Twelve passages fit. With 16 passages requested, passages 13 to 16, the lowest ranked, fall off the tray. Without the answer reserve, 15 passages fit and no room is left for the model to reply.8,000-token window (illustrative)12 of 16 passages fit · 4 dropped · 200 tokens unused02k4k6k8kanswer12345678910111213141516system500history1,500retrieved passages, in rank order400 eachreserved1,000falls off: lowest ranked first
Every passage you add takes room from something else. With 500 tokens of rules, 1,500 of history and 1,000 reserved for the reply, 5,000 tokens remain: 12 passages of 400. Request more and the lowest-ranked fall off. Drop the answer reserve and three more passages squeeze in, leaving the model no room to write.

The problem: the answer that stopped mid-sentence

You build a document chat. To be thorough, you retrieve 20 passages for every question and paste them all into the prompt. It works in testing. In production, long conversations start returning answers that end halfway through a sentence, and a few requests fail outright with “maximum context length exceeded”.

Nothing is wrong with retrieval. You ran out of room. A language model can only handle a fixed number of tokens per call, its context window, and everything competes for that space: the instructions, the conversation so far, the retrieved passages, the question, and the answer the model is about to write.

How it works: one budget, five claims on it

Think of the window as a tray of fixed length. Each part of the request takes a slice:

  • System prompt: rules, citation format, grounding instructions. The same every turn.
  • History: earlier questions and answers, so follow-ups like “and who approves it?” make sense.
  • Retrieved passages: the evidence.
  • The question.
  • The answer: reserved up front, because the model writes into the same window. If the input fills it, there is no room left to reply.

A worked example with an illustrative 8,000-token window and passages of 400 tokens each:

Slice Tokens
System prompt 500
History 1,500
Reserved for the answer 1,000
Left for passages 8,000 − 500 − 1,500 − 1,000 = 5,000

5,000 / 400 = 12.5, so 12 passages fit. Ask for 16 and four must go. The question takes a few tokens too; we round it into the history here to keep the arithmetic simple.

Real windows are much larger, commonly 128,000 tokens or more, but the arithmetic is the same and so is the trap. A token is roughly four characters of English prose. Code, tables, numbers and non-Latin scripts tokenize less efficiently, so a character-based estimate can undercount them badly.

Choosing what falls off

When the parts don’t fit, something gets cut. The worst policy is the accidental one: truncating the end of the prompt, which usually removes the question or the last passages without anyone deciding to. Choose deliberately:

  • Drop the lowest-ranked passages first. Retrieval already ordered them; the 16th is the least likely to matter. This is why top-k and a relevance floor matter more than raw window size.
  • Trim history from the oldest turn. Keep the last few exchanges verbatim; older turns rarely matter, or can be summarized into a line.
  • Compress before you drop. Contextual compression keeps only the sentences of each passage that bear on the question, so more sources fit in the same space.
  • Say what you left out. If a summary covers 40 of 50 documents because the rest didn’t fit, tell the reader. A silent cut looks like a complete answer.

Put the budget in code, not in hope: count or estimate tokens for each slice, reserve the answer first, then fill passages in rank order until the next one would overflow.

Where it breaks

  • A bigger window is not a better answer. Models use information at the start and end of long inputs more reliably than the middle. See lost in the middle. Twelve relevant passages usually beat forty loosely related ones.
  • Every token costs money and time. Providers bill input tokens on every call, and an agentic RAG loop resends the whole growing conversation at each step. Prompt caching helps with the stable prefix, not with fresh passages.
  • Estimates drift. A 4-characters-per-token rule is fine for budgeting English prose and wrong for a spreadsheet dump. Leave a safety margin, or count with the model’s own tokenizer.
  • The answer reserve is a cap. Reserve 1,000 tokens and a detailed answer that needs 1,400 gets cut off, usually flagged by a finish_reason of length. Check that field instead of shipping half an answer.

When it matters

Always, but most when inputs vary in size: long conversations, documents that differ wildly in length, tables and code, and agent loops that accumulate tool results. Design the budget once, as context assembly: fixed slices for rules, history and the answer, and a ranked, trimmable slice for evidence.

In DocSlurp

In DocSlurp

DocSlurp budgets context explicitly. Chat loads the last 20 turns; the project-overview path stops at min(60,000 characters, 12,000 tokens × 4) and reports which summaries it left out (core/server/src/services/search.ts, core/server/src/services/project-chat.ts); structured extraction reads at most 100,000 characters of Markdown (core/worker/src/lib/trusted-llm-task.ts).

Sources

Jump to