Context window
Context length · Token budget
The maximum number of tokens a language model can handle in one call. Instructions, history, retrieved passages and the answer all share it.
In DocSlurpIllustrated explainer
On this page
The problem: the answer that stopped mid-sentence
You build a document chat. To be thorough, you retrieve 20 passages for every question and paste them all into the prompt. It works in testing. In production, long conversations start returning answers that end halfway through a sentence, and a few requests fail outright with “maximum context length exceeded”.
Nothing is wrong with retrieval. You ran out of room. A language model can only handle a fixed number of tokens per call, its context window, and everything competes for that space: the instructions, the conversation so far, the retrieved passages, the question, and the answer the model is about to write.
How it works: one budget, five claims on it
Think of the window as a tray of fixed length. Each part of the request takes a slice:
- System prompt: rules, citation format, grounding instructions. The same every turn.
- History: earlier questions and answers, so follow-ups like “and who approves it?” make sense.
- Retrieved passages: the evidence.
- The question.
- The answer: reserved up front, because the model writes into the same window. If the input fills it, there is no room left to reply.
A worked example with an illustrative 8,000-token window and passages of 400 tokens each:
| Slice | Tokens |
|---|---|
| System prompt | 500 |
| History | 1,500 |
| Reserved for the answer | 1,000 |
| Left for passages | 8,000 − 500 − 1,500 − 1,000 = 5,000 |
5,000 / 400 = 12.5, so 12 passages fit. Ask for 16 and four must go. The question takes a few tokens too; we round it into the history here to keep the arithmetic simple.
Real windows are much larger, commonly 128,000 tokens or more, but the arithmetic is the same and so is the trap. A token is roughly four characters of English prose. Code, tables, numbers and non-Latin scripts tokenize less efficiently, so a character-based estimate can undercount them badly.
Choosing what falls off
When the parts don’t fit, something gets cut. The worst policy is the accidental one: truncating the end of the prompt, which usually removes the question or the last passages without anyone deciding to. Choose deliberately:
- Drop the lowest-ranked passages first. Retrieval already ordered them; the 16th is the least likely to matter. This is why top-k and a relevance floor matter more than raw window size.
- Trim history from the oldest turn. Keep the last few exchanges verbatim; older turns rarely matter, or can be summarized into a line.
- Compress before you drop. Contextual compression keeps only the sentences of each passage that bear on the question, so more sources fit in the same space.
- Say what you left out. If a summary covers 40 of 50 documents because the rest didn’t fit, tell the reader. A silent cut looks like a complete answer.
Put the budget in code, not in hope: count or estimate tokens for each slice, reserve the answer first, then fill passages in rank order until the next one would overflow.
Where it breaks
- A bigger window is not a better answer. Models use information at the start and end of long inputs more reliably than the middle. See lost in the middle. Twelve relevant passages usually beat forty loosely related ones.
- Every token costs money and time. Providers bill input tokens on every call, and an agentic RAG loop resends the whole growing conversation at each step. Prompt caching helps with the stable prefix, not with fresh passages.
- Estimates drift. A 4-characters-per-token rule is fine for budgeting English prose and wrong for a spreadsheet dump. Leave a safety margin, or count with the model’s own tokenizer.
- The answer reserve is a cap. Reserve 1,000 tokens and a detailed answer that needs 1,400 gets cut off, usually flagged by a
finish_reasonoflength. Check that field instead of shipping half an answer.
When it matters
Always, but most when inputs vary in size: long conversations, documents that differ wildly in length, tables and code, and agent loops that accumulate tool results. Design the budget once, as context assembly: fixed slices for rules, history and the answer, and a ranked, trimmable slice for evidence.
In DocSlurp
In DocSlurp
DocSlurp budgets context explicitly. Chat loads the last 20 turns; the project-overview path stops at min(60,000 characters, 12,000 tokens × 4) and reports which summaries it left out (core/server/src/services/search.ts, core/server/src/services/project-chat.ts); structured extraction reads at most 100,000 characters of Markdown (core/worker/src/lib/trusted-llm-task.ts).