Skip to content

LearnGenerate & answer

Prompt caching

Prefix caching

A provider feature that stores the processed state of a repeated prompt prefix, so later requests that start with the same tokens are cheaper and faster.

In DocSlurpGlossary entry

On this page

Prompt caching lets a model provider reuse work across requests that begin with the same tokens. Reading a prompt means computing internal state (the attention keys and values) for every token. If the next request starts with the identical prefix, the provider can load that state instead of recomputing it, and bills those cached tokens at a discount.

It pays off wherever a long prefix repeats: a fixed system prompt, tool definitions, a long document asked several questions, or an agent loop. An agentic RAG turn resends the whole conversation at every step, so step 4 starts with everything steps 1 to 3 sent.

An illustrative calculation, using the multipliers Anthropic documents (cache writes cost 1.25 times the base input price, cache reads 0.1 times): an agent turn sends a 3,000-token fixed prefix on each of 4 steps.

Prefix tokens billed, in base-price equivalents
No cache 4 × 3,000 = 12,000
Cache 3,000 × 1.25 + 3 × 3,000 × 0.1 = 3,750 + 900 = 4,650

Check current pricing before relying on these multipliers; they vary by provider and cache lifetime.

The gotcha

The cache matches an exact prefix, token by token. Put a timestamp, a user name or a per-turn count near the top of the prompt and every request misses. Order prompts from most stable to least stable. Some providers cache automatically; others cache only up to explicit cache_control breakpoints you place, and entries expire after minutes of disuse.

In DocSlurp

In DocSlurp

The chat agent keeps its fixed rules ahead of per-turn counts so the prefix stays stable, and adds a cache_control breakpoint on the latest message for Anthropic and Gemini models, which cache only at explicit markers (promptCacheBreakpoint in core/server/src/lib/chat-agent-model.ts, core/server/src/services/chat-mcp-agent.ts). Cached reads are counted per turn.

Sources

Jump to