Prompt Caching for LLM Apps: How Reusing a Prompt Prefix Cuts Cost and Delay, What Breaks It and What to Check for Privacy

Many AI apps send the same long block of text with every request: instructions, tool descriptions, policy documents. Prompt caching lets a provider reuse the work it already did on that repeated text. Done well, it lowers cost and shortens the wait for the first word of the answer. Done badly, it quietly does nothing. This article explains how prompt caching works, how three providers document it, what breaks it, and what it means for privacy.

A note on evidence. Provider documentation was read on 2026-11-03 and carries no publication date, so details such as model names, prices and limits can change. Treat the numbers below as documented at that time, and check the current pages before you budget.

What prompt caching is

A language model first reads your whole prompt, then writes an answer. Reading is called the prefill step, and its result is stored as internal data known as the key-value (KV) state. If the next request starts with exactly the same text, the provider can reuse that stored state instead of recomputing it.

OpenAI's guide describes this directly. It says caching stores the KV tensors, not the tokens themselves, and that a hit needs the whole rendered prefix to match up to a breakpoint (OpenAI prompt caching guide). Anthropic adds that matching is exact: segments must be 100% identical, and caching has no effect on output content (Anthropic prompt caching documentation).

Think of it like a bookmark. The model marks how far it read, and a new request that begins with the same pages skips ahead. Anything that differs before the bookmark forces a full re-read from that point.

How three providers document it

The three providers differ in how much you control. The facts below come from each provider's own page.

OpenAI

For its newest listed models, OpenAI documents a 1,024-token minimum for a cacheable prefix. Cache reads cost 0.1 times the ordinary input rate, and writes cost 1.25 times with the default 30-minute lifetime or 2.5 times with a 12-hour lifetime. Earlier models use different retention settings. You check hits in usage.input_tokens_details.cached_tokens (OpenAI guide).

Anthropic

Anthropic lets you mark where the cached prefix ends with cache_control, and allows up to 4 explicit breakpoints. The default lifetime is 5 minutes and refreshes on each use. A 1-hour option exists. Writes cost 1.25 times (5 minutes) or 2 times (1 hour) the base input price, and reads cost 0.1 times, with lower multipliers for some models. Minimum lengths vary by model, and shorter prompts are simply processed without caching and without an error (Anthropic documentation).

Google Gemini

Google documents implicit caching as on by default for Gemini 2.5 and newer models, with no setup. Minimum sizes listed are 2,048 tokens for Gemini 2.5 models and 4,096 for the newer models in the table. The page gives no lifetime and no discount percentage, so you cannot plan costs from it alone. Cache-hit tokens appear in usage.total_cached_tokens (Gemini caching documentation).

Prompt layout rules that earn cache hits

The core rule is simple: put what stays the same first and what changes last. Stable tool definitions, instructions and reference documents go at the top. The user's question, the date and any per-request data go at the end. Google gives the same advice: large common content first, and similar requests close together in time (Gemini documentation).

For agents, which call tools in a loop, the details matter more. A research paper tested caching on 500+ agent sessions with 10,000-token system prompts across OpenAI, Anthropic and Google. The authors report cost reductions of 41–80% and time-to-first-token improvements of 13–31%. They also report that careful control of what gets cached beat naive full-context caching, which could even increase latency (Lumer et al., arXiv 2601.06007). These are the authors' measurements on one benchmark, not a guarantee for your app.

The paper's practical advice is to keep dynamic content, such as changing tool lists and tool results, out of the cached part. That fits the provider rules above.

What breaks the cache

Because matching is exact, small edits cause misses. Anthropic's page lists several. Changing tool definitions (names, descriptions or parameters) invalidates the tools, system and message levels. Changing tool_choice or adding or removing images invalidates the messages level. Changes to thinking settings also invalidate cached messages. A change at one level invalidates that level and everything after it (Anthropic documentation).

Other common causes are less obvious, and the next three are my own explanation rather than documented lists:

A timestamp or request ID placed at the top of the system prompt.
Tool lists that reorder or change between requests.
Gaps longer than the cache lifetime, so the stored entry expires.

Short prompts also miss. Below the model's minimum length, nothing is cached, and with Anthropic you get no error to warn you.

Privacy and security questions

Caching stores something about your prompt, even if briefly. Provider documentation says the following:

OpenAI states that caches are not shared across organizations, that cached KV tensors may be held in GPU-local storage for at most 24 hours, that caching is compatible with Zero Data Retention (ZDR), and that manual cache clearing is not available (OpenAI guide).
Anthropic states that caches are isolated between organizations, and on some platforms also between workspaces. For eligible models, KV representations and hashes are held in memory for the lifetime and then deleted promptly (Anthropic documentation).

There is also a research warning. A cache hit is faster than a miss, and that timing difference can leak information. Researchers from Stanford audited commercial APIs and report finding global cache sharing across users at seven providers, including OpenAI. In a shared cache, an attacker could infer whether someone recently sent a particular prefix (Gu et al., arXiv 2502.07776). The paper was submitted in February 2025, so provider behavior may have changed since; the current provider statements above are the better guide today.

What this means in practice: keep personal data and secrets out of the stable prefix when you can, since the prefix is what gets stored and reused. Do not share one cache namespace across customers if your provider offers separation, such as OpenAI's optional prompt_cache_key for separate cache accounting, for example per customer. Confirm your contract's retention terms.

A worked example

Illustrative scenario, not a test. A support bot sends a fixed 6,000-token prefix (policy text and tool descriptions) plus a 200-token customer question. Assume Anthropic-style multipliers from the documentation: writes at 1.25 times and reads at 0.1 times the base input price, with a 5-minute lifetime. I count cost in "units," where one uncached input token costs 1 unit, and I ignore output tokens.

Baseline without caching. Input: 6,200 tokens per request. The provider processes all of it, costing 6,200 units. Ten requests cost 62,000 units. This is the number to beat.
Reorder the prompt. Input: your current prompt, with a timestamp at the top. Move the timestamp and the question to the end, behind the fixed prefix. Result: the first 6,000 tokens are now identical on every request. Why it matters: exact matching only works on an unchanged beginning.
Mark the breakpoint. Input: the fixed prefix of 6,000 tokens, which is above the model's minimum length. Add cache_control after the prefix. Result: the first request writes the cache.
Cost the first request. Write: 6,000 × 1.25 = 7,500 units. Add the 200-token question at 200 units. Total: 7,700 units. This is more than the baseline, which is the price of creating the cache.
Cost the next nine requests. If each arrives within 5 minutes of the last, each read costs 6,000 × 0.1 = 600 units, plus 200 for the question: 800 units. Nine requests cost 7,200 units.
Compare. Total with caching: 7,700 + 7,200 = 14,900 units, against 62,000 without. The saving here comes from the assumed multipliers and traffic pattern. Slower traffic or a changed prefix would raise it.
Verify with usage fields. Read cache_creation_input_tokens and cache_read_input_tokens on each response. On request 1 you expect about 6,000 created. On later requests you expect about 6,000 read. If both are zero, the prompt was not cached, so check the minimum length and the breakpoint position. If reads stay at zero while writes repeat, something in the prefix is changing, so diff two requests.

Common mistakes

Putting variable text first. A date or user name at the top changes the prefix every time. Move it to the end.
Assuming caching is on. Anthropic needs cache_control; the others cache automatically only above their size minimums. Read the usage fields instead of guessing.
Ignoring the lifetime. A 5-minute entry expires during quiet periods, and a write costs more than a normal read. For sparse traffic, compare the longer lifetime's higher write price before choosing it.
Changing tools mid-session. Editing tool definitions resets the cache for everything after them. Keep the tool list fixed and let the model choose which to use.
Treating a cache as private storage. Do not place secrets in the stable prefix, and read your provider's retention and isolation terms.
Quoting savings from a blog post. Prices and multipliers differ by model and change over time. Recompute with the current pricing page.

Recap

Prompt caching reuses the stored processing of an identical prompt beginning, so repeated long prefixes cost less and start answering sooner.
Put stable content first and changing content last, because matching is exact.
Providers differ in minimum length, lifetime and price, so plan from each provider's current documentation, not a general rule.
Track hit rates through the usage fields; a silent miss looks like normal operation.
Caching stores something derived from your prompt. Check isolation and retention terms, and keep sensitive data out of shared prefixes.

Research Sources

OpenAI: Prompt caching guide (documentation, read 2026-11-03, undated)
Anthropic: Prompt caching documentation (documentation, read 2026-11-03, undated; I read the first part of a long page)
Google: Gemini caching documentation (documentation, read 2026-11-03, undated)
Gu et al., Auditing Prompt Caching in Language Model APIs, arXiv 2502.07776 (submitted 11 February 2025, revised 13 July 2025, ICML 2025)
Lumer et al., Don't Break the Cache, arXiv 2601.06007 (submitted 9 January 2026, revised 31 January 2026)

Prompt Caching for LLM Apps: How Reusing a Prompt Prefix Cuts Cost and Delay, What Breaks It and What to Check for Privacy

Article index

What prompt caching is