Prompt caching architecture for Claude applications

1. The problem caching solves

A multi-turn conversation or agent loop resends a lot of unchanged content on every call. The system prompt (instructions, persona, operating rules) is usually identical across calls in a session. Tool definitions, every callable function with its full schema, must be present on every turn whether or not that turn uses them, and they don't change mid-session. A large reference document, a long set of few-shot examples, a big block of retrieved context: all of it can be the same bytes call after call, for a whole session or across many sessions with the same setup.

Without caching, the API can't know this content is unchanged. It reprocesses the full input on every request, paying full cost and latency for content that adds nothing new. Prompt caching lets the provider recognize a previously seen prefix and skip reprocessing it, so the unchanged portion is much cheaper and faster, while the new part (the latest user message, the newest tool result) is processed normally. The saving scales with how much of the prompt is stable and how often that stable part is reused. That's why caching rewards a specific prompt structure.

The mechanism shapes the design rules below. Caching matches an exact, stable prefix of the request. It doesn't find similar content elsewhere in the prompt and doesn't understand meaning. A cached entry persists for a limited window and is refreshed each time it's reused within that window. Content reused repeatedly inside the window stays cheap; content cached once and then left alone expires and must be fully reprocessed next time.

2. The architectural implication: what goes where

Because caching matches a stable prefix, prompt structure becomes a performance concern. Content that's static across many calls (system instructions, tool definitions, a large reference document) goes early in the request and stays byte-for-byte identical, so the provider recognizes it as a prefix it has seen. Content that changes every turn (the latest user message, the most recent tool result) goes after it, because caching only helps what precedes the point of change. Volatile content placed early breaks the cache for everything after it.

This catches teams off guard: you can't add caching to an existing prompt with a flag, because most prompts weren't written with this ordering in mind. A prompt that interleaves stable and changing content (a system prompt with today's date near the top, a tool list filtered per user) has no long stable prefix to cache, however the request is marked. To get the benefit, change how the prompt is assembled. Separate content that never changes for this deployment, content that changes on some cadence, and content that changes every call, and render them in that order, consistently.

A rough ordering that tends to cache well, from most stable to least:

Tool definitions (fixed for the deployment) → system instructions (fixed, or changed rarely and deliberately) → long-lived reference material or few-shot examples (fixed per use case) → conversation history accumulated so far (grows, but each prior turn stays unchanged once written) → the newest user message or tool result (different on every call, and belongs at the end).

The application doesn't track a cache or manage cache keys; the provider does the prefix matching. The application's job is more basic: render the stable parts the same way, in the same order, every time, and put the changing parts at the end. Get the order wrong and no cache configuration recovers the benefit, because there's no stable prefix to match.

3. The real tradeoff: what counts as "static enough"

The hard part is usually spotting the one place a mostly static system prompt varies. Take a fixed system prompt with the user's name interpolated into the opening line ("You are a helpful assistant. You are currently helping Priya with..."). It looks static, but the name makes every user's prompt bytes different, so nothing caches across users, even though almost all of it (the instructions) is identical for everyone. The same failure comes from a timestamp in a header, a session ID threaded into the instructions, or a feature flag that toggles a sentence by cohort.

Keep personalizing; it's often useful. Separate the personalized part from the rest. Put the static instructions in one uninterrupted block and the personalized detail after it, as a short addendum at the end of the system content or, better, in the first user turn. The cache then covers the whole static block for every user, and only the small personalized fragment is processed fresh. That beats losing the cache on everything because ten words at the top varied.

This is prompt design discipline more than a caching trick. A prompt that keeps "true for everyone, always" separate from "true for this call" is easier to reason about and test, and it caches well as a side effect. A prompt built as one long string with variables spliced in wherever convenient is harder to reason about and defeats caching, for the same reason: when the two are tangled line by line, neither a reader nor the cache can tell where static ends and variable begins.

4. The practical benefit for agentic loops specifically

Caching helps any application that reuses a stable prefix, and an agentic loop is close to the ideal case. A single call (a one-off summarization, a one-time classification) sends its prompt once, with nothing to reuse it against. An agent loop sends the same system prompt and full tool inventory on every iteration, potentially dozens of times per task, as the agent reads a file, calls a tool, reads the result, picks the next action, and calls the model again. That's the "static content reused many times within one task" pattern caching is built for. The number of reuses determines how much it saves.

The saving compounds as the loop runs longer. Each iteration's request includes the fixed system prompt and tools plus the full history so far: every previous tool call and result. If that history is rendered consistently and appended to, never rewritten, each prior turn stays part of a stable, growing prefix the next call reuses, and only the newest turn is new. A twenty-iteration loop built this way pays full processing for each new increment of history once, instead of reprocessing the whole growing history twenty times.

The previous section's discipline matters most here, because agent loops are where dynamic content sneaks into "static" sections. A loop that injects a running token count, a live timestamp, or an adjusted instruction into the system prompt on every iteration, to keep the model aware of changing state, breaks its own caching on every turn. If state must travel with every call, append it to the message history at the end of the request. Keep the system prompt fixed for the whole task.

5. The limits of caching: it's not a universal free optimization

Caching has limits. Knowing them stops a team from marking everything cacheable and wondering why the bill didn't move.

First, a cached entry has a finite freshness window, and content must be reused within it to pay off. Content called once and not again until long after the window lapses gets nothing; it's reprocessed fresh anyway, and writing it to the cache may cost slightly more than not caching it. Caching bets that the same content will be reused soon. A workload of isolated, widely spaced, one-off calls, the opposite of an agent loop, never collects.

Second, caching only helps the stable portion before the point of change. A prompt that's mostly unique on every call (a different long document pasted in each time, wrapped in a short fixed instruction) has little to gain however it's structured, because there's little reusable content. Caching amplifies the repetition a workload already has; it can't create repetition.

The pattern that benefits: high repetition, short gaps. Agent loops (many calls per task, seconds apart), a chat session with a long, fixed system prompt (many turns, short gaps), or a service that fields many concurrent requests all sharing one large fixed instruction set and tool list: the same stable prefix, reused constantly, by one session or across many simultaneous ones.

The pattern that doesn't: low repetition, long gaps. A nightly batch job that processes each record with a unique document, once, and moves on. A rarely-used feature that calls the model a handful of times a day with hours between calls. In both cases there's either nothing stable to cache, or too much time between reuses for the cached entry to still be there when it would help.

Evaluate caching against the shape of a specific workload: how much of the prompt is stable, and how often that part is reused. For workloads that fit, it's one of the more reliable free wins. For workloads that don't, it does nothing.

6. Worked example: a prompt that defeats caching vs. one designed for it

The two versions of this agent's prompt carry the same instructions and the same live state. They differ only in where the dynamic content sits.

Version A — defeats caching (dynamic content interleaved throughout):

  system = f"""
  You are a support agent. The current time is {datetime.now()}.
  Session ID: {uuid4()}.
  User's plan tier: {user.plan_tier}.

  Follow these rules when answering:
  1. Always check order status before refunding.
  2. Never share another customer's data.
  3. Escalate anything involving a legal threat.
  ... (2,000 more words of otherwise-fixed policy) ...

  Today's promotional note: {get_current_promo_banner()}.
  """

Every one of those interpolated values sits inside the instructions rather
than after them, so the entire 2,000-word policy block — the one part that
never actually changes — never gets to be a stable prefix. It's preceded and
interrupted by content that's different on every single call.

Version B — designed to cache well (static block first, dynamic content
isolated at the end):

  STATIC_POLICY = """
  You are a support agent.

  Follow these rules when answering:
  1. Always check order status before refunding.
  2. Never share another customer's data.
  3. Escalate anything involving a legal threat.
  ... (2,000 more words of otherwise-fixed policy) ...
  """  # identical across every call — this is the cacheable prefix

  system = STATIC_POLICY  # unchanged, byte-for-byte, call after call

  messages = [
      # per-call context goes in the first user turn instead of the
      # system prompt, after the stable block, not woven into it
      {"role": "user", "content":
          f"[context: plan={user.plan_tier}, promo={get_current_promo_banner()}] "
          f"{actual_user_message}"},
      ...
  ]

The plan tier and promo banner still reach the model on every call. Version B keeps the 2,000-word policy block identical and uninterrupted across requests, so it becomes a stable prefix the provider can skip reprocessing, and the small per-call content moves to the end, where it can't contaminate anything upstream. Version A pays full processing on the whole prompt on every call, indefinitely, because the timestamp and session ID at the top guarantee no two calls share a cacheable prefix. The fix costs no capability, only some restructuring of how the prompt is assembled.

The same discipline applies to the context an agent accumulates as it runs: what stays in the history and what gets summarized or dropped as a session grows. For that side, see Context engineering: what actually goes into the context window, which covers the token-budget and relevance questions that pair with caching's stable-prefix discipline.