Lecture: How Agent Caching Works — Mastering Stable Prefix Caching in Production
A 2,000+ word deep dive into KV-cache mechanics for multi-turn AI agents: how prefix matching works under the hood, why dynamic system prompts silently bust your cache, how to place breakpoints across tools, system prompts, and turn history, and how to verify cache hit rates in code.
In a single-turn chat completion, you pay for your system prompt and tool schemas once. In an agentic loop that runs 8 to 15 tool-calling turns to answer a single user request, you resend the entire system prompt, every tool schema, every reference document, and the growing transcript on every single turn. Without caching, a 10-turn agent run with a 12,000-token static prefix and 2,000 tokens of tool outputs per turn bills over 200,000 input tokens—and forces the model to recompute attention keys and values for the same prefix ten times in a row.
Stable prefix caching (also called prompt caching) changes the economics and latency of multi-turn agents more than almost any other engineering optimization. Done right, it cuts input token costs on cached prefixes by up to 90% and slashes time-to-first-token (TTFT) by 50%–80%. Done carelessly—by injecting a single changing timestamp or unsorted JSON dictionary near the top of your request—your cache hit rate drops to zero while you still pay cache write premiums.
In this lecture, we examine how transformer KV caching works under the hood, how provider prefix hierarchies evaluate your request byte-for-byte, the seven most common "cache busters" in agent harnesses, and how to design a rock-solid stable prefix architecture in Python.
1. What Prompt Caching Actually Caches: The Transformer KV Cache
To understand why prefix caching is strictly sequential—and why changing a single token at position 50 invalidates tokens 51 through 20,000—you have to look at the self-attention mechanism inside a causal transformer.
During the prefill phase (everything that happens before the first output token is emitted), the model projects each input token at each transformer layer into three vectors: a Query (Q), a Key (K), and a Value (V). Because causal attention masks future tokens, token i only attends to tokens 1..i. That means:
- The Key and Value tensors
(K_i, V_i)for tokenidepend exclusively on the exact token sequencet_1, t_2, ..., t_ithat came before it. - If two API requests share the exact same token sequence
t_1, t_2, ..., t_kfrom index1tok, their internal(K_1..k, V_1..k)tensors across every layer are mathematically identical. - Conversely, if token
t_10changes (for example, because you updatedcurrent_time: 19:42:01to19:42:09at the top of your system prompt), the attention context for tokent_11and every subsequent token changes. None of the downstream KV tensors can be reused.
This is why provider prompt caching is not a fuzzy semantic cache or a key-value database lookup. It is a cryptographic prefix hash over the exact tokenized sequence from the very first token of the request up to an explicit or automatic cache breakpoint.
2. The Prefix Evaluation Order: Tools → System → Messages
When you send a request to the Claude Messages API (or similar frontier model APIs), the provider serializes your structured JSON payload into a single contiguous token stream in a fixed canonical order:
toolsarray: Every tool'sname,description, andinput_schema, in the exact order they appear in the list.systemblocks: Your system instructions, reference documents, and persistent guidelines, in array order.messagesarray: Alternatinguserandassistantturns (includingtool_useandtool_resultblocks) from oldest to newest.
Because the serialized stream starts with tools, moves to system, and finishes with messages, each upstream section acts as a foundation for the next:
| If You Mutate... | What Stays Cached? | What Gets Invalidated? |
|---|---|---|
tools (add/remove/reorder a tool) |
Nothing (0% hit) | tools + system + all messages |
Top of system (inject dynamic timestamp/UUID) |
tools (if breakpoint set on tools) |
system + all messages |
Middle of messages (edit or drop turn 2) |
tools + system + turn 1 |
Turns 2..N |
Append to end of messages (normal agent loop) |
tools + system + turns 1..(N-1) |
Only the newly appended turn N |
Look at that first row: mutating your tools array mid-conversation invalidates the entire cache—including your system prompt and every prior turn. This is one of the most expensive hidden traps in agent design.
3. The 7 Silent Cache Busters in Agent Harnesses
Even experienced engineers accidentally sabotage prefix caching by introducing subtle non-determinism into their request builder. Audit your codebase for these seven patterns:
Cache Buster 1: Injecting datetime.now() into the System Prompt
Putting f"Current time: {datetime.now().isoformat()}" at the top of your system prompt guarantees a 100% cache miss on every turn that crosses a second boundary. If the model needs the current date or time, either clamp it to the calendar date (YYYY-MM-DD) at the very end of the system prompt after your static cache breakpoint, or pass the exact timestamp inside the user message metadata.
Cache Buster 2: Dynamic Tool Filtering Per Turn
Some frameworks try to "save tokens" by dynamically adding or removing tools on each turn based on a router classifier (e.g., passing 5 tools on turn 1, 8 tools on turn 2, and 3 tools on turn 3). Because tools sit at the very front of the KV prefix, changing the tool list forces a full cache recomputation of your system prompt and entire conversation history—costing far more than keeping a stable, sorted tool list across the entire session.
Cache Buster 3: Non-Deterministic Set or Dictionary Ordering
In Python, iterating over a set to build a list of allowed tools or concatenating JSON schemas generated from unordered sources can produce different key/item orderings across worker processes. Always sort tool definitions by name and serialize JSON schemas deterministically.
Cache Buster 4: Stuffing RAG Chunks into the System Prompt
If your RAG pipeline injects freshly retrieved document passages directly into the system prompt on every user query, the system prompt changes on every turn—invalidating the KV cache for all subsequent messages. Instead, keep the system prompt 100% static and inject query-time RAG passages inside the current user turn (or in a dedicated context block at the tail of the message list).
Cache Buster 5: Sliding-Window Truncation on Every Turn
Suppose your agent keeps a sliding window of the "last 10 messages." On turn 11, you drop message 1; on turn 12, you drop message 2. Because the start of the messages array shifts on every turn after turn 10, every single turn busts the message-history cache! Instead of dropping 1 turn at a time, let the history grow cleanly from a fixed anchor until occupancy reaches a threshold (e.g., 65% of the window), compact a large batch at once, and then let the new anchor stay stable for the next 20 turns.
Cache Buster 6: Mutating Old tool_result Payloads
Some harnesses retroactively compress or rewrite older tool_result blocks two turns after they were used. If you edit turn 3's tool_result on turn 5, you invalidate the cached KV states for turns 3, 4, and 5. Truncate oversized tool outputs before you append them the first time, or only compact when establishing a new rolling checkpoint.
Cache Buster 7: Ignoring Minimum Token Thresholds and TTLs
Model providers enforce minimum token counts for a cache breakpoint to activate—typically 1,024 tokens on standard models and 2,048 tokens on smaller/faster models—along with a sliding TTL (usually 5 minutes, refreshed on each hit, with optional 1-hour extended TTL tiers). Marking a 300-token system prompt with cache_control does nothing.
4. Designing a Multi-Breakpoint Stable Prefix in Python
In the Claude Messages API, you can place up to four explicit cache_control: {"type": "ephemeral"} breakpoints across a request. A clean production agent uses three progressive breakpoints so every layer of stability is captured:
- Breakpoint 1 — End of
tools: Locks in all tool schemas across every user and session. - Breakpoint 2 — End of Static
systemPrefix: Locks in your core system instructions, safety policies, and static domain reference guide across all users. - Breakpoint 3 — Rolling Conversation Checkpoint (Latest User/Tool Turn): Placed on the last content block of the newest turn so the next iteration of the agent loop reuses 100% of prior turns.
import copy
def build_cached_agent_request(
static_tools: list[dict],
static_system_rules: str,
user_memory_block: str,
messages: list[dict],
) -> dict:
"""Assemble a Messages API payload with stable prefix cache breakpoints."""
# 1. Deterministic tool order + Breakpoint #1 on the final tool
tools_payload = copy.deepcopy(sorted(static_tools, key=lambda t: t["name"]))
if tools_payload:
tools_payload[-1]["cache_control"] = {"type": "ephemeral"}
# 2. Split system into Static (Cached Breakpoint #2) + Per-User Memory
system_payload = [
{
"type": "text",
"text": static_system_rules,
"cache_control": {"type": "ephemeral"},
}
]
if user_memory_block.strip():
system_payload.append({
"type": "text",
"text": f"<user_memory>\n{user_memory_block.strip()}\n</user_memory>",
})
# 3. Clone messages, strip old message breakpoints (max 4 total allowed),
# and place Breakpoint #3 on the final block of the latest message.
msgs_payload = strip_message_cache_controls(copy.deepcopy(messages))
if msgs_payload:
mark_last_block_ephemeral(msgs_payload[-1])
return {
"tools": tools_payload,
"system": system_payload,
"messages": msgs_payload,
}Let's look closely at why strip_message_cache_controls is essential. Because the API allows at most 4 cache_control blocks across the entire request (and we already used 2 on tools and system), if we simply appended a new cache_control marker on every turn of a 7-turn agent loop without removing the marker from earlier turns, turn 4 would fail with a 400 error for exceeding 4 cache breakpoints! Moving the message breakpoint forward to the tail of the transcript on each turn keeps you within the 4-breakpoint limit while extending the cached KV prefix after every tool call.
def strip_message_cache_controls(messages: list[dict]) -> list[dict]:
for msg in messages:
content = msg.get("content")
if isinstance(content, list):
for block in content:
if isinstance(block, dict):
block.pop("cache_control", None)
return messages
def mark_last_block_ephemeral(message: dict) -> None:
content = message.get("content")
if isinstance(content, str):
message["content"] = [
{
"type": "text",
"text": content,
"cache_control": {"type": "ephemeral"},
}
]
elif isinstance(content, list) and content:
last_block = content[-1]
if isinstance(last_block, dict):
last_block["cache_control"] = {"type": "ephemeral"}5. Understanding the Math: Cache Write Premium vs. Cache Read Discount
Prompt caching is not free on the first turn. With standard ephemeral caching on Claude models:
- Cache Creation (
cache_creation_input_tokens): Billed at 1.25× the base input token price (a 25% write premium to store the KV states). - Cache Read (
cache_read_input_tokens): Billed at 0.10× the base input token price (a 90% discount on every subsequent hit). - Uncached Input (
input_tokens): Billed at 1.00× base price for the new tokens after the last cache hit breakpoint.
When does a cache breakpoint break even? After just one cache hit! On turn 1 (cache miss/write) + turn 2 (cache hit/read), your combined multiplier on the static prefix is 1.25 + 0.10 = 1.35×, compared to 2.00× without caching—an immediate 32.5% savings by turn 2, growing to 77.5% savings over a 10-turn agent loop.
| Agent Loop Turn | Prefix Tokens (Tools + System + Prior Turns) | New Turn Tokens | Uncached Cost Multiplier | Cached Cost Multiplier |
|---|---|---|---|---|
| Turn 1 (Initial Call) | 10,000 (Cache Write) | 500 | 10,500× base | 13,000× base (1.25× write) |
| Turn 2 (Tool Result 1) | 10,500 (Cache Hit) | 1,500 (Cache Write) | 12,000× base | 2,925× base |
| Turn 3 (Tool Result 2) | 12,000 (Cache Hit) | 1,500 (Cache Write) | 13,500× base | 3,075× base |
| Turn 4 (Tool Result 3) | 13,500 (Cache Hit) | 1,500 (Cache Write) | 15,000× base | 3,225× base |
| Turn 5 (Final Synthesis) | 15,000 (Cache Hit) | 400 | 15,400× base | 1,900× base |
| 5-Turn Total | — | — | 66,400× base | 24,125× base (63.7% saved) |
6. Instrumenting and Auditing Cache Hit Rates in Production
Never assume your cache breakpoints are hitting just because you added "cache_control": {"type": "ephemeral"} to your code. Always log the three input token counters from response.usage in your JSONL telemetry and compute the Effective Cache Hit Ratio on every turn after turn 1:
def compute_cache_telemetry(usage, pricing: dict) -> dict:
uncached_in = getattr(usage, "input_tokens", 0) or 0
cache_write = getattr(usage, "cache_creation_input_tokens", 0) or 0
cache_read = getattr(usage, "cache_read_input_tokens", 0) or 0
out_tokens = getattr(usage, "output_tokens", 0) or 0
total_in = uncached_in + cache_write + cache_read
hit_ratio = (cache_read / total_in) if total_in > 0 else 0.0
# Pricing in USD per 1M tokens
base_in_rate = pricing["input_per_mtok"] / 1_000_000
out_rate = pricing["output_per_mtok"] / 1_000_000
cost_usd = (
(uncached_in * base_in_rate)
+ (cache_write * base_in_rate * 1.25)
+ (cache_read * base_in_rate * 0.10)
+ (out_tokens * out_rate)
)
return {
"total_input_tokens": total_in,
"cache_read_tokens": cache_read,
"cache_write_tokens": cache_write,
"uncached_input_tokens": uncached_in,
"cache_hit_ratio": round(hit_ratio, 4),
"cost_usd": round(cost_usd, 6),
}In a healthy multi-turn agent loop, turn 1 should show cache_write_tokens > 0 (or cache_read_tokens > 0 if another user recently warmed the shared static tools + system prefix), and turns 2 through N should consistently show a cache_hit_ratio above 0.75–0.90. If you ever see cache_read_tokens == 0 and cache_write_tokens > 10000 on turn 2, turn 3, and turn 4 of the same run, you have a cache buster mutating your prefix.
7. Writing a Deterministic Prefix-Stability Unit Test
You don't need to spend a single cent on live API calls to test whether your harness preserves a stable prefix. Write a deterministic unit test that builds two consecutive turns of an agent run and asserts byte-for-byte equality on the serialized prefix:
import json
def test_agent_prefix_stability():
req_turn_1 = build_cached_agent_request(
static_tools=MY_TOOLS,
static_system_rules=SYSTEM_PROMPT,
user_memory_block="User prefers concise Python examples.",
messages=[{"role": "user", "content": "Find the retry policy in docs."}],
)
req_turn_2 = build_cached_agent_request(
static_tools=MY_TOOLS,
static_system_rules=SYSTEM_PROMPT,
user_memory_block="User prefers concise Python examples.",
messages=[
{"role": "user", "content": "Find the retry policy in docs."},
{"role": "assistant", "content": [{"type": "tool_use", "id": "t1", "name": "search_corpus", "input": {"query": "retry"}}]},
{"role": "user", "content": [{"type": "tool_result", "tool_use_id": "t1", "content": "Max 3 retries."}]},
],
)
# 1. Tools and System must be 100% identical byte-for-byte
assert json.dumps(req_turn_1["tools"], sort_keys=True) == json.dumps(req_turn_2["tools"], sort_keys=True)
assert json.dumps(req_turn_1["system"], sort_keys=True) == json.dumps(req_turn_2["system"], sort_keys=True)
# 2. Turn 1 message content (minus the moved cache_control marker) must match Turn 2's first message
m1_clean = strip_message_cache_controls(copy.deepcopy(req_turn_1["messages"]))[0]
m2_first = req_turn_2["messages"][0]
assert m1_clean == m2_first
# 3. Total cache_control markers in Turn 2 must never exceed 4
raw_json = json.dumps(req_turn_2)
assert raw_json.count('"cache_control"') <= 4Adding this 25-line test to CI guarantees that no future PR can accidentally slip a dynamic timestamp, an unsorted tool set, or a 5th cache_control breakpoint into your agent's request builder.
8. Subagent Delegation and Shared Prefix Warming
Stable prefix caching becomes even more powerful when your architecture spawns specialized subagents (for example, fanning out three parallel research workers to analyze three different competitors or filings). If all three subagents share the exact same tools array and static system instructions—and differ only in the task description passed in the first user message—then:
- Make one lightweight warm-up call (or let the first subagent start 1–2 seconds ahead of the batch) to write the shared
tools + systemprefix into the provider's KV cache. - Every parallel subagent that starts within the 5-minute TTL window immediately hits the cached prefix on turn 1, paying only
0.10×on the shared instructions and cutting cold-start latency across the entire fan-out batch.
By treating your prompt prefix as an immutable, versioned artifact—sorted tools first, static system rules second, per-user memory third, and append-only turn history fourth—you get fast, predictable agent loops at a fraction of uncached API costs.