Caching Strategies for AI Agents

Here's a result that surprised even the engineers who caused it: a team turned on caching to cut their agent's latency, and the agent got slower. Not a little slower — measurably worse on the exact metric they were trying to improve. They'd done the obvious thing, wrapped the whole context in a cache and assumed a win, and the naive approach backfired. AI agent caching is one of the highest-return optimizations available, and one of the easiest to get wrong in a way that quietly costs you the benefit.

The fix wasn't turning caching off. It was understanding that caching has distinct layers doing distinct jobs, that only some of your bill is even cacheable, and that where you draw the cache boundary matters more than whether you cache at all. Here's what one team learned rebuilding it.

The Problem the Platform Engineer Faced

A platform engineer at a customer-support SaaS company was staring at an LLM bill that had tripled in a quarter as the agent scaled. Latency was creeping up too — the agent processed a large knowledge base and a long system prompt on every single request, re-reading the same tens of thousands of tokens for every user question. The waste was obvious. The same instructions, the same tools, the same reference docs, reprocessed from scratch thousands of times a day.

So the team reached for caching, and reached for it the way most teams do: they cached everything. Wrap the full context, flip the switch, wait for the savings. What they got was a modest cost improvement and, bafflingly, worse latency on a chunk of requests. The optimization that was supposed to make the agent faster and cheaper had made it faster and cheaper on paper and slower in practice for real users.

The engineer had treated caching as a single on/off feature. It isn't. It's three different mechanisms with three different cost profiles, and using them interchangeably is how a caching project produces a worse agent.

The Wrong Approach

The original attempt cached the entire request context indiscriminately and matched only on exact repeats.

[object Object], ,[object Object],(,[object Object],):
    key = ,[object Object],(system_prompt + kb + question)     ,[object Object],
    ,[object Object], cache.has(key):
        ,[object Object], cache.get(key)
    result = model.generate(system_prompt + kb + question)
    cache.,[object Object],(key, result)
    ,[object Object], result

What this does: it hashes the full prompt including the user's unique question and only returns a cached answer on a byte-for-byte identical request — which almost never happens, so the hit rate is near zero while every miss still reprocesses the entire knowledge base.

The design has two deep problems. First, keying on the whole request including the variable question means the cache almost never hits, because two users rarely phrase a question identically — the hit rate on natural language exact-match caching is low, often ten to fifteen percent even in friendly conditions. Second, and worse, wrapping the entire context in one undifferentiated cache boundary is exactly the pattern that can raise latency rather than lower it. When you cache indiscriminately, you pay cache-management overhead on parts of the request that were never going to repeat, and the machinery costs more than it saves.

⚠️ Common mistake: Treating caching as one switch you flip on the whole request. Caching everything by default is not the safe choice — it's the choice that produces near-zero hit rates and can make latency worse. Where you draw the cache boundary is the actual decision, and "everywhere" is usually the wrong answer.

The Correct Approach

The rebuild split caching into three layers, each matched to the part of the request it actually fits. A response cache for exact repeats, a semantic cache for paraphrased repeats, and a prompt cache for the stable prefix everything shares.

[object Object], ,[object Object],(,[object Object],):
    ,[object Object],
    ,[object Object], hit := response_cache.get(exact_key(question)):
        ,[object Object], hit
    ,[object Object],
    ,[object Object], hit := semantic_cache.get(embed(question), threshold=,[object Object],):
        ,[object Object], hit
    ,[object Object],
    answer = model.generate(
        stable_prefix=[system_prompt, kb],    ,[object Object],
        dynamic=question,
    )
    response_cache.,[object Object],(exact_key(question), answer)
    semantic_cache.,[object Object],(embed(question), answer)
    ,[object Object], answer

What this does: it checks a cheap exact-match cache first, then a semantic cache that recognizes paraphrases, and only on a real miss calls the model with the stable system prompt and knowledge base marked as a reusable cached prefix — so the huge unchanging context is processed once, not per request.

Results and What Changed

The three-layer approach did what the blanket cache couldn't. The prompt cache alone cut a large share of the cost, because the system prompt and knowledge base — the tens of thousands of tokens reprocessed every time — were now computed once and reused, with each request paying full price only for the small new question. Time to first token dropped sharply on cache hits, because the model no longer re-read the entire prefix before starting.

The semantic cache added a second win the exact-match cache never could. Common questions phrased differently — "how do I reset my password" and "I forgot my password, what now" — now hit the same cached answer instead of both calling the model. For a support agent where a handful of questions make up most of the volume, that bypassed the model entirely on a meaningful fraction of traffic.

But the engineer also learned the boundary of caching, and it reframed their savings estimate. Caching only ever discounts input tokens — the prompt side. It does nothing for output tokens, the words the model generates. So the "we'll cut the bill by 80%" projection was wrong from the start, because a big share of their bill was generation, which caching can't touch. The honest number was still excellent, just smaller than the fantasy, and knowing that stopped them from over-promising to their finance team.

⚡ Pro tip: Model your savings on input tokens only, never output. No caching scheme discounts the tokens the model generates. Any projection that applies a cache discount to your whole bill is overstating the win — separate input from output spend before you promise anyone a number.

⚡ Pro tip: Watch your semantic cache threshold like a hawk. Set the similarity threshold too loose and the cache returns a stored answer for a question that's only sort of similar, confidently serving the wrong response. A slightly-too-eager semantic cache is worse than no cache, because it fails silently — tune the threshold against real query pairs and err strict.

How to Apply This AI Agent Caching Setup to Your Situation

Start with the layer that fits your workload. If your agent carries a large stable prefix — a long system prompt, a big reference document, a fixed toolset — prompt caching is your biggest win, and it's the one to deploy first. Mark the stable part of your context as cacheable and keep the dynamic part outside that boundary.

Add a semantic cache when a small set of questions dominates your traffic. A support or FAQ-style agent where the top questions repeat constantly benefits enormously; a highly varied research agent where every query is novel benefits little, so measure your query distribution before investing.

Keep an exact-match response cache as the cheap floor. It's trivial to build on something like Redis with a TTL, its lookups are sub-millisecond, and while its hit rate is low on conversational traffic, it's excellent for templated or repeated structured queries and costs almost nothing to run.

⚡ Pro tip: On providers where cache writes cost extra, remember the break-even. Some prompt caches charge a premium to write the cache the first time, so a cached prefix only pays off after it's read a couple of times. For a prefix used once and discarded, caching can cost more than it saves — cache the things you'll reuse, not the things you touch once.

⚡ Pro tip: Cache tool outputs with a TTL, not just model calls. If your agent reads a file or calls an API whose result doesn't change every second, a short time-to-live on that tool's output skips redundant calls entirely. Tool caching is often the most overlooked layer and one of the cheapest to add.

One caution that applies across all three layers: decide how a cached answer gets invalidated before you ship, not after. A semantic cache that happily serves a customer last week's pricing after prices changed isn't saving money — it's generating wrong answers fast. Response and semantic caches need a clear expiry policy and a way to purge entries when the underlying facts move, or they quietly turn into a source of confidently stale information. A support team learned this when a cached answer kept citing a discontinued plan for days after it was retired, because nobody had set a TTL or a purge trigger on the knowledge that fed it. The rule of thumb: the more volatile the underlying data, the shorter the cache lifetime, and anything tied to prices, availability, or account state needs an explicit invalidation path rather than a passive expiry.

Next Steps

Audit where your tokens actually go before caching anything. Measure how much of each request is stable prefix versus dynamic content, and how much of your bill is input versus output. Those two numbers tell you which cache layers will pay off and roughly how much — and they stop you from repeating the "cache everything and hope" mistake.

As you settle on cache boundaries, thresholds, and TTLs, keep those configurations reusable. Teams that store their caching patterns and prompt structures in a shared library like PromptABCD apply a proven AI agent caching setup to each new agent instead of relearning where the boundary goes every time. The win from caching is real and large — it just belongs to the teams that cache deliberately, not the ones that cache everything.