Prompt caching has become a real production control, not a vendor footnote. OpenAI's June 26 GPT-5.6 preview introduced explicit cache breakpoints and a 30-minute minimum cache life, while Google's July 7 Gemini caching guidance made the same design principle visible across another major API: repeated prompt prefixes can be cheaper and faster when they stay identical.
For a support chatbot, the useful move is to split every request into a stable prefix and a fresh suffix. Cache instructions and tool definitions that rarely change. Keep the visitor's message, retrieved sources, account data, prices, inventory, and timestamps outside the cached block.
Use this prompt order as the starting blueprint:
| Position | Prompt block | Cache policy | Refresh trigger |
|---|---|---|---|
| 1 | Tool names, descriptions, and schemas | Cache | Tool contract or permission changes |
| 2 | Role, safety rules, tone, and escalation policy | Cache | Approved prompt version changes |
| 3 | Stable examples and response format | Cache | QA finds a bad pattern |
| 4 | Shared policy text used on nearly every turn | Cache only if versioned | Source owner publishes a revision |
| 5 | Retrieved snippets, customer data, current state, and user message | Never treat as stable | Rebuild on every turn |
That boundary captures most of the savings without asking an old cache entry to answer a new question.
What Prompt Caching Reuses
Prompt caching does not save and replay the chatbot's answer. It reuses model work for an identical beginning of a request. The model still reads the uncached suffix and generates a new response.
That distinction matters. Suppose every request starts with 4,000 tokens of system instructions, tool schemas, formatting rules, and examples. Sending those same 4,000 tokens on every turn creates repeated input cost. A cache hit lets the provider reuse the computed representation of that prefix, then process the new question and context normally.
The prefix must usually match exactly. A timestamp inserted near the top, a reordered tool definition, a rotating request ID, or one changed space can move the cache boundary and turn an expected hit into a miss. Anthropic's cache diagnostics documentation calls out the same failure pattern: small changes early in the request can silently invalidate everything after them.
This is why prompt caching starts as an application-structure task. Buying a model with a cache discount does not help if every request begins differently.
A Worked Chatbot Request
Consider a returns chatbot for an ecommerce store. The request is assembled in this order:
CACHED PREFIX — version support-prompt-17 [tools] - look_up_order(order_id, email) - create_support_handoff(order_id, reason, summary) [system] - Answer only from supplied sources or tool results. - Never promise a refund before checking eligibility. - Escalate policy conflicts with a structured summary. - Keep answers under 90 words unless the customer asks for detail. [examples] - Missing order access -> explain limitation, collect fields, hand off. - Conflicting policy -> quote neither as final, flag source conflict. FRESH SUFFIX — rebuilt for this turn [retrieved source] Returns policy revision 2026-07-09: Standard items may be returned within 30 days. Clearance purchases are final sale. [customer state] Order 48192: clearance item, delivered 12 days ago. [user] Can I return this? It arrived later than I expected.
The tool contracts, safety rules, format, and examples can be reused across many conversations. The policy revision, order record, and customer question cannot.
A sound answer would say that the item is marked clearance and the current policy makes clearance purchases final sale, then offer a handoff if the customer believes the order was labeled incorrectly. The cache helped process the stable operating rules; it did not decide eligibility from yesterday's data.
If your system prompt is still changing weekly, first tighten it with the chatbot prompt engineering guide. Cache efficiency improves after the prompt has a clear owner and version, not while several teams inject live text into it.
Keep Fresh Data After the Breakpoint
The most common caching mistake is putting too much into the stable prefix. A long cacheable block looks efficient until it contains something that changes during the day.
Prices and plan limits. Retrieve them from the current source or service. Do not bury a copied pricing table inside a long-lived cached instruction block.
Inventory, delivery estimates, and appointment availability. These are live state. Even a short cache lifetime can be too long when the underlying value changes after each booking or purchase.
Customer identity and permissions. Build these for the current session after authentication. A cache key is an optimization hint, not an authorization boundary.
Retrieved knowledge chunks. Most retrieval-augmented chatbots select different snippets for each question. Keep those snippets in the dynamic suffix unless you deliberately maintain a small, versioned policy pack used in nearly every request.
Current time and random values. Do not interpolate them into the first system block. Put them late in the request, and only when the answer actually needs them.
This ordering also makes source debugging easier. When a transcript is wrong, reviewers can separate stable behavior rules from the exact evidence retrieved for that turn. The chatbot optimization guide covers the retrieval side of that problem; prompt caching should preserve its observability, not hide it.
Give Cached Knowledge a Freshness Contract
Sometimes a shared knowledge block is worth caching. A support bot may include the same legal disclaimer, return-policy summary, or product taxonomy on almost every turn. Treat that content like a deployable artifact.
Give it four fields:
| Field | Example | Purpose |
|---|---|---|
| Content version | returns-policy-2026-07-09 | Makes the active revision visible in logs |
| Source digest | sha256:8fe2... | Detects silent content changes |
| Owner | support-ops | Names who can approve a refresh |
| Maximum age | 60 minutes | Prevents indefinite reuse after an update |
When the source changes, change the version or cache key and let the old entry expire. Do not rely on someone remembering to clear a provider-side cache manually.
The version should appear in the conversation trace or request metadata. If a customer reports an outdated answer, support can then identify whether the bot retrieved the wrong page, used an old cached block, or generated a bad interpretation from current evidence.
For sensitive policies, pair the version with a small regression set. Ask the bot about the normal case, every important exception, a conflicting-source case, and a missing-data case. Run those prompts after each refresh. Caching reduces repeated computation; it does not reduce the need for answer review.
Compare Provider Behavior Before You Implement
The stable-prefix principle travels well, but provider controls differ. Check the current documentation for the model and endpoint you actually use.
| Provider behavior | What the application must do | What to monitor |
|---|---|---|
| OpenAI GPT-5.6 supports explicit breakpoints and a 30-minute minimum cache life | Choose a boundary before per-turn data; account for cache-write pricing | Cached input tokens, write overhead, hit rate |
| Anthropic supports automatic or explicit caching, with a five-minute default and optional one-hour lifetime | Keep tools, system, then messages in a stable order | Cache creation and cache-read input tokens |
| Gemini uses implicit caching by default on supported newer models | Put large common content first and keep similar requests close together | total_cached_tokens and model minimums |
| Mistral offers a prompt cache key to improve hit probability | Reuse a non-sensitive key for compatible prefixes | Cached-token count; a key does not guarantee a hit |
Avoid a portability wrapper that flattens these differences into one Boolean called cache: true. Your integration needs to preserve provider-specific usage fields, cache lifetimes, minimum prompt sizes, and invalidation behavior. Otherwise the cost dashboard will report a feature as enabled without proving that any request benefited.
Calculate Savings From Repeated Prefixes
Estimate the opportunity with four numbers:
repeated prefix tokens per day = conversations x model turns x stable prefix tokens
Suppose a chatbot handles 10,000 conversations per day, averages five model turns per conversation, and repeats a 4,000-token stable prefix.
10,000 x 5 x 4,000 = 200,000,000 repeated input tokens per day
For an illustrative model charging $2.50 per million uncached input tokens and $0.25 per million cached input tokens, the repeated prefix costs $500 per day without caching.
At an 80% cache-hit rate:
40M uncached tokens x $2.50 / 1M = $100 160M cached tokens x $0.25 / 1M = $40 repeated-prefix cost = $140 per day, before cache-write overhead
That is a $360 daily difference in the example. Your result will vary with prompt size, traffic shape, cache writes, model pricing, and whether many visitors arrive close enough together to share a live cache entry.
Do not optimize only for the discount. A smaller stable prompt may save more than caching a bloated one. Remove duplicate instructions and unused tool definitions first. Then cache the prefix that remains.
Measure Hits, Misses, and Answer Quality Together
A rollout dashboard needs both infrastructure and support metrics:
| Metric | Formula | Investigate when |
|---|---|---|
| Cache-hit rate | Cached prefix reads / eligible requests | It falls after a deploy or configuration change |
| Cached-token share | Cached input tokens / total input tokens | It stays low despite long repeated prompts |
| Cost per resolved chat | Model spend / resolved conversations | Spend falls but resolution falls faster |
| First-token latency | Median time until response starts | Cache hits do not improve long-prefix latency |
| Freshness failures | Outdated answers / reviewed answers | Any sensitive policy answer uses an old revision |
Watch the metrics by prompt version and model. A global hit rate can hide one high-volume route that misses because it inserts a timestamp before the breakpoint. Compare consecutive requests at the byte level when a known-stable prefix stops hitting.
Keep support quality in the same review. A team can celebrate lower input spend while the bot answers from stale copied policy text. Sample conversations before and after the rollout, especially pricing, refunds, account access, and other questions where recency matters. The AI chatbot QA process provides a practical review loop for turning those failures into regression cases.
Prompt caching also complements, rather than replaces, traffic controls. Cache hits make repeated work cheaper; they do not make abusive or infinite conversations free. Keep the session boundaries from the chatbot rate limits guide in place.
Ship the Boundary, Then Tune It
Start with one cache boundary after stable tools, instructions, and examples. Log the prompt version, cache usage fields, source version, response latency, and answer outcome. Run the rollout on a small traffic slice for a week before adding multiple breakpoints or longer-lived knowledge blocks.
The durable rule is simple: stable behavior belongs before the cache boundary; live evidence belongs after it. That structure lowers repeated input work while keeping the answer grounded in the visitor's current question and the latest available facts.
No credit card required.
