LLM Prompt Caching: Cut API Costs by 90% (All 3 Providers)

LLM prompt caching lets you reuse previously processed tokens across API calls -- cutting input costs by up to 90% and reducing time-to-first-token by up to 85%. If you're sending the same system prompt, tool definitions, or few-shot examples on every request, you're paying full price for work the GPU already did.

This guide covers OpenAI, Anthropic, and Gemini with the same chatbot implemented in all three SDKs -- something no other guide does. We'll also cover Anthropic's February 2026 automatic caching update, production cost scenarios with real dollar amounts, and the anti-patterns that silently destroy your cache hit rate.

Quick Summary -- All Three Providers at a Glance

Before diving into implementation details, here's the full comparison. If you already know which provider you're using, jump to their section. If you're evaluating, this table tells you everything in 10 seconds.

Key takeaway: OpenAI is the simplest (zero config, 50% discount). Anthropic gives the deepest discount (90%) with the most control. Gemini offers configurable TTL and implicit caching on 2.5+ models with comparable discounts to Anthropic.

How Does LLM Prompt Caching Work?

You don't need to understand transformer internals to use prompt caching effectively. But you do need to understand one concept: prefix matching.

The KV Cache in 60 Seconds

When an LLM processes your prompt, it computes attention states (key-value pairs) for every token. These KV cache entries are the expensive part -- they're what eats GPU memory and compute time. Prompt caching stores these computed states so the next request with the same prefix skips recomputation entirely.

The critical word is prefix. The cache matches from the beginning of your prompt forward. If the first 2,000 tokens match a cached entry but token 2,001 differs, those first 2,000 tokens are served from cache. Everything after the divergence point gets computed fresh.

This is why prompt ordering matters. Structure your prompts like this:

1. Tool definitions (most static)
2. System prompt
3. Static few-shot examples
4. Retrieved context (semi-dynamic)
5. Conversation history (grows per turn)
6. User query (always different)

Static content first, dynamic content last. The more tokens that match the cached prefix, the bigger your savings.

Prompt Caching vs Semantic Caching vs Response Caching

These three terms get confused constantly. Prompt caching (what this guide covers) reuses computed KV states at the GPU level for identical token prefixes -- zero accuracy loss, same output as uncached. Semantic caching uses embedding similarity to return previously generated responses for "similar enough" queries -- faster but can return wrong answers. Response caching stores exact input-output pairs and returns the cached response verbatim -- only works for truly identical requests.

Prompt caching is the only "free optimization" -- it reduces cost and latency without any accuracy trade-off. For the deep transformer math behind KV caching, Hugging Face's technical explainer measured a ~5.21x speedup on T4 GPUs.

How Does OpenAI Handle Prompt Caching?

OpenAI's prompt caching is fully automatic. Since October 2024, every API call with 1,024+ input tokens automatically benefits from caching. You don't opt in, you don't add headers, you don't change your code.

How OpenAI's Automatic Caching Works

When you send a request with at least 1,024 tokens, OpenAI checks if the prefix matches a recent request from your organization. Cache hits cost 50% of the standard input token price. After the initial 1,024-token threshold, the cache matches in 128-token increments.

The cache lives for 5-10 minutes during normal usage and can persist up to 24 hours with extended retention during off-peak periods. It's scoped per organization, so different projects within the same org benefit from shared caches.

Supported models include GPT-4o, GPT-4o-mini, GPT-4.1, o1, o3-mini, and all newer models.

OpenAI Python SDK Example

from openai import OpenAI

client = OpenAI()

# This system prompt is ~2,000 tokens -- well above the 1,024 minimum
SYSTEM_PROMPT = """You are a senior Python developer specializing in async programming.
You follow PEP 8, use type hints, and write comprehensive docstrings.
When reviewing code, check for: race conditions, resource leaks, error handling,
and performance bottlenecks. Always suggest specific fixes with code examples.
[... imagine 1,800 more tokens of coding guidelines, examples, and rules ...]"""

def chat(user_message: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
    )

    # Check if caching kicked in
    usage = response.usage
    cached = usage.prompt_tokens_details.cached_tokens
    total_input = usage.prompt_tokens
    print(f"Cached: {cached}/{total_input} tokens ({cached/total_input*100:.0f}%)")

    return response.choices[0].message.content

# First call: cache miss (full price)
chat("Review this async function for race conditions...")

# Second call within 5-10 min: cache hit (50% off on cached tokens)
chat("Now optimize the same function for throughput...")

The first call processes everything at full price and populates the cache. The second call reuses the cached system prompt tokens at half price. You'll see something like Cached: 1920/2048 tokens (94%) in the output.

Verdict: OpenAI is the easiest to start with -- zero configuration, caching just happens. The 50% discount is the lowest of the three providers, but you can't beat the simplicity.

How Does Anthropic/Claude Handle Prompt Caching?

Anthropic offers two modes: automatic caching (enabled by default since February 2026) and explicit caching with cache_control breakpoints. The headline number is hard to ignore -- cached reads cost just 10% of standard input price, a 90% discount.

Automatic vs Explicit Caching (2026 Update)

As of February 5, 2026, Anthropic enables automatic caching by default for all eligible prompts. You don't need the old beta header anymore. The system determines optimal cache breakpoints automatically.

Explicit caching is still available when you want fine-grained control. You place cache_control: {"type": "ephemeral"} on specific content blocks to mark exactly where the cache boundary should be. This is useful when your prompt has a specific structure and you want to guarantee certain sections are cached.

Two TTL options exist:

• 5-minute cache (default): writes cost 1.25x base input price, reads cost 0.1x. Pays for itself after 1 cache hit.
• 1-hour cache: writes cost 2x base input price, reads cost 0.1x. Pays for itself after 2 cache hits. Available on Claude 4.5+ models.

Cache isolation changed from organization-level to workspace-level on February 5, 2026. This means different workspaces within the same organization maintain separate caches.

When working with Anthropic's caching, it helps to structure your prompt for optimal caching -- placing static content before dynamic content is even more important here since you're paying a write premium.

Anthropic Python SDK Example

import anthropic

client = anthropic.Anthropic()

SYSTEM_PROMPT = """You are a senior Python developer specializing in async programming.
You follow PEP 8, use type hints, and write comprehensive docstrings.
When reviewing code, check for: race conditions, resource leaks, error handling,
and performance bottlenecks. Always suggest specific fixes with code examples.
[... imagine 1,800 more tokens of coding guidelines, examples, and rules ...]"""

def chat(user_message: str) -> str:
    response = client.messages.create(
        model="claude-sonnet-4-5-20250514",
        max_tokens=1024,
        system=[
            {
                "type": "text",
                "text": SYSTEM_PROMPT,
                "cache_control": {"type": "ephemeral"},  # Explicit breakpoint
            }
        ],
        messages=[
            {"role": "user", "content": user_message},
        ],
    )

    # Read cache metrics from the response
    usage = response.usage
    created = usage.cache_creation_input_tokens
    read = usage.cache_read_input_tokens
    standard = usage.input_tokens
    print(f"Cache write: {created}, Cache read: {read}, Standard: {standard}")

    return response.content[0].text

# First call: cache_creation_input_tokens = ~1920 (write at 1.25x)
chat("Review this async function for race conditions...")

# Second call: cache_read_input_tokens = ~1920 (read at 0.1x -- 90% off!)
chat("Now optimize the same function for throughput...")

Understanding Cache Write vs Cache Read Pricing

Here's where Anthropic's pricing gets interesting. Using Claude Sonnet 4.5 ($3/MTok base input) as an example:

• Standard input: $3.00 per million tokens
• Cache write (5-min): $3.75 per million tokens (1.25x) -- you pay more the first time
• Cache read: $0.30 per million tokens (0.1x) -- 90% cheaper on every subsequent hit

The 5-minute cache pays for itself after just 1 read. The 1-hour cache ($6.00/MTok write) pays for itself after 2 reads. If you're making more than a couple of requests per minute with the same prefix, the math is overwhelmingly in your favor.

Verdict: Anthropic gives the deepest discount (90%) and the most control. Best for high-volume, cost-sensitive workloads.

How Does Google Gemini Handle Prompt Caching?

Gemini takes a different approach with two distinct caching mechanisms: explicit context caching (named cache objects you create and reference) and implicit caching (automatic, zero-config, added in 2026 for Gemini 2.5+ models).

Explicit Context Caching (Named Caches)

Unlike OpenAI and Anthropic where caching is transparent, Gemini's explicit caching requires you to create a named cache object first, then reference it in subsequent requests. The minimum token threshold is 1,024 tokens for Gemini Flash models and 4,096 tokens for Pro models. TTL is configurable -- default is 1 hour, but you can set it to whatever you need.

Cached tokens on Gemini 2.5 Pro are priced at $0.125/MTok vs the standard $1.25/MTok input price -- a 90% discount. There's also a storage cost of $4.50 per million tokens per hour for Pro, and $1.00 for Flash.

Implicit Caching in Gemini 2.5 (2026)

Starting with Gemini 2.5 Pro and Flash, Google added implicit caching -- automatic caching that works like OpenAI's approach. No configuration needed. Place large, common content at the beginning of your prompt and send requests with similar prefixes in quick succession. The system automatically detects cache-eligible content and passes on savings.

Gemini Python SDK Example

from google import genai
from google.genai import types

client = genai.Client()

SYSTEM_PROMPT = """You are a senior Python developer specializing in async programming.
You follow PEP 8, use type hints, and write comprehensive docstrings.
When reviewing code, check for: race conditions, resource leaks, error handling,
and performance bottlenecks. Always suggest specific fixes with code examples.
[... imagine 1,800 more tokens of coding guidelines, examples, and rules ...]"""

# Step 1: Create a named cache object
cache = client.caches.create(
    model="gemini-2.5-flash",
    config=types.CreateCachedContentConfig(
        display_name="python-review-guidelines",
        system_instruction=SYSTEM_PROMPT,
        ttl="3600s",  # 1 hour
    ),
)

print(f"Cache created: {cache.name}, expires: {cache.expire_time}")

# Step 2: Use the cache in requests
response = client.models.generate_content(
    model="gemini-2.5-flash",
    contents="Review this async function for race conditions...",
    config=types.GenerateContentConfig(
        cached_content=cache.name,
    ),
)

# Check cache usage in the response
metadata = response.usage_metadata
print(f"Cached tokens: {metadata.cached_content_token_count}")
print(f"Total input tokens: {metadata.prompt_token_count}")

The explicit approach has one big upside: you control the TTL precisely. If you know your batch job runs for 4 hours, set a 4-hour TTL and avoid cache expiration in the middle of processing.

Verdict: Gemini's configurable TTL and dual caching modes (explicit + implicit) make it versatile. The minimum threshold is now comparable to other providers, and the 90% discount on cached reads matches Anthropic.

Side-by-Side Code Comparison -- Same Use Case, All 3 Providers

Here's the same chatbot with a cached system prompt, implemented in all three SDKs. Compare the developer experience directly.

# --- OpenAI: Zero config, just call the API ---
from openai import OpenAI
client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},  # Cached automatically
        {"role": "user", "content": user_message},
    ],
)
cached = response.usage.prompt_tokens_details.cached_tokens

# --- Anthropic: Explicit cache_control breakpoint ---
import anthropic
client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-4-5-20250514",
    max_tokens=1024,
    system=[{
        "type": "text",
        "text": SYSTEM_PROMPT,
        "cache_control": {"type": "ephemeral"},  # Mark cache boundary
    }],
    messages=[{"role": "user", "content": user_message}],
)
cached = response.usage.cache_read_input_tokens

# --- Gemini: Named cache object ---
from google import genai
from google.genai import types

client = genai.Client()
cache = client.caches.create(
    model="gemini-2.5-flash",
    config=types.CreateCachedContentConfig(
        system_instruction=SYSTEM_PROMPT,
        ttl="3600s",
    ),
)
response = client.models.generate_content(
    model="gemini-2.5-flash",
    contents=user_message,
    config=types.GenerateContentConfig(cached_content=cache.name),
)
cached = response.usage_metadata.cached_content_token_count

If you want zero-effort savings, go with OpenAI. If you want the deepest discount and fine-grained control, choose Anthropic. If you need configurable cache lifetimes or you're already on Google Cloud, pick Gemini.

Production Cost Calculator -- Real Savings at Scale

Abstract percentages don't drive decisions. Dollar amounts do. Here are three production scenarios with real cost estimates using Claude Sonnet 4.5 ($3/MTok input), GPT-4o ($2.50/MTok input), and Gemini 2.5 Pro ($1.25/MTok input).

Pricing verified March 2026. Check Anthropic pricing, OpenAI pricing, and Gemini pricing for current rates.

Assumptions: 80% cache hit rate (realistic for well-structured prompts), output tokens excluded since caching only affects input costs.

At the Growth tier, Anthropic caching saves $576/month despite having a higher base price than OpenAI. At Enterprise scale, you're looking at $7,200/month in savings with Anthropic -- or $86,400 per year. That's a senior engineer's worth of savings from a configuration change.

The pattern is clear: the higher your request volume and the longer your static prefix, the more caching saves. Anthropic's 90% discount dominates at scale, but Gemini's lower base price makes it competitive when you factor in total cost.

Prompt Caching Anti-Patterns -- When NOT to Cache

Caching seems simple until your cache hit rate mysteriously sits at 0%. Here are the mistakes that silently break prompt caching -- and how to fix them.

Cache-Breaking Mistakes (With Fixes)

Timestamps in system prompts -- The most common mistake. If your system prompt includes datetime.now(), the cache key changes every second.

# BAD: Cache misses every single request
system_prompt = f"""You are a helpful assistant.
Current time: {datetime.now().isoformat()}
Always be helpful and accurate."""

# GOOD: Move the timestamp to the user message
system_prompt = """You are a helpful assistant.
Always be helpful and accurate."""

user_message = f"[Current time: {datetime.now().isoformat()}]\n{user_query}"

User-specific content before static content -- If you put session_id or user preferences at the start, every user gets a unique prefix.

# BAD: Unique prefix per user = zero cache reuse
messages = [
    {"role": "system", "content": f"User ID: {user_id}\nPreferences: {prefs}\n{GUIDELINES}"},
    {"role": "user", "content": query},
]

# GOOD: Static content first, user context at the end
messages = [
    {"role": "system", "content": GUIDELINES},  # Same for all users -> cached
    {"role": "user", "content": f"Context: User {user_id}, prefs: {prefs}\n{query}"},
]

When Prompt Caching Genuinely Doesn't Help

Some scenarios won't benefit from caching even if you structure your prompts perfectly:

• Single-use prompts: If every request has a completely unique context and no shared prefix, there's nothing to cache.
• Very short prompts: Under 1,024 tokens (OpenAI/Anthropic) or 4,096 tokens (Gemini Pro), caching doesn't activate.
• Infrequent requests: If requests are hours apart, the cache expires before a second request arrives. OpenAI's 5-10 minute window and Anthropic's 5-minute default TTL mean you need consistent traffic.

Does Prompt Caching Work With Streaming?

Yes. Prompt caching and streaming are independent -- caching operates on input tokens, streaming affects output delivery. They solve different problems at different stages of the request lifecycle.

The cache handles the prefill phase (processing your input prompt). Streaming handles the decode phase (generating and sending output tokens incrementally). You get both benefits simultaneously: faster prefill from the cache hit, plus progressive output delivery from streaming.

Here's a streaming example with caching enabled:

import anthropic

client = anthropic.Anthropic()

with client.messages.stream(
    model="claude-sonnet-4-5-20250514",
    max_tokens=1024,
    system=[{
        "type": "text",
        "text": SYSTEM_PROMPT,
        "cache_control": {"type": "ephemeral"},
    }],
    messages=[{"role": "user", "content": "Explain Python's GIL..."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

    # After streaming completes, check cache metrics
    usage = stream.get_final_message().usage
    print(f"\nCache read: {usage.cache_read_input_tokens} tokens")

The TTFT improvement from caching is actually most noticeable with streaming. Without caching, you wait for the full prefill before the first token streams back. With caching, the prefill is near-instant, so tokens start flowing almost immediately.

How to Monitor Cache Hit Rates in Production

Setting up caching is half the battle. Knowing whether it's actually working is the other half. If your cache hit rate drops below 50%, something changed in your prompt structure and you're leaving money on the table.

Provider-Specific Cache Metrics

A Simple Cache Hit Rate Logger

Here's a utility function you can drop into any project to track cache hit rates via API response fields:

import logging

logger = logging.getLogger("cache_monitor")

def log_cache_metrics(provider: str, usage: dict) -> float:
    """Extract and log cache metrics from any provider's response. Returns hit rate."""

    if provider == "openai":
        cached = getattr(usage.prompt_tokens_details, "cached_tokens", 0)
        total = usage.prompt_tokens
    elif provider == "anthropic":
        cached = usage.cache_read_input_tokens
        created = usage.cache_creation_input_tokens
        total = cached + created + usage.input_tokens
    elif provider == "gemini":
        cached = getattr(usage, "cached_content_token_count", 0)
        total = usage.prompt_token_count
    else:
        raise ValueError(f"Unknown provider: {provider}")

    hit_rate = (cached / total * 100) if total > 0 else 0
    logger.info(f"[{provider}] Cache hit rate: {hit_rate:.1f}% ({cached}/{total} tokens)")

    if hit_rate < 50:
        logger.warning(f"[{provider}] Low cache hit rate! Check prompt structure.")

    return hit_rate

A healthy production system should sustain 70-90% cache hit rates. If you're below 50%, revisit the anti-patterns section. You can also integrate this with automated evaluation metrics to catch regressions in your prompt pipeline.

Prompt Caching in Real-World Use Cases

The chatbot examples above illustrate the mechanics, but prompt caching really shines in specific architectural patterns.

RAG Pipelines

In a RAG setup, your system prompt and few-shot examples are static across all queries. The retrieved documents change every time. Structure your prompt to maximize the cached prefix:

1. System prompt (cached)
2. Few-shot examples (cached)
3. Retrieved documents (dynamic -- goes last)
4. User query (always unique)

With a 5,000-token system prompt and 3,000 tokens of few-shot examples, that's 8,000 tokens cached on every request. At 1,000 requests/day on Anthropic, you'd save roughly $6.50/day just on the cached prefix alone. When you retrieve and cache context blocks, make sure the retrieval output comes after the static prefix.

Multi-Turn Chatbots

Multi-turn conversations are a sweet spot for prompt caching. Each turn adds to the conversation history, but the entire previous conversation is already cached from prior turns. The cache benefit compounds -- by turn 10, you might have 15,000 tokens of cached history with only 200 fresh tokens from the latest user message.

Agentic Systems and MCP Tool Definitions

If you're building agents with tool use, your tool definitions are static JSON schemas repeated on every single API call. A typical agent might have 20+ tools totaling 3,000-5,000 tokens of definitions. That's prime caching material.

This is especially relevant for MCP-based architectures where server tool definitions get sent on every call. With Anthropic's explicit cache_control, you can mark the tools array for caching and guarantee those tokens are reused.

Which Provider Should You Choose?

If you're already using one provider, start there -- prompt caching doesn't require switching. LiteLLM acts as a proxy layer that normalizes caching parameters across providers, which is useful if you're routing requests to multiple models.

FAQ -- LLM Prompt Caching

What is prompt caching in LLMs?

Prompt caching stores the computed attention states (KV cache) from previously processed prompt prefixes. When a subsequent request starts with the same token sequence, the provider reuses those stored states instead of recomputing them -- reducing both cost and latency with zero impact on output quality.

How much does prompt caching save on API costs?

Savings range from 50% to 90% depending on the provider. OpenAI offers a 50% discount on cached input tokens. Anthropic offers up to 90% off (cached reads at 0.1x base price). Gemini offers roughly 90% off on cached reads. Actual savings depend on your cache hit rate, prompt length, and request frequency.

Does OpenAI prompt caching happen automatically?

Yes, since October 2024. Any API call with 1,024+ input tokens automatically benefits from caching. No opt-in, no headers, no code changes required. The cache matches token prefixes from the beginning of the prompt.

What is the difference between prompt caching and semantic caching?

Prompt caching matches exact token prefixes at the GPU level -- there's no accuracy loss, and outputs are identical to uncached requests. Semantic caching uses embedding similarity to find "close enough" previous queries and returns cached responses -- it's faster but can return incorrect or outdated answers. They solve fundamentally different problems.

How long does the prompt cache last?

It varies by provider. OpenAI: 5-10 minutes (up to 24 hours with extended retention). Anthropic: 5 minutes (default) or 1 hour (available on Claude 4.5+ models, costs 2x write). Gemini: configurable, defaults to 1 hour for explicit caches. Implicit caching TTL is managed by Google automatically.

What is the minimum token length for prompt caching?

OpenAI: 1,024 tokens. Anthropic: 1,024 tokens for most current models. Gemini: 1,024 tokens for Flash models, 4,096 for Pro models. Prompts below these thresholds won't activate caching -- this is the most common "it's not working" gotcha.

Does prompt caching work with streaming responses?

Yes. Caching and streaming operate on different phases of the request. Caching speeds up the input prefill phase; streaming delivers output tokens incrementally. Both work simultaneously, and you'll actually notice the TTFT improvement more with streaming enabled.

When should I NOT use prompt caching?

Avoid relying on caching when your prompts are under the minimum token threshold, when you include timestamps or session IDs in the system prompt, when you rotate few-shot examples between calls, or when requests are too infrequent to hit the cache before it expires (5-10 minute window for OpenAI/Anthropic).

Can I use prompt caching with LangChain or LiteLLM?

Yes. LangChain passes provider-specific caching parameters through its API wrappers. LiteLLM provides a unified caching syntax that normalizes cache_control across Anthropic, OpenAI, Gemini, Vertex AI, and Bedrock -- particularly useful for multi-provider setups.

What is a cache hit vs cache miss?

A cache hit means the provider found a matching prefix in memory and reused the stored KV states -- you pay the discounted cached token rate and get faster TTFT. A cache miss means no match was found, so the full prompt is processed from scratch at standard pricing. Check the cached_tokens (OpenAI), cache_read_input_tokens (Anthropic), or cachedContentTokenCount (Gemini) fields in the API response to see which occurred.

Final Verdict

Prompt caching is the lowest-effort, highest-return optimization in the LLM API stack. You don't change your model, you don't sacrifice quality, and the implementation ranges from "do nothing" (OpenAI) to "add one field" (Anthropic) to "create a cache object" (Gemini).

Start with your current provider's automatic caching. Measure your cache hit rate with the logging utility above. If you're below 70%, restructure your prompts (static first, dynamic last) and eliminate the anti-patterns. Most teams see 50-80% cost reduction within a day of implementing these changes.

Sources

• Anthropic Prompt Caching Documentation
• OpenAI Prompt Caching Guide
• Gemini Context Caching Documentation
• Anthropic Model Pricing
• Gemini API Pricing
• KV Caching Explained -- Hugging Face Blog
• LiteLLM Prompt Caching Documentation