How Much Does LLM Inference Cost? A 4-Scenario Breakdown With Real Math

How Much Does LLM Inference Cost? A 4-Scenario Breakdown With Real Math

GPT-4 cost $30 per million input tokens in March 2023. Three years later, GPT-5.6 runs the same million for $10. Claude Opus 4.5 sits at $15. The floor? Two cents. That's a 1,500x spread between the cheapest and priciest option for the identical unit of work. LLM inference cost is the recurring bill you pay every time a trained model reads your input and generates an output, priced per million tokens by every major provider. Budget models run $0.02-$0.30 per million input tokens. Frontier models charge $15-$75 for the same volume. The gap matters because your workload, not your ambition, determines which tier you actually need.

Key Takeaways:

• Budget models cost $0.02-$0.30 per million input tokens; frontier models run $15-$75 (July 2026).
• Output tokens cost 3-5x more than input tokens because generation is sequential, not parallel.
• A 50K-conversation support chatbot costs roughly $9-$420/month depending on model tier.
• Inference prices have fallen ~10x per year; Gartner projects 90% lower by 2030.

How Much Does LLM Inference Cost Per Million Tokens?

LLM inference pricing follows a three-tier structure as of July 2026. Budget models from providers like Google (Gemini 2.5 Flash) and open-source options (Llama 4, Qwen 3) charge $0.02-$0.30 per million input tokens. Mid-tier models (GPT-4.1, Claude Sonnet 4) land between $0.55 and $15. Frontier reasoning models (o3, Claude Opus 4.5) command $15-$75. Every provider publishes these rates on their official pricing pages (OpenAI, Anthropic), and PricePerToken.com tracks 300+ models in a live grid.

As of July 2026, you can run a million input tokens for as little as two cents, or pay seventy-five dollars for the same million on a frontier model.

Cached-input discounts cut your bill 50-90% on repeated prompts (prompt caching cuts input costs explains the mechanics). For the live 300-model grid with every provider's current rates, see our full per-token pricing table.

What Drives LLM Inference Cost? The 4 Components

Your inference bill breaks into four cost drivers: GPU compute time per token, memory for model weights and the KV cache, the input/output asymmetry that makes generation pricier than reading, and infrastructure overhead (power, cooling, networking). Understanding which component dominates your workload tells you where optimization pays off.

Compute: GPU Time Per Token

Every token passes through every layer of the model. A 70B-parameter model at FP16 needs roughly 140 GFLOPs per token. Quantizing to INT4 cuts that to ~35 GFLOPs. NVIDIA's inference benchmarking guide breaks down the full TCO formula: hardware amortization plus electricity plus operational overhead, divided by tokens served.

Memory: Model Weights + KV Cache

A 70B model at FP16 occupies ~140 GB of VRAM just for weights. The KV cache grows with context length and concurrent batch size. At 128K context with 32 concurrent requests, you can burn another 80 GB. This is why VRAM requirements by model matter so much for self-hosting decisions.

Input vs Output Asymmetry

Output tokens cost 3-5x more than input tokens because the model generates them one at a time, in order. It cannot parallelize the way it reads your prompt.

Here's the plain version: reading your 2,000-token prompt (the "prefill" phase) processes all tokens simultaneously across GPU cores. Generating 500 output tokens (the "decode" phase) runs one token per step, each depending on the last. The GPU sits mostly idle waiting on memory bandwidth between steps. That idle time is what you pay for at 3-5x the input rate.

Infrastructure Overhead

Power, cooling, networking, and the GPUs sitting idle between requests. Hugging Face's optimization docs cover how continuous batching and speculative decoding squeeze more tokens per GPU-hour out of the same hardware, directly cutting this overhead share.

What Does LLM Inference Cost for 4 Real Workloads?

A typical support chatbot costs $9-$420/month, a RAG pipeline runs $168-$3,360/month, a 200-developer code assistant bills $123-$2,078/month, and batch document processing lands between $240 and $6,400/month. The spread depends almost entirely on model tier choice. We built these four scenarios from token volumes from our client deployments, priced against official July 2026 pages. Every dollar figure below is reproducible: stated token assumptions multiplied by published rates. Our analysis, not vendor claims.

Customer-Support Chatbot: 50K Conversations/Month

Average conversation: 800 input tokens (system prompt + user messages + retrieved context), 400 output tokens. Monthly volume: 50,000 conversations = 40M input tokens, 20M output tokens.

• Budget pick (Gemini 2.5 Flash): 40M × $0.075/M + 20M × $0.30/M = $9/month. Almost suspiciously cheap.
• Mid pick (Claude Sonnet 4): 40M × $3/M + 20M × $15/M = $420/month.

For a support bot handling tier-1 tickets, the budget model handles 90% of queries fine. Route the hard 10% to mid-tier with a classifier.

RAG Pipeline: 10K Queries/Day

Average query: 3,200 input tokens (retrieved chunks + system prompt + user question), 600 output tokens. Monthly: 300K queries = 960M input tokens, 180M output tokens.

• Budget pick (Llama 4 Scout via API): 960M × $0.10/M + 180M × $0.40/M = $168/month.
• Mid pick (GPT-4.1): 960M × $2/M + 180M × $8/M = $3,360/month.

The RAG workload is input-heavy, so cached-input discounts hit hard here. With 70% prompt caching, the mid-tier drop to ~$2,100/month.

Code Assistant: 200 Developers

Average session: 4,500 input tokens (file context + conversation), 1,200 output tokens. Assume 15 requests/developer/day, 22 working days = 66,000 requests/month = 297M input, 79M output.

• Budget pick (Qwen 3 32B): 297M × $0.20/M + 79M × $0.80/M = $123/month.
• Mid pick (Claude Sonnet 4): 297M × $3/M + 79M × $15/M = $2,078/month.

Developers notice quality gaps fast. For code, mid-tier is usually the floor. Budget models work for autocomplete-style suggestions but struggle with multi-file refactors.

Batch Document Processing: 1M Docs/Month

Average doc: 2,000 input tokens, 300 output tokens (extraction, classification, summarization). Monthly: 2B input, 300M output.

• Budget pick (Gemini 2.5 Flash): 2B × $0.075/M + 300M × $0.30/M = $240/month.
• Mid pick (GPT-4.1): 2B × $2/M + 300M × $8/M = $6,400/month.

At this volume, the 27x price gap between tiers is a $6,160/month line item. Budget models handle structured extraction well. For the hardware sizing if you consider self-hosting this volume, check VRAM requirements by model.

def monthly_cost(input_tokens, output_tokens, requests, price_in, price_out):
    """Estimate monthly LLM inference cost.
    
    Args:
        input_tokens: avg input tokens per request
        output_tokens: avg output tokens per request
        requests: monthly request volume
        price_in: $/M input tokens
        price_out: $/M output tokens
    """
    total_in = input_tokens * requests / 1_000_000
    total_out = output_tokens * requests / 1_000_000
    return (total_in * price_in) + (total_out * price_out)

# Example: support chatbot on Claude Sonnet 4
cost = monthly_cost(
    input_tokens=800,
    output_tokens=400,
    requests=50_000,
    price_in=3.00,
    price_out=15.00
)
print(f"${cost:,.2f}/month")  # $420.00/month

API or Self-Hosted: When Does Each Win?

API wins below ~20K requests/day or when your team lacks ML infrastructure experience. Self-hosted wins above 200K requests/day with sustained GPU utilization over 50%. The breakeven point, per Introl's analysis, lands around 50-80K requests/day for a 70B-class model on a single H100 node. Below that threshold, the API provider's multi-tenant efficiency beats your single-tenant hardware math.

When API Wins

You ship in days, not weeks. No ML engineer salary ($180K-$250K/year), no on-call rotation for GPU failures, no capacity planning. For most teams under 50 engineers, the API is the correct default. Period.

When Self-Hosted Wins

You've crossed 200K requests/day, your workload is predictable, and you already employ ML infrastructure people. Open-source models (Llama 4, Qwen 3, Mistral) run on your hardware at electricity cost plus amortization. The vLLM vs SGLang benchmarks show throughput differences of 15-30% depending on workload shape, which matters at this scale.

The Breakeven Number

Self-hosted wins only above ~50% sustained GPU utilization. Below that, you're paying for idle silicon. The hidden people-costs (ML eng time, on-call, model updates, security patching) are the real reason most teams stay on API even when the raw GPU math looks favorable. If you do self-host, deploying models on Modal removes the infrastructure management layer while keeping per-token costs below API rates.

The Hidden Costs Nobody Budgets For

Raw token spend is 60-80% of your real bill. The rest hides in six line items that never show up in a pricing calculator. Budget an extra 20-40% on top of your token estimate to cover them.

• Monitoring and observability tooling: $200-$1,500/month. Token usage dashboards, latency tracking, cost attribution per feature. An LLM observability stack pays for itself the first time you catch a prompt regression before it burns through your budget.
• Retry and failure re-runs: 3-8% of requests fail or need retry (timeouts, rate limits, malformed outputs). That's 3-8% of your token bill, spent producing nothing.
• Prompt-engineering iteration hours: 20-60 hours per quarter at $100-$200/hour loaded engineering cost. Every prompt tweak re-runs test batches.
• Eval and regression testing: $100-$800/month in token spend for automated eval suites. You're paying the model to grade itself.
• Multi-region redundancy: 1.5-2x infrastructure cost if you need failover across regions for latency or compliance.
• Cold-start latency penalties: Serverless GPU deployments (Modal, AWS Lambda) charge for idle time. Cold starts add 2-8 seconds and bill you for the warm-up.

The rule of thumb: multiply your raw token estimate by 1.3 for a realistic budget. Multiply by 1.4 if you're in a regulated industry with compliance overhead.

Why Does LLM Inference Keep Getting Cheaper?

LLM inference prices have fallen roughly 10x per year since 2023. A workload that cost $1,000/month in 2024 costs closer to $100 today. Three forces drive this: hardware improvements (NVIDIA Blackwell FP4, Google TPU v6), competitive pressure (DeepSeek, open-source models forcing price wars), and software optimization (speculative decoding, continuous batching, quantization). Gartner projects inference costs will drop another 90% by 2030, though that's a projection, not a guarantee.

Epoch AI's price tracking documents this decline with hard data points. a16z's "LLMflation" analysis coined the term and framed the economic implications: inference is deflating faster than any prior compute category.

Training vs Inference Cost

Training a frontier model runs $50M-$200M (one-time). Inference for that same model, across all users, runs $5M-$50M per year depending on scale. For most teams, the ratio is 10-100x: training costs 10-100 times what one year of inference costs. But training is a one-time capital expense. Inference is the recurring operating bill that compounds with user growth. You don't pay for training unless you're building a foundation model. You pay for inference every single day.

The Energy Line Item

An NVIDIA H100 draws ~700W under load. At $0.10/kWh (US average), that's $0.07 per GPU-hour. A 70B model on a single H100 generates roughly 2,000 output tokens/second at batch. Over one hour: 7.2M tokens. Electricity cost per million output tokens: ~$0.01. Our calculation, labeled as such. Energy is 1-3% of your API bill but 8-15% of a self-hosted TCO. It matters more if you're running your own GPUs in a high-electricity-cost region (Germany at $0.30/kWh triples this number).

The practical takeaway: prices are falling fast enough that a 12-month commitment to a specific model tier is risky. Re-evaluate quarterly. The 12 ways to cut your LLM bill covers tactical moves you can make right now while the macro trend does the heavy lifting.

How Do You Estimate YOUR Inference Cost?

Estimate your monthly LLM inference cost in four steps: count tokens per request, multiply by monthly volume, apply tier pricing, then add 20-40% overhead. In our experience sizing inference budgets for clients, most teams skip step four and wonder why their actual bill exceeds the estimate by a third. Here's the process we use.

1. Count tokens per request. Log your average input tokens (system prompt + context + user message) and output tokens (model response) over 100+ real requests. Don't guess. Measure.
2. Multiply by monthly volume. Requests/day × 30 = monthly requests. Multiply by your per-request token counts.
3. Apply tier pricing. Multiply total input tokens by the model's $/M input rate. Same for output. Sum them.
4. Add 20-40% overhead. Retries, evals, prompt iteration, monitoring. This is the number that goes in your budget, not step 3.

def estimate_monthly_budget(avg_input_tokens, avg_output_tokens,
                            requests_per_day, price_in, price_out,
                            overhead_pct=0.30):
    """Full monthly budget estimate with overhead."""
    monthly_requests = requests_per_day * 30
    raw_cost = monthly_cost(
        avg_input_tokens, avg_output_tokens,
        monthly_requests, price_in, price_out
    )
    return raw_cost * (1 + overhead_pct)

# RAG pipeline: 10K queries/day on GPT-4.1
budget = estimate_monthly_budget(
    avg_input_tokens=3200,
    avg_output_tokens=600,
    requests_per_day=10_000,
    price_in=2.00,
    price_out=8.00,
    overhead_pct=0.30
)
print(f"${budget:,.0f}/month")  # $4,368/month

Once you know your number, LLM gateway tools route traffic to the cheapest model that clears your quality bar. A gateway with per-request model selection can cut your blended cost 30-50% without touching your prompts.

Frequently Asked Questions

Is LLM inference expensive?

It depends on scale and model choice. A million input tokens costs $0.02 on Gemini 2.5 Flash or $75 on a frontier reasoning model. For a small app processing 10K requests/day, expect $50-$400/month. For enterprise workloads at 1M+ requests/day, budgets run $5,000-$40,000/month. The per-unit cost is low; volume makes it add up.

How much is 1 million tokens in LLM?

One million tokens equals roughly 750,000 words, or about 10 full-length novels. In July 2026, processing 1M input tokens costs $0.02 (budget tier) to $75 (frontier tier). Output tokens for the same volume run $0.06 to $300. Most production workloads land in the mid-tier: $2-$15 per million input tokens.

Why is AI inference so expensive?

Inference requires running every token through billions of model parameters on GPUs that cost $25,000-$40,000 each. The decode phase generates tokens sequentially, leaving GPU cores idle between steps. You're paying for specialized hardware, electricity, cooling, and the provider's engineering team that keeps the serving stack running 24/7.

Are AI inference costs falling?

Yes, roughly 10x per year since 2023. GPT-4 launched at $30/$60 per million tokens (input/output). Equivalent capability now costs $2-$10. Epoch AI's data confirms this trajectory across all providers. Gartner projects another 90% decline by 2030, driven by hardware improvements and competitive pressure from open-source models.

How much does LLM inference cost in 2026?

Budget models: $0.02-$0.30 per million input tokens. Mid-tier: $0.55-$15. Frontier: $15-$75. A typical production workload (support chatbot, RAG pipeline, or code assistant) costs $150-$4,000/month on mid-tier models. Budget models handle high-volume, low-complexity tasks for 10-30x less.

What is the difference between training cost and inference cost?

Training is the one-time expense of teaching a model from scratch: $50M-$200M for a frontier model, requiring thousands of GPUs for months. Inference is the recurring cost of using that trained model: running inputs through it to get outputs, billed per token. For most teams, inference is the only bill they pay. Training is for companies building foundation models.

How do I calculate LLM inference cost for my app?

Multiply average input tokens per request by your monthly request volume, then by the model's $/M input rate. Repeat for output tokens. Sum both. Add 30% for retries, evals, and monitoring overhead. The Python snippets in this post automate the math. Measure real token counts rather than guessing; logs from 100+ requests give a reliable average.

Do output tokens cost more than input tokens?

Yes, typically 3-5x more. Input processing (prefill) parallelizes across all tokens simultaneously, fully utilizing GPU cores. Output generation (decode) produces one token at a time, each depending on the previous. The GPU waits on memory bandwidth between steps. Providers price output higher to reflect this lower hardware efficiency.

Is self-hosted inference cheaper than using an API?

Only above ~200K requests/day with sustained GPU utilization over 50%. Below that, API providers' multi-tenant efficiency beats your single-tenant hardware. Self-hosting also adds hidden costs: ML engineer salaries ($180K-$250K/year), on-call rotations, model updates, and security patching. Most teams under 50 engineers should stay on API.

Does LLM inference use a lot of energy?

An H100 GPU draws ~700W under load. At $0.10/kWh, that's $0.07/GPU-hour, translating to roughly $0.01 per million output tokens for a 70B model. Energy is 1-3% of an API bill but 8-15% of self-hosted TCO. In high-electricity regions (Germany, $0.30/kWh), the energy share triples and materially affects self-hosting economics.

The Bottom Line

Four numbers to remember: $0.02 (budget floor per million input tokens), 3-5x (output premium over input), 20-40% (overhead to add on top of raw token math), and 10x (annual price decline since 2023). Your actual bill depends on volume, model tier, and how aggressively you cache, batch, and route traffic.

At Techsy, our team sizes inference budgets and picks model tiers for B2B clients running production LLM workloads. If you want a second set of eyes on your cost model, get a free consultation.