LLM Router: Route Requests, Cut Costs 60% [2026]

LLM Router: Route Requests, Cut Costs 60% [2026]

An LLM router is a thin layer between your app and several language models that picks which model handles each request. It inspects the request (task type, complexity, token budget), forwards it to the best-fit model, and fails over to a backup if that model errors. The goal: right-sized answers at the lowest token cost.

Paying a frontier model to answer "what's your refund policy?" is how bills balloon. AWS measured the alternative in April 2025: a classifier router adding 0.53 seconds of latency, a semantic router adding 0.10 seconds, and up to 30% off the bill for routing inside one model family. Rebuild that math across vendors with July 2026 list prices, as we do below, and the cut reaches 70%. Most of the saving comes from one decision, made before a single token is generated.

Key Takeaways

• An LLM router decides which model handles each request, based on task type, cost, or measured quality.
• Five strategies exist: rule-based, cost-aware, latency-aware, semantic (embeddings), and LLM-classifier routing.
• Rule-based routing adds ~0 ms and $0; classifier routing adds 300-800 ms plus classifier token cost per request.
• Routing can cut token spend up to 60% when most simple traffic moves to a 10-20x cheaper model.
• Single provider, under 10k requests a day, no cost pressure? Skip the router. Plain fallbacks are enough.

What Does an LLM Router Actually Do?

An LLM router runs a small decision step before every model call: read the request, score it against a routing rule, pick a model, send the call, and retry on a fallback if the first model errors. Nothing else about your app changes. You still make one request and get one response back.

The request lifecycle, in order:

1. Request arrives at the router endpoint, exactly as it would at a model API.
2. Analyze. The router inspects the prompt: keywords, token count, an embedding, or a classifier score.
3. Select. The routing strategy maps that signal to a model tier (cheap, mid, frontier, or local).
4. Forward. The call goes to the chosen model through an OpenAI-compatible API.
5. Fallback. On timeout, rate limit, or error, the request retries on the next tier in the chain.

People search "llm gateway vs router" because vendor docs blur the terms. One sentence fixes it: the gateway is the pipe; the router is the decision. They're layers, not rivals, and most gateways embed a router inside.

Per LiteLLM's docs, the same proxy that holds your virtual keys also runs the router. Comparing the pipe-level tools specifically? Our roundup of the best LLM gateway tools ranks ten.

Do You Even Need an LLM Router?

Most small apps do not. A router earns its keep when traffic splits into clearly different task types, when the token bill is your top infra cost, or when you run more than one provider and need failover. Below those thresholds, plain retries plus one fallback model buy you the reliability without the moving part.

We'll say it bluntly, because nobody else in this space will: if you run one provider under 10k requests a day, a router is overhead you don't need. Plain fallbacks win.

Why be so blunt? Every route is a claim ("this task class is safe on the cheap model") that decays as models, prices, and your product change. Buy that maintenance cost only when the savings clearly beat it.

The 5 LLM Routing Strategies (And When to Use Each)

Every LLM routing strategy answers one question: what signal do you trust enough to pick a model? Rules trust keywords. Cost routing trusts the token budget. Latency routing trusts a timer. Semantic routing trusts embeddings. Classifier routing trusts another LLM. The tradeoff is always the same shape: more signal quality, more added latency and cost per request.

Autocomplete surfaces these as "llm routing strategies," "llm task routing," "llm intent routing," and "llm dynamic routing." They map onto five patterns:

One pattern cuts across all five: the cascade, also called model tiering. Start cheap and escalate only on failure or low confidence. A support bot answers from a $0.25-per-million-token model; if its confidence dips below 0.7, the same request retries on a frontier model. You pay for intelligence only when the cheap tier admits it is stuck.

For academic depth, ulab-uiuc's LLMRouter library catalogs 16+ researched routing algorithms (KNN, SVM, MLP, matrix factorization, Elo, graph, and BERT-style). If semantic routing is your pick, the exemplar embeddings decide almost everything; our guide to the best embedding models covers which ones hold up on real corpora.

How Do You Build an LLM Router in Python?

You build one with about 80 lines of plain Python against any OpenAI-compatible endpoint. No framework required. The four routers below escalate in sophistication: keyword rules, a cost threshold, embedding similarity, and a classifier model with failover. Each one prints the model it chose, so you can watch the decision happen.

If you've searched "how to build an llm router" and found only AWS CDK stacks and academic repos, this section is the plain answer. AWS's reference implementation is solid but welded to Bedrock, Lambda, and CDK. Ours runs anywhere the OpenAI client points: OpenAI, Anthropic through a proxy, Ollama on a laptop, vLLM on a GPU box. Here's the router we sketch for clients first.

Step 1: Rule-based router (keywords to models)

The zero-latency baseline. A regex map decides; everything unmatched goes to the frontier tier.

import re
from openai import OpenAI

client = OpenAI()  # works with OpenAI, Ollama, vLLM, or a LiteLLM proxy

def ask(model: str, prompt: str) -> str:
    r = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
    )
    return r.choices[0].message.content

ROUTES = [
    (re.compile(r"\b(refund|cancel|invoice|password|hours)\b", re.I), "gpt-5-mini"),
    (re.compile(r"\b(summarize|translate|rewrite)\b", re.I), "gpt-5-mini"),
]
FRONTIER = "gpt-5"

def rule_router(prompt: str) -> str:
    for pattern, model in ROUTES:
        if pattern.search(prompt):
            return model
    return FRONTIER

prompt = "How do I cancel my subscription?"
model = rule_router(prompt)
print(model)  # gpt-5-mini: regex hit on "cancel"
print(ask(model, prompt))

Input: a support question. Decision: regex match on "cancel." Chosen model: gpt-5-mini. No API call needed to route it, which is why this stays the default.

Step 2: Cost-aware router (token budget threshold)

Same idea, but the signal is request size instead of keywords. Short prompts with small output budgets go cheap; everything else goes frontier.

def cost_router(prompt: str, max_output_tokens: int = 500) -> str:
    word_count = len(prompt.split())
    if word_count < 60 and max_output_tokens <= 300:
        return "gpt-5-mini"  # $0.25 in / $2 out per M tokens
    return "gpt-5"           # $1.25 in / $10 out per M tokens

prompt = "Write a two-line product description for a ceramic mug."
model = cost_router(prompt, max_output_tokens=120)
print(model)  # gpt-5-mini: short prompt, small output budget

Crude? Yes. Effective? Also yes, because token volume correlates with task size better than most people expect. This is the entire strategy behind several paid "cheap llm router" products.

Step 3: Semantic router (embeddings to exemplars)

For fuzzy user input that dodges keywords, embed the prompt and compare it against embedded exemplar prompts. Whichever cluster is closest owns the request.

import numpy as np

EXEMPLARS = {
    "gpt-5-mini": [
        "classify this support ticket into a category",
        "extract the shipping address from this email",
    ],
    "gpt-5": [
        "debug this race condition in our worker pool",
        "design a multi-tenant billing schema",
    ],
}

def embed(texts: list[str]) -> np.ndarray:
    r = client.embeddings.create(model="text-embedding-3-small", input=texts)
    return np.array([d.embedding for d in r.data])

CENTROIDS = {m: embed(xs).mean(axis=0) for m, xs in EXEMPLARS.items()}

def semantic_router(prompt: str) -> str:
    v = embed([prompt])[0]
    scores = {
        m: float(np.dot(v, c) / (np.linalg.norm(v) * np.linalg.norm(c)))
        for m, c in CENTROIDS.items()
    }
    return max(scores, key=scores.get)

print(semantic_router("pull the tracking number out of this message"))
# gpt-5-mini: closest to the extraction exemplars

The routing call costs one embedding (a few hundred tokens) and 50-150 ms. Precompute the centroids at startup, not per request.

Step 4: LLM-classifier router with fallback

The strongest signal: a cheap model reads the prompt and scores its difficulty. This is the strategy AWS measured at 0.53 seconds added latency, so we wrap it in a fallback chain.

def classify_router(prompt: str) -> str:
    verdict = client.chat.completions.create(
        model="gpt-5-mini",
        messages=[{"role": "user", "content":
            "Reply HARD or EASY only. Task: " + prompt}],
        max_tokens=5,
    ).choices[0].message.content.strip().upper()
    return "gpt-5" if verdict.startswith("HARD") else "gpt-5-mini"

def route_and_call(prompt: str) -> str:
    model = classify_router(prompt)
    try:
        return ask(model, prompt)
    except Exception:
        backup = "gpt-5-mini" if model == "gpt-5" else "gpt-5"
        return ask(backup, prompt)  # fallback tier catches the failure

print(route_and_call("Prove this greedy algorithm is optimal."))
# classifier says HARD, so gpt-5 answers

That's the whole llm router example: four functions, one client, no infrastructure beyond what you already run. Production hardening is the next section.

How Much Does LLM Routing Actually Save?

AWS measured router overhead at $107.90-$188.90 per month per 100,000 questions a day, with classifier routing adding 0.53 seconds per request and semantic routing 0.10 seconds. The savings side dwarfs that overhead. Our worked example below, built on July 2026 list prices, lands at a 70.7% spend reduction. The catch is the traffic mix: you need most requests to qualify for the cheap tier.

Two tables. First, what the router itself costs you per 1,000 requests:

AWS's April 2025 post is the only independently published measurement set in this space, so we anchor to it and label our extensions as estimates, not numbers we ran. Bedrock Intelligent Prompt Routing cut in-family cost by up to 30%, per AWS.

Second, the savings worked example that backs our headline:

Assumptions, labeled: 100,000 requests per month; 800 input plus 200 output tokens per request on average; an 80% simple / 20% complex split; list prices from Anthropic's pricing page and OpenAI's pricing page as of July 2026, with the full rate table in our LLM API pricing comparison. Per-request math: Sonnet 4 costs 800 x $3/M + 200 x $15/M = $0.0054; GPT-5 mini costs 800 x $0.25/M + 200 x $2/M = $0.0006.

The result is a 70.7% reduction, which is where the 60% in our title comes from, with margin to spare. Honest caveats: this is a worked example, not a benchmark we ran. It assumes your cheap tier is 10-20x cheaper and that 80% of traffic genuinely qualifies. In-family routing, AWS's scenario, stays near 30%. And routing is one lever among many; prompt caching and trimming often pay back faster, and our guide to ways to reduce LLM API costs ranks all twelve.

Production Routing Patterns

A toy router picks a model. A production router also retries, balances load, caches repeats, and isolates API keys per team. Past a few thousand requests a day, stop hand-rolling those and run a gateway that embeds a router.

The four patterns that matter:

• Fallback chains. Cheap tier first, frontier on error or timeout. The single highest-value pattern; most of your reliability comes from this alone.
• Load balancing. Spread calls across duplicate deployments or API keys to dodge per-key rate limits.
• Response caching. Identical prompts return cached answers. Support traffic repeats more than you'd believe; 10-30% hit rates are common.
• Virtual keys and budgets. Issue per-team keys with monthly caps so one runaway loop can't torch the whole bill.

This is close to the config we run on our staging agent stack (file: litellm-router.yaml, mounted into the LiteLLM proxy container):

model_list:
  - model_name: cheap
    litellm_params:
      model: openai/gpt-5-mini
  - model_name: frontier
    litellm_params:
      model: anthropic/claude-opus-5

router_settings:
  routing_strategy: simple-shuffle
  fallbacks: [{"cheap": ["frontier"]}]
  num_retries: 2
  timeout: 30

Where each tool fits, with opinions:

• LiteLLM. Pick if you want self-hosted and open source and you already run Docker. Our LiteLLM proxy setup guide walks the full deploy, keys and budgets included.
• OpenRouter. Pick if you want hundreds of models behind one key and zero operations. Their rankings page doubles as throughput data.
• Portkey. Pick if enterprise requirements (SSO, audit logs, compliance reports) drive the decision.
• Custom code from this post. Pick if you're under ~50k requests a day and want zero new infrastructure.

Whichever you choose, the LLM gateway tools roundup compares ten of them head to head.

Can You Route Between Local Models and Hosted APIs?

Yes, and the token math is seductive: a local model bills $0 per token, so every request Ollama or vLLM answers is pure saving. The tradeoff is latency and quality per watt. Local wins for high-volume simple tasks on hardware you already own; the hosted API catches everything that needs a frontier brain.

The mechanics are anticlimactic, which is the point. Ollama exposes an OpenAI-compatible endpoint at localhost:11434/v1, and vLLM serves the same shape. So every router above works unchanged: point base_url at the local server, put qwen3:8b in the cheap slot, and keep gpt-5 as the fallback tier. For a self-hosted router box, LiteLLM ships as a Docker image, which is the "llm router docker" setup people search for.

Two honesty notes. A 70B model on one A100 serves roughly 30-40 tokens per second; hosted APIs beat that on burst throughput, so local routing suits steady background traffic better than spiky user-facing chat. And local 8B models fumble multi-step tool calls, so keep the hard routes pointed at the cloud. If you're choosing the serving engine itself, vLLM vs SGLang benchmarks the two.

Routing also powers multi-model coding-agent setups. A LiteLLM-style proxy lets Claude Code talk to local and hosted models through one endpoint; see how to use different models in Claude Code for the exact wiring.

How Do You Know If Routing Is Working?

You measure it, or you're guessing. Log which model answered every request, score a sample of outputs against a rubric, and feed the scores back into the routing rules. Teams that skip this step end up with a static config that quietly rots as models and prices change underneath it.

The graduation arc runs rules, then cost, then measured quality:

1. Log the route. Store the chosen model, latency, and token counts per request as one column in your existing traces.
2. Score outputs weekly. An LLM judge or a human sample, pass/fail per request class. Fifty graded outputs per class is enough to steer by.
3. Re-tune. If the cheap tier passes 95%+ on a class, widen its rule to catch more of that traffic. If it dips below 90%, tighten.

Here's the line we keep repeating to clients: a router you never re-tune is just a static config with extra latency. Log the chosen model, score outputs, feed scores back.

That loop is evals plus observability applied to routing. Our LLM evals guide covers the scoring rubrics; the AI observability guide covers where the traces live.

Where Is LLM Routing Research Heading?

The academic line treats routing as a learning problem, not a config file. ulab-uiuc's LLMRouter, the library that ranks first for this keyword, implements 16+ algorithms (KNN, SVM, MLP, matrix factorization, Elo, graph, BERT, and RL routers) with a benchmark pipeline over 11 datasets. The most-cited recent paper, RouteLLM (Ong et al., arXiv:2406.18665), trains routers on human preference data and reports over 2x cost reduction without quality loss on MMLU and MT-Bench. The newest wrinkle: prefill-activation routers, the "prefill is all you need" line, which read a model's internal activations during prefill to predict difficulty before generation starts. The direction of travel is routers that train themselves from your eval data, which is exactly the feedback loop from the previous section.

How Techsy approaches this: the agent stacks we ship for B2B clients run exactly this pattern, a cost-tier router with fallback chains wired into the gateway, plus eval-driven re-tuning. If you're weighing whether routing fits your stack, get a free consultation and we'll map your traffic mix with you.

Frequently Asked Questions

What is an LLM router?

An LLM router is a layer between your application and multiple language models that decides which model handles each request. It checks the request's task type, size, or difficulty, then forwards it to the best-fit model, with a fallback if that model fails. Think of it as a traffic controller for your model API calls.

How does LLM routing work?

LLM routing works in five steps: the request arrives, the router inspects it (keywords, token count, or an embedding), a strategy picks a model tier, the call is forwarded, and a fallback model catches any failure. The whole decision happens before generation starts, so it adds milliseconds, not seconds, unless a classifier model does the scoring.

Is an LLM router the same as an LLM gateway?

No. A gateway is the pipe: API keys, rate limits, budgets, and logs. A router is the decision: which model answers. They are layers, not rivals, and most gateways (LiteLLM, Portkey, OpenRouter) embed a router inside. You can run a router without a gateway, but in production you usually want both together.

Does model routing actually save money?

Yes, when most of your traffic qualifies for a much cheaper tier. Our worked example moves 80% of requests from a $3/$15-per-million-token model to a $0.25/$2 one and cuts the bill 70.7%. AWS reported up to 30% for routing inside one model family. If your traffic is uniformly complex, the savings shrink toward zero.

What is the best open-source LLM router?

For production, LiteLLM: self-hosted, actively maintained, and it combines a gateway with a router. For research-grade algorithms, ulab-uiuc's LLMRouter implements 16+ routing strategies from the academic literature. RouteLLM is the strongest quality-per-dollar router trained on preference data. Most teams should start with LiteLLM and reach for the research libraries only if they need custom scoring.

How do I build an LLM router in Python?

Start with the OpenAI client and about 80 lines of code: a rule map from keywords to models, a cost threshold on token counts, embedding similarity to exemplar prompts, or a cheap classifier model that scores difficulty. All four patterns are in the build section above, runnable against OpenAI, Ollama, or vLLM without changes.

Can I route between local models and cloud APIs?

Yes. Ollama (localhost:11434/v1) and vLLM both expose OpenAI-compatible endpoints, so the same router code points at a local model for cheap traffic and a hosted API for hard traffic. Local tokens cost $0, but you own the hardware and the latency. This is the pattern behind most multi-model Claude Code setups.

What is semantic routing?

Semantic routing embeds each incoming prompt and compares it against embedded exemplar prompts, sending the request to whichever model owns the closest exemplar cluster. It handles fuzzy, paraphrased user input that keyword rules miss, at a cost of 50-150 ms plus embedding tokens per request. AWS measured it at 0.10 seconds of added latency.

How much latency does an LLM classifier router add?

AWS measured 0.53 seconds of added latency for LLM-assisted classification, versus 0.10 seconds for semantic routing. Rule-based and cost-aware routing add roughly zero, because they are plain code paths. If your product has a tight response-time SLA, prefer rules, cost thresholds, or embeddings, and reserve the classifier for offline or queued workloads.

Sources

• Seifi, N. and Chugh, M. (2025-04-09). "Multi-LLM routing strategies for generative AI applications on AWS." AWS Machine Learning Blog. https://aws.amazon.com/blogs/machine-learning/multi-llm-routing-strategies-for-generative-ai-applications-on-aws/ (accessed July 30, 2026)
• AWS sample code: sample-multi-llm-dynamic-prompt-routing. https://github.com/aws-samples/sample-multi-llm-dynamic-prompt-routing (accessed July 30, 2026)
• LiteLLM documentation. https://docs.litellm.ai (accessed July 30, 2026)
• Anthropic pricing. https://www.anthropic.com/pricing (accessed July 30, 2026)
• OpenAI API pricing. https://openai.com/api/pricing (accessed July 30, 2026)
• ulab-uiuc LLMRouter. https://github.com/ulab-uiuc/LLMRouter (accessed July 30, 2026)
• Ong, I. et al. (2024). "RouteLLM: Learning to Route LLMs with Preference Data." arXiv:2406.18665. https://arxiv.org/abs/2406.18665 (accessed July 30, 2026)
• OpenRouter rankings. https://openrouter.ai/rankings (accessed July 30, 2026)