The Complete Guide to LLM Routing

By Sean Donahoe · Published July 16, 2026 · accurate as of this date

What is LLM routing?#

A system that sits behind one API endpoint and picks a suitable model for each request at runtime, instead of your code calling one hardcoded model name. That's it. That's the whole definition.

Stop hardcoding one model name into your app and calling it done. Send every request to a single endpoint instead, and a router picks a model suited to that specific job at runtime, normalizes the API across providers, records what it did, and fails over when an upstream breaks. You're still in control, a tier alias or a hard pin, but the default behavior is: right model, this request, now.

Here's the thing nobody puts on a landing page. The model is an input. It is not your product. Inputs decay. They get deprecated, they get repriced, a better one ships next month and makes the one you picked look slow and expensive by comparison. If you wire your app to one static model name, that decay becomes your problem, on your schedule, whenever the provider feels like changing it. Routing is the decision to stop babysitting that.

Underneath the marketing, a router does four jobs, and they're worth naming plainly before anything else in this guide makes sense. It normalizes the API so you write to one shape across many providers. It chooses a model for each request. It attributes cost per request so you know what each call actually cost, not a blended average. And it observes and fails over, telling you which model served the call and substituting when an upstream goes down. Everything below is one of those four jobs, examined properly.

What routing is not: it is not a model. It is not a fine-tune. It does not guarantee a better answer on every single call, because sometimes the router and you would have picked the same thing anyway. What it guarantees is that model choice stops being a hardcoded constant and starts being a decision made per request, with a record of what was decided.

The field didn't arrive at "just pick a model per request" by one path. Worth a quick survey, because you'll see all of these described as "routing" and they are not the same idea. Rules-based routing, hand-written if-this-then-that logic. Task or prompt classification, tag the request, route by tag. Cascades, try the cheap model first and escalate only on failure. Semantic or embedding routing, match the request to a model by similarity. And predictive or learned routing, a system trained on outcomes that gets better at the assignment over time. An AWS machine learning blog post catalogs most of these under one banner; the research team behind RouteLLM reported that a learned router can land close to a strong model's quality at meaningfully lower cost on mixed traffic, though that's one paper's numbers on one benchmark, not a universal law, and it's worth treating as a data point rather than gospel.

Why a single hardcoded model is your biggest single point of failure#

Two separate problems, and most teams only notice the first one until the second one takes their app down at 2am.

The first is churn. Models get deprecated, repriced, and dethroned on a cadence measured in months, not years. The model you picked as "the best one" in January is not the best one by summer, and the provider isn't obligated to warn you before the number on the invoice changes or the endpoint gets sunset. "Best model" isn't a fact you get to bank. It's a moving target, and if your code has that target's name burned into it, you inherit the churn as your own maintenance chore. We wrote a whole piece on just how fast that shelf life turns over, worth a read if you want the receipts: <a href="/blog/model-shelf-life">why the best model keeps changing</a>.

The second is availability, and it's the one that actually pages you. A single provider having a bad hour takes your entire product down with it if that provider is your only path to an answer. Your reliability is capped at your narrowest single point, full stop. Doesn't matter how good your code is elsewhere. If the one model you call goes dark, so does everything downstream of it. We cover the specific version of this that keeps happening in production: <a href="/blog/one-provider-one-bad-day">what one provider's bad day costs you</a>.

Neither of these is a hypothetical. Both are just what happens when you treat an input like a fixed dependency. The fix isn't "pick a better model." There isn't a permanently better model. The fix is to stop needing there to be one.

What a router actually does for you: the four jobs#

Behavior, not magic. Here's each of the four jobs stated as something you can actually observe.

Normalize is the boring one and the one that saves you the most actual engineering time. Flux is one OpenAI-compatible API and key across a wide catalog of providers, with the base URL at https://api.fluxrouter.ai/v1, and an Anthropic-SDK-compatible path at https://api.fluxrouter.ai/anthropic if that's the shape your code already speaks. One integration, many backing models, and you never rewrite your client library because a provider changed their SDK again.

Choose is the part everyone assumes is the whole product, and it's genuinely one of four. Cost-attribute matters more than people think it does until they've tried to explain a monthly AI bill to a finance person using nothing but a blended per-token average, which tells you almost nothing about which feature or which customer actually drove the spend. A number on the response, per call, fixes that conversation before it starts.

Observe and failover is the job that actually saves your weekend. Flux routes across a broad catalog of models and multiple providers on one key, and when an upstream breaks, that becomes a routing decision instead of your outage. Flip that around and it's not just insurance, it's a reliability feature: your uptime stops being capped at your single narrowest dependency, the one provider having a bad hour. Worth saying plainly, though. This only works if there's actual breadth behind the router. A router that fronts one provider and calls itself resilient is marketing, not architecture. Ask what's actually behind the wall before you trust the failover story. Full detail on the failure mode this solves: <a href="/blog/one-provider-one-bad-day">what one provider's bad day costs you</a>.

Rolling all four of these yourself, the normalization layer, the per-provider retry logic, the cost accounting, the health checks, is a real engineering project. Somebody on your team becomes a part-time plumber for infrastructure that isn't your product. We've written about that tax specifically: <a href="/blog/engineers-plumbers">the plumbing tax of rolling your own</a>.

Smart routing, adaptive selection, and verified work#

The "choose" job deserves its own section because this is where the interesting behavior actually lives, and where I'm going to be upfront about a limit.

flux-auto looks at each request and picks a model sized for the work in front of it. Light work gets routed to something cheap and fast. Harder work gets routed to something stronger. That's the whole idea, and it's adaptive, meaning it learns from live results and keeps improving its picks over time, and it's cost-aware.

I'm not going to walk you through exactly how that decision gets made under the hood. That's not me being cagey for the sake of it, it's the actual moat, and I built it, so no. What I'll tell you instead is what you can observe: it's adaptive, it learns from results, and you can watch its picks change over identical calls, which is covered later in this guide with an actual header dump.

Now the agentic angle, because this matters more every month. Agent workloads don't make one call. They make dozens, sometimes hundreds, chained across a task. Which means the unit that matters isn't the token, it's the answer. A cheap model that gets the wrong answer and forces three retries cost you more than the expensive model that got it right the first time. We go deeper on that math here: <a href="/blog/count-answers-not-tokens">count answers, not tokens</a>.

And sometimes an answer matters enough that you want more than a good guess. That's what verified work is for. On the calls where being wrong is expensive, Flux can double-check the answer before it comes back. The receipt is stamped checked. Not verified. Words matter here, and that one's deliberate. There's a whole guide on exactly when that trade is worth it: <a href="/blog/quality-cliff">when a cheaper model falls off the quality cliff</a>.

Auto-route vs. pin: adaptivity vs. determinism#

This is the trade-off the whole guide keeps circling back to, so let's put it flat on the table.

flux-auto, or a tier alias like flux-fast, flux-standard, or flux-reasoning, right-sizes each request. A flux-pinned-* id locks one exact model, every single time, no substitution. Both routes run through the exact same metered key. Nothing architecturally different about how you call one versus the other.

Pin when reproducibility is the point. If you're running evals, a regression suite, or a golden set, you need the same model answering the same way every time, or your comparisons are noise. Same story if you need a hard capability guarantee, a specific output format, or a long-context model and nothing else will do. Reach for flux-auto or a tier alias for everything else, general chat, content generation, mixed traffic where any capable model in the pool gets the job done.

And the router has to tell you the truth about which one you got. That's the part most people miss. X-Flux-Routed reports false when a pin was honored, true when the router substituted something. You don't have to trust that it obeyed you. You read it off the response. Full docs on the trade-off: <a href="/docs/concepts/auto-vs-pinning">flux-auto vs pinning a model</a>. And the live catalog of usable ids, pinned and otherwise, is always GET /v1/models against your own key, which is the only authoritative source since the list itself changes.

When you should NOT use a router#

Every vendor page skips this section, which is exactly why it's the most useful one in this guide.

Routing adds a hop and an adaptive variable to your call path. That's a fair trade in most production systems, but not all of them. If you have exactly one model and one provider and you're genuinely happy, no complaints about cost, no complaints about uptime, a router adds a layer of indirection for a problem you don't have. Don't add it.

If you're running against a hard real-time latency budget where a single extra network hop is the difference between meeting an SLA and missing it, that's a real cost, and it's worth measuring before you route. There's a full breakdown of that trade-off here: <a href="/blog/latency-cost-quality-triangle">the cost, latency, and quality triangle</a>.

If you're in a strict reproducibility or compliance environment where you want exactly one model and nothing adaptive involved, ever, the honest answer is to pin, or in some cases skip the router entirely and go direct to the provider. Pinning gets you most of the normalization and observability benefits without the adaptivity, which is often the right compromise. Going fully direct is the right call less often than people assume, but it does exist as an option, and I'd rather tell you that than pretend routing is universally correct.

One more honest caution, and it applies even when routing is the right call: adaptive systems can regress silently if nobody's reading the receipts. A router that's quietly making worse picks looks exactly like a router that's working fine, unless you're actually watching the headers on your calls. If you route, route with the receipts on. That's not a slogan, it's the only way you'd ever catch a regression before a customer does.

LLM router vs. AI gateway: what's the actual difference#

This is the confusion that trips up almost every team evaluating this space, and almost nobody writes about it cleanly, so here's the one-line test.

Are you asking which model answers a request, or are you asking about the key, the spend limits, and the logging that wraps around all your model traffic? The first question is a router question. The second is a gateway question.

A router is specifically about model selection: which one answers this particular call. An AI gateway is the broader control plane around that: one key, authentication, spend limits, logging, policy, all the traffic-management stuff that has nothing to do with which model gets picked. In practice, most products that call themselves a router also do gateway things, one key and normalized auth being the obvious overlap, and most products that call themselves a gateway also do some form of model selection. The terms blur on purpose because the products blur on purpose. Flux, for what it's worth, is one API and key across many providers with routing built in, which puts it on both sides of that line.

Run the test against one actual call and it stops being abstract. You hit https://api.fluxrouter.ai/v1/chat/completions with model: flux-auto. Ask "which model actually answered that one" and you're asking the router question, and X-Flux-Model on the response is the answer. Ask "how much of my team's spend did that call count against, and how close is the account to its spend ceiling" and you're asking the gateway question, and nothing in the response headers touches that, it lives in billing and account controls instead. Same call, two completely different questions, two different parts of the stack answering them. Most vendor pages answer one and go quiet about which one, which is the whole reason this section exists.

If you're evaluating vendors and the sales page just says "gateway" or just says "router" without telling you which of these two questions it actually answers, ask directly. We're building out a dedicated guide to the gateway side of this specifically, the deeper cut on one key, spend limits, and policy, lives there when it ships. In the meantime, the build-it-yourself version of that question is covered here: <a href="/blog/build-vs-buy-gateway">build vs buy the gateway</a>.

How to trust a router: what one live response tells you#

A router earns trust by what it shows you about one live request. Not by its pricing page. Not by a logo wall of providers. A response, right now, with facts on it you can check.

There are five documented headers, and they're the whole trust mechanism. X-Flux-Model names the model that actually served the call. X-Flux-Original-Model names what you asked for. X-Flux-Routed tells you whether it substituted, true if it did, false if your pin was honored. X-Flux-Request-Id is the handle you quote if you ever need to contact support about a specific call. X-Flux-Cost-Usd is the per-request cost, present on non-streaming responses. Five facts, on the request itself, that a pricing page can't fake because they're generated by the actual call you just made.

Here's the exact curl. Point it at a non-streaming call and filter for just those five:

curl -sD - "https://api.fluxrouter.ai/v1/chat/completions" \
  -H "Authorization: Bearer $FLUX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "flux-auto", "messages": [{"role": "user", "content": "Reply with the word ok."}]}' \
  -o /dev/null | grep -iE '^x-flux-(model|original-model|routed|request-id|cost-usd): '

A probe run on July 16, 2026 had this shape:

X-Flux-Model: <whichever model in the pool actually answered>
X-Flux-Original-Model: flux-auto
X-Flux-Routed: true
X-Flux-Request-Id: <uuid>
X-Flux-Cost-Usd: <per-call cost, a fraction of a cent for a short reply>

Run that same call again tomorrow and the model named will probably be different, because routing is adaptive and the picks change with live results. That's not a bug in the demo, that's the actual point. The header count matters too. Filter for the five named above, specifically, and ignore the rest.

Pair this with the checked-not-verified discipline from the section above and you've got the trust picture in full: what answered, what you asked for, whether it substituted, and what it cost. Full header reference: <a href="/docs/concepts/transparency-headers">the transparency headers</a>.

How to evaluate any router: a buyer's checklist#

Run this against any router. Flux included. This is the moat move, actually, we're publishing the test that could make us look bad, because the alternative is asking you to trust a pricing page, and that's worth less than nothing.

The criteria that count are the ones a router literally cannot fake in a live response. Here's the compact version:

Weight the top three the heaviest. Anything a vendor can only tell you about in a sales call, instead of showing you on a live response, isn't a receipt, it's a claim. And leaderboards won't settle this for you either, benchmarks measure something, but rarely the thing your actual traffic needs: <a href="/blog/benchmarks-lying">why leaderboards won't pick your model</a>.

If you want the extended version of this checklist with a scorecard you fill in yourself, that's coming as its own dedicated guide. Until then, the five rows above will get you further than most vendor evaluations manage in a week.

Here's the part that actually matters, though. You already know how to run the five-header probe on Flux, it's two sections up. Now go run that same probe against whatever router you're currently evaluating, ours included. Point their SDK at their base URL, send one call, and see if the response even carries five comparable facts or just a model name and a shrug. A router that won't show you which model answered and what it cost is asking for trust it hasn't earned yet.

If cost tracking across a whole team, not just one call, is the part you're stuck on, that's its own problem with its own guide: <a href="/blog/nobody-knows-cost">nobody knows what the AI bill actually costs</a>.

FAQ#

What is an LLM router?#

A system that sits behind one API endpoint and picks a suitable model for each request at runtime, instead of your code calling one hardcoded model name. It normalizes the API across providers, attributes cost per call, and can fail over if an upstream breaks.

How does LLM routing work?#

Behavior-wise: each request gets evaluated and routed to a model sized for that specific job, with the router adapting its picks over time based on live results. You can also pin an exact model when you want determinism instead of adaptivity.

What is the difference between an LLM router and an AI gateway?#

A router answers "which model serves this request." A gateway is the broader control plane around all your model traffic, the key, auth, spend limits, and logging. Most real products do some of both.

When should I NOT use a router?#

When you have one model and one provider and are genuinely happy with it, when a hard real-time latency budget makes the extra hop too costly, or when strict reproducibility or compliance needs mean you want exactly one model and nothing adaptive. Pin, or in some cases go direct.

How do I know which model actually answered my request?#

Read X-Flux-Model off the response. It names the model that served the call, and X-Flux-Original-Model shows what you actually asked for, so you can see if the router substituted.

Can I still force one specific model?#

Yes. Call a flux-pinned-* id and the router locks to that exact model every time, with X-Flux-Routed: false confirming the pin was honored. Same metered key either way.