By Sean Donahoe · Published July 17, 2026 · accurate as of this date
What is an LLM gateway?
An LLM gateway is a single API endpoint and one key sitting in front of many model providers. It normalizes the request shape across them, routes each call to a model, records what happened and what it cost, and fails over when a provider breaks. That's the whole definition. Everything past this paragraph is just one of those four jobs, examined properly.
Here's the longer version, because "single endpoint" undersells what's actually going on. The moment your app talks to more than one model or more than one provider, you inherit a pile of plumbing that has nothing to do with your product: separate auth for each provider, separate key rotation, separate request quirks, separate cost tracking, and a failover plan you have to write yourself if you want one. A gateway is where that plumbing lives instead of your codebase. We've written the long version of that tax specifically, worth a read if you're the one currently owning it: <a href="/blog/engineers-plumbers">the plumbing tax of doing it yourself</a>.
Most teams don't decide to build a gateway. They back into one. First it's a fallback for when the primary provider hiccups. Then it's a second model because the first one's expensive for easy requests. Then someone in finance asks what the AI spend actually costs by feature, and the honest answer is a shrug. None of that plumbing was ever the plan. It just accreted, request by request, until somebody had to own it.
What a gateway is not, and this matters more than the definition does. It's not a model. It's not a fine-tune. It won't make any single answer better on its own, because plenty of the time it picks the exact model you'd have picked yourself anyway. And it's not the same thing as grabbing an API key from one provider and calling it done. That's a client integration. One provider, one set of quirks, none of the plumbing shared across anything else. A gateway only earns its keep once there's more than one provider to manage. If you want the deep cut on the routing decision specifically, the piece that's entirely about which model answers a given call rather than the control plane wrapped around it, that's here: <a href="/blog/llm-routing-guide">the complete guide to LLM routing</a>.
LLM gateway vs LLM router vs raw provider SDK
Almost every page in this category blurs these three, and it's the single most useful thing to get straight before anything else clicks. Three things, not two.
A raw provider SDK talks to one provider's models directly. It's the simplest option and it adds zero extra hops. The tradeoff is you own every single provider you add, separately, forever, with none of it shared. A router answers one narrow question: which model serves this particular request. That's the selection decision, and only the selection decision. A gateway is the broader thing wrapping around all of it: one key, one endpoint, normalization across providers, cost attribution, logging, failover, the whole control plane your model traffic runs through. A gateway usually contains a router inside it, but the two words are not interchangeable.
Here's the one-line test, and it's the one worth actually remembering. Are you asking which model answers this request, or are you asking about the key, the endpoint, the limits, or the logging that wraps around every model you call? The first is a router question. The second is a gateway question.
Worth saying plainly, though: most real products labelled "gateway" do both. And "proxy" gets used loosely for the same idea by people who mean either one, or neither one precisely, which is exactly why the confusion persists. If you're comparing vendors and their sales page just says "gateway" or just says "router" without telling you which of these two questions it actually answers, ask them directly before you sign anything. We wrote the deeper cut on the selection half of this on its own, because it deserves the space: <a href="/blog/llm-routing-guide">the complete guide to LLM routing</a>.
What a gateway actually does for you
Behavior you can watch, not a feature you have to take on faith. Four things, and here's each one as something observable rather than something asserted.
| Job | What it means | What you actually see |
|---|---|---|
| One key, one endpoint, one invoice | An OpenAI-compatible base URL and key across many providers, plus an Anthropic-SDK path, so adding a provider isn't a new integration | You point your existing SDK at one base URL instead of stitching in a new client for every provider |
| Provider normalization | The same request shape reaches different providers' models | You write to one shape; the gateway absorbs each provider's quirks |
| Failover | Breadth across providers turns a bad upstream hour into a routing decision, not your outage | A bad upstream becomes a routing decision, not an automatic outage |
| Live cost receipts | Per-request cost surfaced on the response itself | A cost number on the call you just made, not something you back into from a monthly bill |
Take the first one seriously, because it's the boring one and also the one that saves the most actual engineering hours. Flux is one OpenAI-compatible API and key across a wide catalog of providers, base URL at https://api.fluxrouter.ai/v1, plus an Anthropic-SDK-compatible path at https://api.fluxrouter.ai/anthropic if that's the shape your code already speaks. One integration. Every new provider after that is a model id, not a new SDK, a new auth flow, and a new set of edge cases to learn.
Normalization is what makes that first point actually work instead of just sounding nice on a slide. Different providers shape their requests and responses slightly differently, and a gateway absorbs that so your code doesn't have to branch on which provider happens to be serving a given call.
Failover is the one that actually saves your weekend, and it's worth being blunt about why. If your app calls a single provider directly and that provider has a bad hour, your app has a bad hour. Breadth across multiple providers, all reachable through one key, turns that into a routing decision instead of an outage you're paging people about at 2am. This is behavior, not a mechanism I'm going to walk you through. What matters is what you can observe: a bad upstream stops being your single point of failure.
And live cost receipts are the fourth job, the one nobody's dashboard actually does well. Per-request cost on the response itself beats a monthly invoice you have to reverse-engineer into "which feature actually drove this." The evaluation section below shows you exactly what that looks like on the wire, with a runnable curl and real header names, not a screenshot of a dashboard. If cost visibility specifically is the part you're stuck on, we cover it end to end: <a href="/blog/nobody-knows-cost">why nobody knows what their AI actually costs</a>, and the companion piece on the framing that actually fixes it: <a href="/blog/ai-cost-control-playbook">cost per answer, not per token</a>.
When you do NOT need one
This is the section most vendor pages skip entirely, and skipping it is exactly why nobody trusts vendor pages on this topic. So let's go harder than the ones who at least attempt it.
If you call one model from one provider and have no plans to change, a gateway adds a hop and a moving part for nothing. Use the direct SDK. It's genuinely simpler, it adds no network overhead the gateway would introduce, and it's one fewer thing that can break between your app and the model. There's no honest version of this guide that tells you otherwise just because it happens to be selling one.
A few more cases where the honest answer is skip it, or at least pause before you add it. Hard real-time latency budgets, where a single extra network hop is genuinely the difference between meeting an SLA and missing it. That's a real cost, not a hypothetical one, and it's worth measuring before you route anything through an extra layer. Strict single-model reproducibility or compliance requirements, where you want exactly one model and nothing adaptive touching the decision, ever. In that case pin one model through a gateway if you still want the cost and observability benefits, or go fully direct if you don't want the gateway in the path at all. Both are legitimate calls depending on how tight the constraint actually is.
And here's the caution that applies even when you decide a gateway is the right call. Any control plane you can't read live receipts from is a black box you're trusting on faith. A vendor telling you their product is great is not evidence. A vendor showing you what happened on the actual call you just made is.
OpenRouter's blog makes a similar point, and it's worth crediting directly rather than pretending nobody's said it before. In a June 2026 post, they make roughly the same argument: one model, one provider, no plans to change, and a gateway just adds complexity you don't need, with no latency win to show for it. They got there first. Fine. The differentiator isn't who said the honest thing first, it's whether you can back it with a live receipt instead of just a paragraph, which is exactly what the evaluation section below walks you through. If you want the deeper build-vs-buy version of this decision once you've decided you do need one, that's here: <a href="/blog/build-vs-buy-gateway">build vs buy the gateway</a>, and the latency tradeoff specifically gets its own full breakdown here: <a href="/blog/latency-cost-quality-triangle">the latency, cost, and quality triangle</a>.
When you DO need one: the four triggers
Mirror image of the section above. Four concrete triggers, and if any one of them is true right now, a gateway is doing real work for you, not just adding ceremony.
- More than one provider or model, or you expect to add one soon. The plumbing tax multiplies with every provider you add on your own: separate auth, separate quirks, separate everything. A gateway is where that plumbing consolidates instead of forking across your codebase.
- Cost visibility you can't get from N separate dashboards. If your AI spend spans more than one provider, reconstructing "what did this feature actually cost" from N invoices is a spreadsheet nobody wants to own. A single per-request receipt fixes that at the source.
- Failover and reliability actually matter to you. If one provider's bad hour being your outage is unacceptable, breadth across providers behind one key is the only thing that changes that math.
- Provider churn insulation. Models get deprecated, repriced, and dethroned on a cadence measured in months. "The best model" is a moving target, and if that churn has to be your code's problem to babysit on your provider's schedule, that's a maintenance tax you didn't sign up for. Models are an input, not your product.
Each of those deserves its own full read if it's the one that landed. Reliability specifically: <a href="/blog/one-provider-one-bad-day">what one provider's bad day costs you</a>. Churn specifically: <a href="/blog/model-shelf-life">why the best model keeps changing</a>. Cost specifically: <a href="/blog/nobody-knows-cost">why nobody knows what their AI actually costs</a>. And the broader cost framework this all sits under: <a href="/blog/ai-cost-control-playbook">cost per answer, not per token</a>.
Build vs buy a gateway
Once you've decided you actually need one, the next question isn't "which vendor," it's "do I build this myself or rent it." Worth being honest about what you're actually choosing between, because both sides have real costs.
Building it yourself means maintaining auth and normalization for every provider you add, writing and then owning your own failover logic, and building cost attribution that stays correct as providers change their pricing pages without warning you. None of that is a one-time cost. It's ongoing maintenance, and somebody on your team becomes the part-time plumber for infrastructure that isn't your actual product. We've written the long version of exactly that tax: <a href="/blog/engineers-plumbers">the plumbing tax of doing it yourself</a>.
Here's the part that doesn't show up on the whiteboard the day you decide to build it. It's not the initial integration, that part's usually fine, a week, maybe two. It's month four, when a provider quietly reshapes their pricing tiers and your cost-attribution code is now confidently wrong in production, and nobody notices until finance asks why the AI line item doesn't add up. It's the on-call rotation absorbing "provider X is having a bad morning" as a page, instead of a routing event you'd never have seen if a gateway had eaten it upstream. That's the real bill for building it yourself, and it doesn't show up until you're already paying it.
Buying it means a dependency, an extra hop, and trust in someone else's control plane. Which is precisely why the receipts matter so much, because trust in a black box you didn't build is trust you have to actually earn a reason to give. The evaluation section below is the test you run before you hand anyone that trust.
There's no universal answer here, and anyone telling you there is one is selling something. What there is instead is a set of decision axes: how many providers are you actually touching right now, how much is the maintenance burden of building it worth against your team's time, how much does reliability actually matter to your specific product, and how comfortable are you running a control plane you can't inspect. Answer those honestly and the build-vs-buy call mostly answers itself.
Picture the two ends of that spectrum. A two-person team shipping one feature on one model has no business building this. The maintenance alone would eat more of their week than the feature they're actually trying to ship. A fifteen-engineer platform team routing a meaningful chunk of their infrastructure through five providers, with a team who'd rather own the failure modes than trust someone else's, has a legitimate case for building. Most companies asking this question sit closer to the first end than they'd like to admit.
Full breakdown of the tradeoff: <a href="/blog/build-vs-buy-gateway">build vs buy the gateway</a>.
How to move to one
Assuming you've landed on buy, here's the part that actually matters before you commit: what it costs to switch, and what it costs to leave.
A gateway is, in the overwhelming majority of cases, a drop-in base-URL swap. Point your existing OpenAI SDK, or your Anthropic SDK, at the gateway's base URL, keep your endpoints and your JSON shape exactly as they are, and you're done. The interesting part isn't that the switch is small. It's that the cost to leave is the same small diff as the cost to switch in, which is itself a trust argument worth sitting with. If a vendor's own architecture makes leaving expensive, that's a signal about what they're actually confident in.
If you're currently on another gateway and weighing a move, we've published the specific version of that walkthrough: <a href="/blog/openrouter-migration">moving off OpenRouter to a gateway</a>. There are dedicated guides for moving off a raw OpenAI integration, <a href="/blog/migrate-from-openai">migrating from OpenAI</a>, and off a raw Anthropic integration, <a href="/blog/migrate-from-anthropic">migrating from Anthropic</a>. Guides for moving off a self-hosted / DIY gateway and a direct head-to-head against OpenRouter are in progress and will slot in here as spokes once they ship. For the canonical how-to on the swap itself, right now, the getting-started doc has the exact steps: <a href="/docs/getting-started/quickstart">point your SDK at the base URL</a>.
How to evaluate a gateway before you trust it
Here's the moat move of this whole guide, and it's the section that actually separates a real gateway from a sales page. A gateway earns trust by what it shows you about a single live request. Not its pricing page. Not a logo wall. A response, right now, with facts on it you can check yourself.
There are five documented transparency headers on Flux, and every one of them comes straight off a live response, not off a sales page. X-Flux-Model names the model that actually served the call. X-Flux-Original-Model names what you asked for. X-Flux-Routed tells you whether it substituted, true if it did, false if your pin was honored. X-Flux-Request-Id is the handle you quote if you ever contact support about a specific call. X-Flux-Cost-Usd is the per-request cost, present on non-streaming responses. Five facts, generated by the actual call you just made, which is exactly why a pricing page can't fake them.
Here's the runnable version. Point it at a non-streaming call and filter for just those five:
curl -sD - "https://api.fluxrouter.ai/v1/chat/completions" \
-H "Authorization: Bearer $FLUX_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "flux-auto", "messages": [{"role": "user", "content": "Reply with the word ok."}]}' \
-o /dev/null | grep -iE '^x-flux-(model|original-model|routed|request-id|cost-usd): '
A live probe against that exact call came back shaped like this on July 17, 2026. The five header names are the documented, stable part of the response, not a per-call fact, so it's still exactly what you'll get if you run this today:
X-Flux-Model: <the specific model that answered this call>
X-Flux-Original-Model: flux-auto
X-Flux-Routed: true
X-Flux-Request-Id: <uuid>
X-Flux-Cost-Usd: <per-call cost in USD>
Run that exact same call again tomorrow and the model named will likely be different, because the routing behind it is adaptive and its picks change with live results. That's not a flaw in the demo. That's the actual point of showing you this instead of a static screenshot.
Now turn it into a rubric you can run against any gateway, Flux included, before you route production traffic through it.
| Check | The test | Why it's the one that matters |
|---|---|---|
| Names the model that answered | Read the model-served header off a real call | A pricing page can't fake a header generated by the actual response |
| Shows per-request cost | Read the cost header off the same call | A blended monthly average tells you nothing about what one call cost |
| Tells you when it substituted | Read the routed/substitution header | The only honest signal a pin was or wasn't respected |
| Lets you pin an exact model | Call a pinned id and confirm the substitution flag reads false | Determinism has to be a real, testable option, not a promise on a slide |
| Lets you leave in an afternoon | Swap the base URL and key, nothing else changes | Same SDK, same JSON, exit cost equal to entry cost |
Weight the top three the hardest. Anything a vendor can only tell you about on a sales call, instead of showing you on a live response, isn't a receipt. It's a claim. Deeper evaluation walkthroughs and a full header reference are coming as dedicated guides; in the meantime the canonical doc has every header documented in full: <a href="/docs/concepts/transparency-headers">the transparency headers</a>.
Before you route production traffic through any gateway, Flux included, run this exact probe against it first. A gateway that won't show you which model answered, whether it substituted, and what the call cost is asking for trust it hasn't earned yet.
A quick tour of what a good gateway gives you back
Keeping this one short on purpose, because the point of this guide is the definition, not a feature dump. But it's fair to show you one gateway that actually passes its own checklist above.
flux-auto right-sizes each request to a model suited for the work in front of it, and it's adaptive, meaning it learns from live results and keeps improving its picks over time. If you want the opposite of adaptive, a flux-pinned-* id locks one exact model, every call, no substitution, running through that same metered key. Full detail on that tradeoff and when to reach for which: <a href="/docs/concepts/auto-vs-pinning">auto-route vs pin a model</a>. That's the whole pitch, honestly. Not "trust us." Read the receipt.
FAQ
What is an LLM gateway?
A single API endpoint and one key sitting in front of many model providers. It normalizes the request shape across them, routes each call to a model, records what happened and what it cost, and fails over when a provider breaks.
What is the difference between an LLM gateway and an LLM router?
A router answers one question: which model serves this request. A gateway is the broader control plane wrapping around all your model traffic, one key, one endpoint, normalization, cost attribution, logging, and failover, and it usually contains a router inside it. The deeper cut on the routing half specifically lives here: <a href="/blog/llm-routing-guide">the complete guide to LLM routing</a>.
What is the difference between an LLM gateway and an AI gateway?
In practice, not much, and the terms are used loosely enough by vendors that you shouldn't read too much into which word a given product picked. Both describe the same control-plane idea: one key and endpoint wrapping normalization, routing, cost tracking, and failover across many model providers. If a vendor's page uses one term and not the other, that's branding, not a technical distinction worth chasing.
Do I need an LLM gateway?
If you call more than one model or provider, need cost visibility across them, care about reliability, or don't want provider churn to become your code's problem, yes. If you're on one model from one provider with no plans to change, no.
When should I NOT use an LLM gateway?
When you genuinely have one model and one provider with no plans to change, when a hard real-time latency budget makes the extra hop too costly, or when strict reproducibility or compliance needs mean you want exactly one model and nothing adaptive touching the decision. Pin, or go direct.
Is an LLM gateway just a proxy?
"Proxy" gets used loosely for the same control-plane idea, so functionally, often yes, though the term doesn't specify whether normalization, cost attribution, and failover are actually included. Ask what a specific product's "proxy" or "gateway" label actually covers rather than assuming from the word alone.
How do I know which model my gateway actually used?
Read it off the response, not a dashboard. On Flux, the model-served header names exactly which model answered, and the original-model header shows what you asked for, so you can see immediately whether it substituted. Run the five-header probe in the evaluation section above against any gateway you're considering, this one included, before you trust it with production traffic.