Build vs Buy: Should You Run Your Own LLM Gateway?

By Sean Donahoe · Published July 16, 2026 · accurate as of this date

Should you build your own LLM gateway?#

Build it if routing is your actual product, if a hard air-gap or data-residency or compliance rule keeps a third party out of the request path, or if you already fund a platform team with real 3am on-call. Otherwise, buy. For most teams the models are an input, the provider list keeps moving under you, and a self-hosted gateway quietly becomes a tier-0 dependency nobody signed up to operate.

That's the whole answer. The rest of this is how to tell which one you actually are, because most teams guess wrong.

And here's the tell you should trust more than my opinion: I run a router company. We build and operate our own gateway on purpose.A vendor in my seat is supposed to tell you nobody should ever build. I'm telling you when you should. That's the useful version of this decision, and it's the one nobody selling you a gateway wants to write down.

The debate is never really "build vs buy"#

Nine times out of ten, "should we build a gateway?" isn't a greenfield question. It's a team that already started. There's a thin OpenAI-compatible wrapper somebody stood up in an afternoon, it's been creaking for a month, and the real question on the table is "do we keep extending this thing or do we go get a real one?"

So reframe it. Forget build versus buy in the abstract. Ask one question instead: is routing your PRODUCT, or is it your PLUMBING?

That single question decides most of it. Everything below is just working out which side of it you're on.

The proxy is the small part of the job#

Here's where the estimate goes wrong. Someone says "it's an OpenAI-compatible passthrough, give me a week." And they're right about the passthrough. A passthrough is a week.

The passthrough is the part you can see. Call it a fifth of the whole gateway, give or take. Most of the job is the operations you inherit the moment real traffic hits it:

• Provider normalization, so every model looks the same to your app
• Failover and retries with backoff when a provider has a bad ten minutes
• Per-request cost attribution, so finance and your own debugging both work
• Key management and rotation across every provider you touch
• Rate-limit handling that doesn't fall over under load
• Streaming edge cases, their own small hell: a connection dropped mid-response, a half-formed chunk that isn't valid JSON yet, the token count you still have to get right when the stream dies early
• Observability you can actually debug an incident with
• And keeping ALL of it current as providers ship breaking changes basically every month

You don't have to take my word on how big the normalization slab alone is. Look at LiteLLM, the best-known open-source gateway. Its own docs describe it as one OpenAI-format interface to 100-plus LLM APIs, with cost tracking, load balancing and logging bundled in.That project exists specifically to absorb this work, and it's a lot of project. Someone already ate that meal. The only real question is whether you want to sit down and eat it again yourself.

This isn't the "wiring is a tax" argument, by the way. I made that case elsewhere. This one's narrower and duller and more important: scope realism. The demo is the small, visible part. The bill is everything under it, and it arrives later.

Nobody budgets for the part that pages you#

Here's the line item that never makes it into the estimate. The second you self-host a gateway, every model call in the company flows through code you now operate. All of it. Support bot, internal tools, the feature customers pay for, the batch job at 2am.

So when the gateway is down, everything is down.

You didn't build a convenience. You built a single point of failure with an on-call rota attached to it. That's not a reason never to do it. Plenty of teams should own exactly that. It's a reason to answer one question before you commit: who gets paged at 3am when the gateway itself is the outage, and are they real, and are they funded?

Because a gateway is cheap to stand up and expensive to run. The day-one demo costs you an afternoon. Day two hundred costs you upgrades, provider churn, incident response, and cost-attribution that drifts the moment you stop babysitting it. The demo is not the thing. Running it for a year is the thing.

When building actually wins#

I said I'd tell you when to build, so here it is, plainly. Four cases. If you're in one of them, build, and don't let a vendor talk you out of it.

Routing is your product. You're building a router, an eval platform, a model-arbitrage play, something where the routing behavior IS the thing customers pay for. Buy it and you've outsourced your moat. This is literally why we built ours. For us, routing is the product, not the plumbing, so owning it isn't optional. Same test I'm handing you.

You have a hard constraint a vendor can't meet. Air-gapped or on-prem. A specific data-residency jurisdiction. A compliance regime that forbids a third party sitting in the request path at all. These aren't preferences you can negotiate. If the request legally can't leave your walls, no managed gateway helps you, and you build.

You have the team AND the rota. Not "a senior who's keen to have a crack at it." A funded platform group with genuine spare capacity and real 24/7 on-call. If that function exists and has room, running a gateway is a normal thing for it to own. If it doesn't, you're volunteering someone's nights.

You only use one provider, and it's stable. Then don't build a gateway at all. It's a 40-line wrapper around one API. Buying a multi-provider router for a single stable provider is overkill, and building a multi-provider router for it is worse. Write the tiny thing and move on.

When buying wins, which is most of you#

Now the larger, less flattering majority. You should buy if the models are an INPUT to your product rather than the product itself. If the words your app generates matter and the plumbing that fetches them doesn't, you're a buyer.

You should buy if the provider list changes under you and you'd rather that churn not live on your roadmap. New model drops, a provider deprecates something, prices move. Someone has to keep up with that forever. If you'd rather it not be your team, buy.

And you should buy if you'd prefer your best engineers shipping features over operating infrastructure that doesn't make your product any better. That's not a slur on infra work. It's just that a gateway you didn't sell doesn't differentiate you, and your seniors' time is the most expensive thing you own.

The thing you get for buying is the whole rest of that list handed to you. A managed gateway like Flux is one endpoint that speaks both OpenAI and Anthropic formats, on one key.It routes and falls back across providers behind that endpoint.And it puts per-request cost and which model actually answered right there on the response, so you get the receipts without building the accounting yourself.That's the failover and the cost attribution and the normalization from the list above, except you operate none of it.

The decision checklist#

Run this on yourself. It's ours, it's free to use, and it's built to be honest rather than to sell. Count your answers.

One honest middle path, because it's where a lot of teams actually land. Open-source self-host, LiteLLM being the obvious example, is "build" with a running start. It hands you the normalization slab so you don't re-eat that meal. But it doesn't move you off the line, it just moves you along it. You still own operation, upgrades, and the on-call. Self-host is still the build column. It's just build minus the worst week of it.

So which are you#

That's the whole call, and it really does come down to the one question. If routing is your moat, build it and own it. We did, and I'd make the same choice again tomorrow.

If routing is your plumbing, don't hand your best engineers a second product to maintain. They'll still be maintaining it next quarter, and the quarter after that, long after everyone forgot it was supposed to be a week.

If you landed on buy and want to see what the bought version looks like, the Flux quickstart is a base-URL-and-key change from where you probably already are. And if you want the argument underneath this one, why the plumbing keeps costing you your best people, that's over in engineers and plumbers.

Receipts#

• LiteLLM is an open-source, self-hostable LLM gateway you can run yourself, per LiteLLM's own docs and GitHub README (docs.litellm.ai, BerriAI/litellm), fetched July 16, 2026.
• LiteLLM describes itself as a single OpenAI-format interface to 100-plus LLM APIs, bundling cost tracking, load balancing and logging, per its own docs and README, fetched July 16, 2026.
• FluxRouter is one endpoint that speaks both OpenAI and Anthropic formats, on one key: Flux quickstart.
• FluxRouter routes and falls back across providers behind that endpoint (behavior only).
• FluxRouter's router is built and operated in-house.
• FluxRouter surfaces per-request cost and which model answered on the response: transparency headers.
• LiteLLM, category example: https://docs.litellm.ai/docs/