How to Choose an LLM: The Framework for Picking the Right Model for Your Workload

By Sean Donahoe · Published July 17, 2026 · accurate as of this date

How do I choose the right LLM for my workload?#

Stop trying to crown one best model. A leaderboard can't do that job, it measures generalized performance on somebody else's tasks and has no idea what your latency budget, your cost ceiling, or your definition of a right answer actually is. Define the job, test two or three candidates on your own traffic, and score what actually decides it: cost, speed, and whether it got YOUR work right.

That's not just a hot take. Amazon's own guide to picking a model for AWS says relying only on public benchmarks "is problematic: they measure generalized rather than domain-specific performance, prioritize easily quantifiable metrics over business-critical capabilities, and can't account for your organization's unique constraints around latency, costs, and safety requirements." Their own example lands the point better than I could: a high-ranking model might ace trivia and still faceplant on your industry's terminology. That's the whole problem in one sentence.

Here's the part that makes it worse.

The model you're benchmarking today might not even be the model you're running in three months, under the same name. Researchers at Stanford and Berkeley tracked GPT-4 across two 2023 snapshots and found real behavioral drift between March and June of that year, on math and code tasks alike. OpenAI pushed back hard on how that study framed things, and they had a fair point, some of what looked like a capability drop was measurement artifact, not the model actually getting dumber. Strip the disputed numbers out entirely and the underlying fact still survives the argument: a model name is not a fixed thing you benchmark once and trust forever. A ranking from six months ago describes a model that, in some real sense, doesn't exist anymore. We go deeper on exactly how those numbers mislead you here: <a href="/blog/benchmarks-lying">benchmarks are lying to you</a>.

So why not just check today's leaderboard instead? Because you can't out-fresh it, and you shouldn't try. Live leaderboards track hundreds of models continuously, scoring intelligence, price, and speed in something close to real time. They exist precisely because a static ranking rots within weeks. Use one as a first-pass filter, a way to knock a field of forty models down to five worth actually testing. Don't use it as a verdict. The leaderboard's job ends where your traffic begins, and that's where this page picks up.

What's the better question to ask than "which model is best"?#

"Which model is best" is the wrong question twice over. First, it has no stable answer, the price-quality frontier moves roughly monthly and a model you hardcoded in January is quietly a worse deal by summer. Second, "best" is meaningless without a job attached to it. Best at what? For whom? Measured how?

You don't have to take my word for the churn. Over roughly 48 hours in August 2024, GPT-4o's input price dropped from $5 to $2.50 per million tokens and Gemini 1.5 Flash got cut by around 79%, neither with much warning to the people already calling those APIs in production. That's the frontier moving in real time, not a hypothetical. Whatever prices are doing as you're reading this, they're doing something, and it isn't standing still.

Replace the question. Ask which model fits THIS workload, measured on MY traffic. That question has a stable, ownable answer, because it isn't trying to describe the whole market. It's describing one job.

A model is an input to your system. It is not the system. It is not your product, your moat, or the thing your customers are actually paying for, and inputs churn, they get repriced, deprecated, and quietly dethroned. Babysitting that churn by hand is a maintenance chore nobody put on your roadmap. And here's the tension I'm not going to dodge: you asked how to choose an LLM, so I'm going to answer it properly, define the job, test it, decide. But the honest answer for plenty of workloads is that you stop needing to choose at all. More on that once we've earned it, in the routing section below.

What actually decides whether a model fits your workload?#

Seven things. Weigh them against the job, never against a model's reputation.

Task difficulty. Most of your traffic is easy. A handful of requests are genuinely hard, and that handful is where a stronger, pricier model earns its keep. Match model strength to per-request difficulty and you stop overpaying on the ninety percent that never needed it. The team behind RouteLLM found a router hitting 95% of GPT-4's benchmark quality on MT-Bench while sending only 14% of queries to GPT-4 itself, the rest went to a cheaper model and did the job fine. Even read that number skeptically, one paper, one benchmark, and it still makes the point: most of what you're routing doesn't need your best model. It needs a model that's good enough for that specific request.

Cost, but per answer, not per token. Per-token price only means something once the answer is right. Retries, loops, and a wrong model on the wrong job multiply the real number fast, and a single agentic task can fan out to dozens of model calls before it's actually done. One published benchmark on coding tasks logged trajectories running up to roughly fifty model calls and forty-nine tool calls apiece, some past forty turns. Measure dollars per finished task. Not the sticker price per thousand tokens. Deeper math here: <a href="/blog/count-answers-not-tokens">count answers that worked, not tokens</a>.

Latency. Two numbers, not one: time to first token, and output speed once it's actually started. Which one you can afford to drop is a property of the job, not the model. Reasoning models will lie to you here if you're not careful, they can post a fast first token and then grind for another ten seconds doing the actual thinking, so "looks quick" and "is quick" are two different claims. Full trade-off: <a href="/blog/latency-cost-quality-triangle">latency vs cost vs quality, pick two</a>.

Reasoning models complicate this further, and it's worth flagging on its own. Left alone, they tend to apply roughly the same compute budget regardless of how easy the task actually is, and routinely overthink problems that didn't need it, that's an efficiency and cost problem, not proof they've hit some accuracy ceiling. The fix, on models that expose it, is a controllable depth knob rather than a blind assumption that more thinking equals a better answer. Cheaping out blind, without measuring, is how quality quietly cliffs on you: <a href="/blog/quality-cliff">cheaping out blind torches trust</a>.

Context length. How much does the job actually need stuffed into the window, and are you paying to stuff it in on every single call? A huge context window is not a free pass to skip retrieval.

Format and JSON reliability. For extraction and tool pipelines, whether a model emits valid structured output every time often outranks how smart it is in the abstract. A brilliant model that occasionally returns malformed JSON costs you more debugging time than a merely competent one that never does.

Vision and multimodal. Needed or not. A hard yes or no filter, and it'll knock half your shortlist out before you've spent a dollar testing anything.

Determinism. Do you need the exact same answer every time, or the best-fit answer per request? If it's "the same, always," you're not really choosing a model, you're choosing a pin, which is a different decision covered below. Some workloads want a second pass on top of that: you can ask for a qualifying answer to get double-checked before it comes back, and the receipt gets stamped "checked" when it passes, deliberately not "verified." Nice to have on the calls where being wrong is expensive. Not the default case for most traffic.

One more thing before you weigh any of this. The frontier price spread is enormous, which is exactly why "fits this job" beats "is best" so often. As of mid-2026 research, top frontier output pricing has run something like two orders of magnitude above the cheapest still-capable models, and even inside one vendor's own lineup the strong-versus-small spread runs several-fold, with the vendor's own docs telling you to reach for the small model on simple work. Paying frontier prices to answer "what's the weather like" is paying Ferrari prices to drive to the corner shop. Full teardown: <a href="/blog/ferrari-prices-corner-shop">paying Ferrari prices to drive to the corner shop</a>.

Here's the honest shape underneath all seven dimensions: latency, cost, and quality form a triangle, and you buy two. Which corner you drop is a decision about the job in front of you, never a fixed property of a model.

How do you test candidate models on your own traffic?#

Pull ten to twenty real tasks from your own traffic. Not benchmark tasks, not synthetic ones, the actual weird inputs your users send you on a Tuesday afternoon. Pick two or three candidate models. Run every candidate against every task. Decide your scoring rubric before you look at a single result, or you'll rationalize the model you already liked going in, and that defeats the entire point of running the test. We've written the discipline of fixing your rubric before you look up separately, worth the read before you run your first test: <a href="/blog/eval-before-trusting-model">eval before you trust a model</a>.

Score three things per task, per model: did it land the answer, what did the finished answer cost, how fast did it come back. Fill in a grid like this against your own tasks, not mine:

This is where a per-request receipt quietly saves you a spreadsheet. Fire each candidate through one endpoint and the cost of that specific call rides back stapled to the response, on a non-streaming Flux call there are exactly five documented headers, and filtering for just those looks like this:

curl -sD - "https://api.fluxrouter.ai/v1/chat/completions" \
  -H "Authorization: Bearer $FLUX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "flux-auto", "messages": [{"role": "user", "content": "Summarize this in one sentence: <your real task>"}]}' \
  -o /dev/null | grep -iE '^x-flux-(model|original-model|routed|request-id|cost-usd): '

Run that exact shape of call on July 17, 2026 and it came back served by deepseek-v4-flash, and the per-request cost swung hard on the same model inside a single day. I ran the identical prompt twice and it billed $0.000102 one call and $0.000948 another, roughly a 9x spread on the same model name, same afternoon. A paired request against flux-pinned-claude-sonnet came back marked x-flux-routed: false, proof the pin held. Those exact numbers are already stale by the time you're reading this, models get repriced and re-served constantly, that's the whole reason this page doesn't lean on them. The header read isn't stale, though. Run it against your own candidates right now and you get today's real numbers, which is the entire point of testing on your own traffic instead of trusting mine.

None of this requires buying anything from anyone, by the way. Any provider's SDK will let you fire the same prompt at two or three models and diff the results by hand. The receipt just saves you from reconstructing "what did that call actually cost" out of a monthly invoice three weeks later, when you've forgotten which feature drove it. If what you're actually testing is a router rather than a single model, the method shifts slightly, full walkthrough here: <a href="/blog/evaluate-llm-router">how to evaluate an LLM router</a>.

How often should you recheck your model choice?#

Whatever you pick, put a date on it. The frontier moves roughly monthly, and the same model name can measurably drift between snapshots. A default you set in January and never revisited isn't a decision anymore. It's a frozen bet against a market that kept moving without you.

The actual deliverable of "choosing a model" was never the model. It's the eval from the section above, plus a cadence to re-run it. Quarterly is a sane default. Monthly if you're cost-sensitive or your traffic mix shifts fast. Full receipts on how quickly this decays: <a href="/blog/model-shelf-life">models have a shelf life</a>.

Set a calendar reminder. I mean that literally, not as a rhetorical flourish.

Do you even need to pick one model?#

Here's the honest end state for a lot of workloads, and it's the part every vendor page skips because it complicates their pitch. If different requests genuinely want different models, picking ONE model is the wrong shape of answer. You route.

A router looks at each request as it comes in and sends it to a model sized for that specific job, adaptively, learning from what actually worked. You can hand that per-request decision to something like flux-auto instead of hardcoding one name into your client and hoping it's still the right call six months from now.

And when you need the same answer every time, a locked contract with a downstream system, a regression suite, a golden set you're comparing against, you pin a specific model and the router honors it. The proof isn't a promise on a pricing page. It's on the response itself: pin a flux-pinned-* id and the answer comes back marked x-flux-routed: false, which means the router didn't touch your choice. You don't have to trust that the pin held. You read it off the header.

The list of what you can actually call, pinned or otherwise, isn't a number worth memorizing, because it changes. GET /v1/models against your own key is the only source that's ever actually current, and it's a live, authoritative list that includes aliases like flux-auto, flux-fast, flux-standard, and flux-reasoning alongside every pinned id. Point your code at the endpoint. Never hardcode a count.

This deserves its own deep treatment, because "trust a router" is a bigger question than one section can honestly answer. We wrote the whole thing: <a href="/blog/llm-routing-guide">the complete guide to LLM routing</a>.

Why does this all come back to cost?#

Because the reason "which model fits this job" beats "which model is best" is fundamentally an economic one. The wrong model on an easy job overpays on every single call, forever, quietly, in the background where nobody's watching. A cheap model on a hard job pays you back in retries, cleanup, and the engineering hours spent working out why the extraction keeps coming back malformed.

The unit that actually matters is cost per finished answer, and the only way to make that visible is to measure it on your own traffic instead of guessing from a rate card. We built a whole pillar around exactly that lens, worth the deeper read: <a href="/blog/ai-cost-control-playbook">cost per answer, not per token</a>.

What if I just want a shortlist of model names?#

Fair. Sometimes you don't want a method, you want three names for "coding" and three names for "cheap extraction," and you want them today. That's a real, legitimate want, and I'm not going to pretend it isn't.

It's also the one thing this page will deliberately not hand you, and here's why. A specific-model shortlist is right for maybe six weeks before something reprices or a new release makes it look dated. Bolting that kind of content onto an evergreen method page turns the whole page into something that needs monthly surgery to stay honest, which defeats the point of writing a method in the first place.

So the split is deliberate. This page is the method, and it doesn't expire. Named shortlists, best for coding, best for long context, cheapest capable, belong in a dated series that gets refreshed on an actual cadence, and that series links back up to this one. Until it's live, your fastest path to a first-pass shortlist is a live leaderboard, used exactly the way the opening section describes it: a filter, never a verdict.

The choosing checklist#

Run this today. Not someday.

1. Write down the job and its constraints before you look at a leaderboard: task difficulty, latency budget, cost ceiling, output format, vision yes or no, and whether you need the same answer every time.
2. Use the leaderboard as a filter, not a verdict. Shortlist two or three candidates, then stop trusting it.
3. Pull ten to twenty real tasks from your own traffic and run every candidate against them. Fix your scoring rubric first.
4. Score three axes: did it land the task, what did the finished answer cost, how fast. Cost per finished answer beats cost per token.
5. Put a date on your pick and a cadence on the re-run. The frontier moves. Your default shouldn't be a frozen bet.
6. If different requests want different models, stop picking one. Route, and pin only where you need determinism.

You don't have to take a leaderboard's word for any of this, mine included. Grab a key from the quickstart, run your two or three candidates through one endpoint against your own tasks, and read the cost of each finished answer right off the response instead of reconstructing it from an invoice at the end of the month. <a href="https://fluxrouter.ai/docs/getting-started/quickstart">Start here</a>.

FAQ#

How do I choose the best LLM for my use case?#

Define the job and its constraints, shortlist two or three candidates off a live leaderboard, then run them against ten to twenty real tasks from your own traffic and score cost per finished answer, latency, and quality. Re-run the test on a cadence, the frontier moves roughly monthly.

Are LLM benchmarks reliable?#

As a first-pass filter, yes. As a verdict, no. Benchmarks measure generalized performance on a fixed public task set and can't see your latency budget, cost ceiling, or format needs, and the same model name can measurably drift between snapshots anyway.

Should I pick one model or route between several?#

If every request in your workload wants roughly the same thing, pick one and pin it. If different requests genuinely want different models, which is common the moment your traffic gets varied, so route instead and pin only the calls that need the exact same answer every time.

How do I measure cost per model call fairly?#

Don't measure per token, measure per finished answer, a wrong or retried answer costs more than a right one even at a lower token price. A per-request cost header on the actual call, instead of a blended monthly average, makes that number a fact instead of an estimate.