The Eval You Should Run Before Trusting Any Model

By Sean Donahoe · Published July 16, 2026 · accurate as of this date

How do you know a model is good enough for your task?#

You don't get it from a leaderboard. Build a small held-out set from your own real inputs, 20 to 50 to start. Write the right answer next to each one. Pick a grader that matches the task. Run every candidate through it unchanged, and score them on cost per correct answer plus worst-case latency. About an afternoon of work, and it's the only number that describes your task instead of someone else's.

Everything below is how to build that afternoon so it actually decides something.

Why the leaderboard can't answer this#

I've written the long version of this already, so here's the short one. The benchmarks-mislead post makes the full case; this is the one paragraph you need before we build.

A benchmark measures an average over its own distribution. Your task is one point on that curve, and the point you land on can behave nothing like the average. Worse, even a clean, un-gamed benchmark decays as a signal over time. A 2024 arXiv survey on benchmark data contamination puts it plainly: models "inadvertently incorporate evaluation benchmark information from their training data, leading to inaccurate or unreliable performance during the evaluation phase." So a public number is a ceiling on how far you can trust it, not a floor. The only score that describes your workload is one you generate on inputs the model has never seen. Build it.

The minimal eval has five parts#

That's it. Five. If you have all five, you have an eval; if you're missing one, you have a vibe.

The grader is the part that quietly sinks home-grown evals, so it gets its own section. First, the set.

Build the set: small, real, frozen, private#

Small. Start at 20 to 50 examples. That's enough to separate a good model from a bad one, and small enough that you can write it today instead of scheduling it. Grow it later, when two models tie and you need finer resolution.

Real, and skewed on purpose. Don't sample randomly. Random sampling buries the cases that actually hurt under a pile of easy ones. Stratify toward pain: the formats that break your parser, the weird edge inputs, the failure you already lie awake about. Then salt in a handful of ordinary cases so you're not only testing the extremes.

Frozen. Once written, the set doesn't change between runs. It's a regression suite now. Change it per run and you can't compare model A to model B, or this week's model to last week's. The whole value is that it holds still while everything else moves.

Private. This one's non-negotiable. Never paste your set into a prompt you log publicly, an issue tracker, or a public repo. A published eval set is a contaminated eval set the day a crawler finds it, for exactly the reason the contamination survey describes. Keep it in a private file. Treat it like a password, because functionally it is one.

Pick a grader that matches the task#

Here's where most people wreck a perfectly good eval. They grade everything by eyeballing it, or they reach for one clever technique and staple it onto the wrong kind of output. The fix is boring and it works: match the grader to the shape of the answer.

Two of those need a word.

Classification: report per class, not just an overall number. A model can ace your majority class, tank the one rare class you actually built the system for, and still post a shiny aggregate accuracy. The average hides the failure that gets you fired.

Open-ended is the one people are scared of, and it's more defensible than you'd think. LLM-as-judge is a real, studied technique. The researchers behind MT-Bench found a strong judge model reaches "over 80% agreement" with human raters, "the same level of agreement between humans." In other words, the judge disagrees with you about as often as two of your own reviewers disagree with each other. Good enough to grade with, if you respect the caveats.

The caveats, stated flat: judges have biases. They can favor longer answers, favor whatever came first, and flatter outputs that look like their own. So use a strong judge, hand it a rubric with explicit pass criteria instead of "is this good," and spot-check a sample by hand before you trust the column. And pick a judge that isn't one of the models you're testing. A model grading its own output isn't an eval, it's a model marking its own homework.

Rule of thumb for the whole table: use the most mechanical grader the task allows. A judge is a fallback for when no rule captures "correct," not a default. If exact match works, exact match wins.

Score on cost per correct answer, not accuracy#

Accuracy alone always picks the biggest model. That's not a decision, that's a reflex, and it's usually the wrong one. The real question is quality at a price you can pay at scale.

So the scoreboard has four columns:

• Pass rate on your frozen set.
• Cost per correct answer: total spend divided by number of passes. Not cost per call. Cost per call that was actually right.
• p95 latency, because the tail is what your users feel.
• A hard worst-case floor: the single worst output you'd tolerate shipping. One answer below that line disqualifies the model no matter how good the averages look.

Cost per correct is the column that changes minds. A model that's four points less accurate but a third of the price often wins a task with a retry path. And a model that's cheap per token but wrong twice as often can cost more per correct answer than the pricier one that nails it first try. Per-token pricing hides that. Per-correct pricing shows it.

To fill that column you need the real dollar cost of each call, not a reconstruction from token counts and a rate card. If you run candidates through Flux, the exact per-request cost comes back on the response as the X-Flux-Cost-Usd header on non-streaming calls. So the cost column fills itself, row by row, from the same responses your grader is already reading. The harness below just reads the header.

Run every candidate the same way#

An eval is only valid if the single thing changing between runs is the model id. Same prompt, same set, same grader. Change two things and your number means nothing, because you can't say which one moved it.

In practice that's one OpenAI-compatible endpoint and a swapped model string. Through Flux it's one key and one base URL; you change the id and re-run. The candidate list is whatever GET /v1/models returns for your key, which is the authoritative, always-current roster, so you're never testing against an id that quietly went away. When you want a model held dead still across a run, point at a pinned id like flux-pinned-claude-sonnet and it stays put, still metered the same way. And log the X-Flux-Model header on every row, so a run is auditable line by line and you can always see which model actually answered.

Not on Python? The whole loop is a curl and a grep:

curl -si https://api.fluxrouter.ai/v1/chat/completions \
  -H "Authorization: Bearer $FLUX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"flux-auto","messages":[{"role":"user","content":"Reply with the word ok."}]}' \
  | grep -iE '^x-flux-(model|original-model|routed|request-id|cost-usd): '

That prints the model that served and the cost of the call. Swap flux-auto for the next candidate, run it again, and you're doing the eval by hand.

The eval is a regression suite. Re-run it on every model bump#

Models change under a stable name. A provider ships a silent update, and your pass rate moves, sometimes up, sometimes down, and nobody sent you a changelog. Your frozen set is the tripwire that catches it. That's the whole reason it's frozen.

Cadence I'd run: before adopting any new candidate, and on a schedule for whatever model you're live on right now. Then diff the scoreboard, not the vibes. "It feels worse this week" is not a bug report. A two-point drop in pass rate on row 14 is.

The harness#

One file. Copy it, drop an eval.jsonl next to it, run it. It loads your rows, calls each candidate through an OpenAI-compatible client, grades per row, reads the cost off the response header, and prints a table: pass rate, cost per correct, p95 latency. Dependency-light, non-streaming on purpose so the cost header is there for every row. The judge is a separate model from your candidates, and the harness refuses to start if you pass the judge id in as a candidate, so nothing can grade itself by accident.

#!/usr/bin/env python3
# minimal llm eval harness. one file, stdlib + the openai client.
# usage: FLUX_API_KEY=sk-... python3 eval.py flux-auto flux-pinned-claude-sonnet
import json, os, sys, time
from openai import OpenAI

client = OpenAI(base_url="https://api.fluxrouter.ai/v1",
                api_key=os.environ["FLUX_API_KEY"])

RUBRIC = "The output must be factually correct and directly answer the input."
JUDGE_MODEL = "flux-pinned-sonar-reasoning-pro"  # a strong, independent judge; never one of your candidates

def grade_exact(out, expected):
    return out.strip().lower() == expected.strip().lower()

def grade_schema(out, expected):          # expected: JSON list of required keys
    text = out.strip()
    if text.startswith("```"):             # strip a ```json ... ``` fence if the model added one
        text = text.split("\n", 1)[-1].rsplit("```", 1)[0]
    try:
        obj = json.loads(text)
    except json.JSONDecodeError:
        return False
    return all(k in obj for k in json.loads(expected))

def grade_judge(out, expected):           # fallback for open-ended output
    prompt = (f"Grade the output against the rubric. Answer PASS or FAIL only.\n"
              f"Rubric: {RUBRIC}\nReference answer: {expected}\n"
              f"Model output: {out}\nVerdict:")
    r = client.chat.completions.create(
        model=JUDGE_MODEL, messages=[{"role": "user", "content": prompt}])
    return r.choices[0].message.content.strip().upper().startswith("PASS")

GRADERS = {"exact": grade_exact, "schema": grade_schema, "judge": grade_judge}

def call(model, prompt):
    raw = client.chat.completions.with_raw_response.create(
        model=model, messages=[{"role": "user", "content": prompt}])
    resp = raw.parse()
    cost = float(raw.headers.get("x-flux-cost-usd") or 0.0)
    served = raw.headers.get("x-flux-model") or model
    return resp.choices[0].message.content, cost, served

def run(model, rows):
    passes, spend, lats, served = 0, 0.0, [], set()
    for row in rows:
        t0 = time.time()
        out, cost, who = call(model, row["input"])
        lats.append(time.time() - t0); spend += cost; served.add(who)
        if GRADERS[row.get("grader", "exact")](out, row["expected"]):
            passes += 1
    n = len(rows)
    p95 = sorted(lats)[min(n - 1, round(0.95 * (n - 1)))]
    per_correct = spend / passes if passes else float("inf")
    return passes / n, per_correct, p95, ", ".join(sorted(served))

def main():
    candidates = sys.argv[1:] or ["flux-auto"]
    if JUDGE_MODEL in candidates:
        sys.exit(f"{JUDGE_MODEL} is the judge, so it can't also be a candidate. "
                 f"A model grading its own answers isn't an eval.")
    rows = [json.loads(l) for l in open("eval.jsonl") if l.strip()]
    print(f"{'model':26}{'pass':>6}{'$/correct':>12}{'p95 s':>8}  served")
    for m in candidates:
        rate, per_correct, p95, served = run(m, rows)
        print(f"{m:26}{rate*100:5.0f}%{per_correct:12.5f}{p95:8.2f}  {served}")

if __name__ == "__main__":
    main()

And a three-row eval.jsonl so you can run it in the next five minutes, one row per grader type:

{"input": "Extract the invoice total as a number only. Subtotal $80, tax $8.50, total $88.50.", "expected": "88.50", "grader": "exact"}
{"input": "Return JSON with keys name and amount for this order: 12 units of Widget.", "expected": "[\"name\", \"amount\"]", "grader": "schema"}
{"input": "In one sentence, what does a load balancer do?", "expected": "It distributes incoming network traffic across multiple servers.", "grader": "judge"}

Run it:

FLUX_API_KEY=sk-... python3 eval.py flux-auto flux-pinned-claude-sonnet

You get one row per candidate: pass rate, dollars per correct answer, p95 latency, and which model actually served. That's your go/no-go, on your data, in one screen.

Don't want to hand-write 20 rows? Hand the job to your coding agent instead. Paste this:

Build me a 20-row eval set for <my task>. I'll paste real examples.
Write eval.jsonl where each row is
{"input": ..., "expected": ..., "grader": "exact" | "schema" | "judge"}.
Stratify toward the inputs that break my parser and the edge cases I already
fear, plus a few ordinary ones. Don't invent facts I can't verify. Keep the
set private: don't commit it to a public repo or paste it anywhere crawlable.

Then run the harness against it. The agent builds the set; the harness scores the models. You make the call.

What people get wrong#

Quick hits, because I've watched every one of these blow up an otherwise fine eval:

• Testing on the same example you used to write the prompt. You tuned the prompt to that input. It'll pass. It proves nothing.
• N too small to tell two models apart, then betting the roadmap on a one-example gap. Two rows out of five is noise, not a signal.
• A grader that quietly rewards length or confidence instead of correctness. Long and wrong still fails.
• Letting a candidate judge its own output. Pick a judge that's off the candidate list, and let the harness enforce it.
• Publishing the eval set, which contaminates it the moment it's indexed.
• Scoring on accuracy and forgetting the bill entirely.

Build it once, run it forever#

The honest version of "which model should I use" is "which model passes my set cheapest, this week." That answer expires, which is exactly why the set is worth building. You write it once; it pays out every single time a model ships, quietly, whether the vendor told you or not. For the wider decision this eval feeds, the whole model-selection call, we wrote how to actually choose an LLM.

So point the harness at a few candidates through one key, read the cost column off the response, and let your own data pick. The models page has the id scheme, the transparency headers doc covers the cost and model headers the harness reads, and if you still think the leaderboard has your answer, go read why it doesn't and then come build your set.