LLM agent evaluation:
why public benchmarks don't transfer.

A model posts a strong number on a public agent benchmark. You license it, point it at your own systems, and the resolve rate is not close. Nothing broke in the deployment. The benchmark measured a different task, in a different environment, driven by a different scaffold, and graded by a different program. One of those four things was the model. The other three were not.

This is the normal outcome rather than the surprising one, and it is the single most common reason a promising pilot stalls. The fix is not a better public benchmark. It is a small, unglamorous eval built out of your own work, which turns out to be the same artifact you need later if you ever want to fine-tune anything.

This post is about why the transfer fails, what to build instead, and how to tell whether the thing you built is telling you the truth.

The short version

A benchmark score is a property of a system, not of a model. Tasks, tools, harness, and grader all move the number, and your deployment replaces all four at once.

Public suites are contaminated in ways nobody can disprove. They sit on the open internet, which is where pretraining corpora come from.

Per-step quality is not what you ship. Agents fail by compounding, and the metric that matters is whether the end state is right, repeatedly, within budget.

The eval you build is reusable as a training signal. A checker that decides pass or fail is a reward function. Building one is not overhead on the way to fine-tuning, it is the first step of it.

What a public benchmark actually measures

Take any agentic suite you have seen quoted. SWE-bench Verified draws issues and their real fixes from public Python repositories and grades an attempt by running the repository's own tests. GAIA asks multi-hop research questions with short, checkable answers. Tau-bench scores multi-turn tool use against a simulated user in a retail or airline domain. WebArena drops an agent into self-hosted clones of common web applications. All four are real engineering, and all four are careful about the thing they set out to measure.

The thing they set out to measure is a general capability, sampled from a public distribution, under a reference harness. That is exactly the right target for comparing model A against model B. It is a different target from the one you have, which is whether a specific model, wrapped in your specific orchestration, can close a specific class of ticket in your specific environment without breaking anything.

Four layers sit between those two questions, and each one moves the score independently of the model.

Four reasons the number doesn't survive the move

1. The task distribution is not yours. Public agent tasks are selected to be solvable, unambiguous, and self-contained, because a benchmark full of underspecified tickets would be unscoreable. Your queue is the opposite. It contains requests that are missing a field, requests where the right answer is to refuse and escalate, and requests that depend on a convention nobody wrote down. An agent tuned to always attempt a solution looks excellent on a suite of solvable problems and reckless on yours.

2. The tools are not yours. A frontier model has read every public API reference and none of your internal ones. The benchmark gives it a handful of clean, well-named, well-documented tools. You have a dozen endpoints with overlapping names, an argument that means something different on the credit desk than it does on rates, and one service that returns a 200 with an error in the body. That gap is the subject of tool-use fine-tuning, and no public score has any visibility into it.

3. The harness is being scored along with the model. This is the one that surprises people. The published number belongs to a model plus a scaffold: a system prompt, a context management policy, a retry rule, a step limit, sometimes a whole planner. Two competent teams can run the same weights on the same suite and report visibly different results because they made different scaffolding choices, and neither is cheating. When you deploy, you supply your own scaffold, so you have replaced a component that was contributing to the score you based the decision on.

4. Contamination, and the ratchet that follows it. Public benchmarks are on the open internet. So are their solutions, the blog posts analysing them, and the repositories that fix the exact issues they sample. Pretraining corpora are built from that same internet. Nobody outside the lab can prove a specific model never saw a specific task, which makes contamination not just a risk but an unfalsifiable one. And there is a softer version that does not require memorization at all: when the entire field optimizes against the same few suites for two years, models drift toward what those suites reward. Both effects lift public numbers without touching performance on work that was never published, which is precisely the work you are buying the model to do.

Public benchmark or task-specific eval

These are not competing options so much as answers to different questions. The mistake is using the first where you needed the second.

Keep reading the leaderboards. They are a reasonable way to shortlist which three open-weight families to try. Just do not let a shortlist masquerade as a decision.

What an agent eval framework is made of

Four parts, and only the last one is difficult. Everything people mean by an agent eval framework is machinery for running the first three repeatably and computing the fourth.

The metrics that survive contact with production

Validation loss tells you nothing about an agent. Neither does per-step accuracy, and the reason is arithmetic: a model that picks the right action 95 percent of the time completes twenty consecutive steps correctly about 36 percent of the time. Step quality that reads as excellent produces a task success rate that reads as broken. Four measurements are worth the trouble.

Task success rate, on end state. Did the reconciled figure match, did the ticket close, did the query return the reference rows. Not whether the trajectory looked sensible, and emphatically not whether an LLM judge liked the write-up.

Reliability, as pass^k. Run each task k times and count only the tasks that passed every single time. This is the number that predicts whether people will trust the agent, and it is brutal. An agent with 80 percent per-attempt success has a pass@5 of 99.97 percent, which is what a demo shows you, and a pass^5 of about 33 percent, which is what a user feels after a week.

Cost and turns per resolved task. An agent that succeeds by making forty tool calls is a different product from one that succeeds in four, and the difference shows up in your serving bill, your rate limits, and how long somebody waits. Normalize by resolved task rather than by run, or a model that fails cheaply will look efficient.

Side-effect rate. Count any write outside the permitted surface as a failure even when the final answer was correct. An agent that gets the right number after mutating a table it should never have touched has not passed, and this is the metric no public benchmark can give you, because only you know which surfaces are yours to break.

One task in the set, written out, makes the shape concrete.

# One eval task: the prompt, the frozen world it runs in, and the check.
TASK = {
  "id":      "recon-042",
  "prompt":  "Reconcile the EOD cash break on book FX-3 for 2026-03-14.",
  "fixture": "snapshots/2026-03-14",   # pinned: identical every run
  "budget":  {"turns": 25, "usd": 0.40},
}

def check(run):
    # Grade the end state, never the wording of the write-up.
    if not run.within_budget(TASK["budget"]):
        return FAIL("over budget")
    if run.wrote_outside(ALLOWED_TABLES):
        return FAIL("side effect")      # right answer, still a failure
    gold = GOLD[TASK["id"]]
    return (run.result("break_amount") == gold["break_amount"]
            and run.result("cause_code") == gold["cause_code"])

# Reliability, not an average: 5 independent runs, all of them must pass.
# pass_hat_k = mean(all(check(r) for r in runs(task, k=5)) for task in TASKS)

Building the first version in a week

The reason teams do not have a task-specific eval is rarely disagreement about whether it would help. It is that it feels like a quarter of work. It is not, if you resist the urge to build a platform.

Day one, mine the logs. Pull the last few hundred instances of the workflow you want to automate. You are looking for thirty to fifty that are representative, including the awkward ones. Resist filtering out the ambiguous cases, because handling ambiguity is most of what separates a demo from a deployment.

Day two, freeze the world. Snapshot the data each task touches and pin the tool schemas alongside it. If the agent hits a live system, an eval that regresses next month tells you nothing, because you cannot separate the model from the data underneath it.

Days three and four, write the checker. Start with exact match against a known-good end state, which covers more cases than people expect. Where the output is genuinely open-ended, prefer a checklist of required facts over a holistic judge. Where you must use a model as a judge, pin its version, keep its rubric in the repository next to the tasks, and re-score a sample by hand every time you change either.

Day five, run it five times. Not once. The first thing you want to learn is variance, and variance is invisible in a single pass. Record cost and turns from the beginning, because backfilling them later means re-running everything.

Then split the set. Roughly two thirds is a development set you are allowed to look at, iterate against, and overfit to. The other third is a holdout that no prompt-tuning session and no training run ever reads. Anything you optimize against stops being a measurement, and that applies to prompt engineering exactly as much as it applies to gradient descent.

How your own eval will lie to you

A private eval removes the contamination problem and introduces four new ones. All four are manageable once named.

It is too small to say what you want it to say. With 50 tasks and an observed success rate near 60 percent, the 95 percent interval is roughly plus or minus 14 points. A model scoring 58 and a model scoring 68 are the same model as far as that eval is concerned. At 200 tasks the interval closes to about plus or minus 7. Report the interval next to the number, every time, or someone will make a purchasing decision out of noise.

The holdout leaks. Eval tasks get pasted into prompts for debugging, then those prompts get logged, then the logs become training data. This happens quietly and it is unrecoverable: a leaked holdout cannot be un-leaked, only rebuilt. Keep it in a separate location with separate access, and treat a suspected leak as a reason to resample rather than a reason to explain the score.

The grader drifts. Every loosening of the checker to stop flagging a false failure is a small upward revision of every future score. Version the verifier, and when it changes, re-run the previous checkpoint through the new version before you compare anything.

The tasks stop being your tasks. Your systems change, a schema is renamed, a desk changes its convention, and a frozen eval slowly becomes a measurement of a business you no longer run. Refresh a slice of it on a schedule, and keep a note of which tasks were replaced so a discontinuity in the trend has an explanation.

Why this is the fine-tuning conversation in disguise

Everything above is framed as measurement, and measurement is the immediate payoff. The larger one is that the artifact you just built is the input to improving the model rather than merely grading it.

A verifier that returns pass or fail is a reward function. That is the entire premise of reinforcement learning with verifiable rewards, covered in SFT versus RL for agents: the program that decides whether an attempt succeeded is what the training loop optimizes against. Teams that can write the checker have a path to training. Teams that cannot are restricted to prompt engineering, whether they frame it that way or not.

The task set does double duty too. Trajectories that pass become supervised training data, which is the raw material of agentic fine-tuning. Trajectories that fail tell you which kind of failure you have, and therefore which method to reach for: mechanical failures such as malformed arguments and wrong schemas are an SFT problem, while judgment failures across a sequence are an RL problem.

Which means the honest order of operations is the reverse of how most projects run. The eval is not the thing you build after choosing a model to prove the choice was right. It is the thing you build first, because it is what makes the choice, and it is the only asset in the project that survives changing your mind about every other part of it.

Where this fits at Numerata

P95 is Numerata's training layer, and it treats the eval as a first-class object rather than a script somebody runs at the end. Runs are tracked against a pinned task set and a versioned verifier, so the comparison between round three and round seven is a real comparison rather than two numbers produced by two slightly different graders. Rollouts and eval runs draw GPUs from the same pool Lupine manages, which is what makes running the whole set five times a scheduling question instead of a budget conversation, and finished checkpoints go to NinetyFive to serve.

The reason this runs inside your environment rather than somebody else's is what the eval is made of. A task-specific eval is a structured recording of how your people do the work: the tickets, the internal endpoints, the snapshot of the data, and the definition of a correct answer. That is a more sensitive artifact than the model weights it grades, and it is not something to hand to a third party in order to find out whether a model is any good. Numerata installs inside your own environment, private cloud, on-premises, or fully air-gapped, so the eval set never leaves.

If you are earlier in the picture, what fine-tuning is covers the basics, tool-use fine-tuning covers the internal-API gap that most agent failures trace back to, and synthetic data for fine-tuning covers what to do when you have tasks but no labels. Once the eval is running, AI agent failure modes covers the eight ways a run goes wrong in production and the signal that catches each one.