How to Benchmark Two Harnesses Head-to-Head

Most agent benchmarks measure the wrong thing. They run two setups, report which got a higher success rate, and declare a winner — as if success rate alone told you which harness to ship. It doesn't. A harness that succeeds 5% more often but costs three times as much and takes twice as long isn't obviously better; it's a trade-off you can't even see from a single number. A proper agent harness benchmark measures the full picture and, crucially, isolates what the harness contributes from what the model contributes. This teardown takes apart the naive benchmark, shows exactly where it misleads, and rebuilds it into something you can actually make decisions from.

The contrarian claim underneath: most head-to-head agent comparisons are secretly measuring the model, not the harness, and drawing harness conclusions from model differences. Fix that confound and the whole exercise becomes useful.

Before: The Weak Benchmark

Here's the benchmark almost everyone runs first — a handful of tasks, count the wins, pick the higher number.

[object Object], ,[object Object],(,[object Object],):
    a_wins = ,[object Object],(harness_a.run(t).succeeded ,[object Object], t ,[object Object], tasks)
    b_wins = ,[object Object],(harness_b.run(t).succeeded ,[object Object], t ,[object Object], tasks)
    ,[object Object],(,[object Object],)
    ,[object Object], ,[object Object], ,[object Object], a_wins > b_wins ,[object Object], ,[object Object],

What this does: Runs each harness once per task, counts successes, and declares whichever won more tasks the winner. It feels like a fair comparison. It's riddled with problems that make its verdict close to meaningless, and worse, confidently so — it hands you a clear winner and a false sense that you measured something, which is more dangerous than admitting you don't know.

Why It Fails

The first failure is run count. Agent runs are non-deterministic — the same task can succeed or fail across runs depending on the model's sampling. Running each task once means you're partly measuring luck. Harness A might "win" a task it would fail half the time, and you'd never know from a single run.

The second is dimensionality. Success rate is one axis; cost, latency, and step count are others, and they often move in opposite directions. A benchmark that reports only success rate hides the harness that wins on quality by burning tokens and time. You can't make a shipping decision without seeing the trade-offs, and one number erases them.

The third, and most insidious, is the model confound. If harness A and harness B use different models, different prompts, or different tool definitions, then a difference in results tells you nothing about the harnesses — it might be entirely the model or the prompt. Yet teams routinely change several things at once and attribute the result to "the harness."

⚠️ Common mistake: Comparing two harnesses while also changing the model or prompt between them, then attributing the difference to the harness. If more than one thing differs, you've measured the sum of the differences, not the harness. Hold the model, prompt, and tools identical across both harnesses, or your benchmark is measuring a confound.

After: The Improved Benchmark

The rebuilt version runs each task many times, measures multiple dimensions, holds everything but the harness constant, and reports results with enough statistics to know whether a difference is real.

[object Object], ,[object Object],(,[object Object],):
    results = {,[object Object],: [], ,[object Object],: []}
    ,[object Object], name, h ,[object Object], [(,[object Object],, harness_a), (,[object Object],, harness_b)]:
        ,[object Object], task ,[object Object], tasks:
            ,[object Object], _ ,[object Object], ,[object Object],(trials):
                run = h.run(task)                    ,[object Object],
                results[name].append({
                    ,[object Object],: task.,[object Object],,
                    ,[object Object],: run.succeeded,
                    ,[object Object],: run.cost,
                    ,[object Object],: run.latency,
                    ,[object Object],: run.steps,
                })
    ,[object Object], summarize(results)

What this does: Runs every task many times per harness, recording success along with cost, latency, and steps for each run, while keeping model, prompt, and tools identical across A and B. Now you have a distribution per metric per harness, not a single lucky number — enough to see both the average and the spread, and to attribute differences to the harness because nothing else changed.

Breaking Down Each Element

Each change fixes one of the three failures.

Repeated trials address non-determinism. With twenty runs per task you can compute a success rate with a confidence interval instead of a coin-flip. The spread matters as much as the mean — a harness that succeeds 80% of the time reliably is different from one that averages 80% by alternating between 100% and 60%.

Multi-metric recording restores the dimensions success rate hides. The metric that most often changes decisions isn't raw success — it's cost per success.

[object Object], ,[object Object],(,[object Object],):
    successes = [r ,[object Object], r ,[object Object], runs ,[object Object], r[,[object Object],]]
    ,[object Object], ,[object Object], successes:
        ,[object Object], ,[object Object],(,[object Object],)
    total_cost = ,[object Object],(r[,[object Object],] ,[object Object], r ,[object Object], runs)   ,[object Object],
    ,[object Object], total_cost / ,[object Object],(successes)

What this does: Divides total spend across all runs by the number of successful ones, so a harness that succeeds often but wastes money on failed attempts is penalized correctly. This single derived metric captures the quality-versus-cost trade-off that raw success rate erases — it's usually the number that actually decides which harness ships.

Holding inputs constant is what makes it an agent harness benchmark rather than a model benchmark. Same model, same prompt, same tools, same tasks — the only thing that differs is the harness code, so any difference in results is attributable to the harness. This is the discipline the naive version skips and the one that makes the results mean something.

⚡ Pro tip: Report the success rate as

pass@k

pass@k

pass@1

Variations for Different Contexts

For cost-sensitive deployments, weight your final ranking toward cost per success and latency; a small quality gain rarely justifies a large cost increase at scale.

For quality-critical deployments — anything where a failure is expensive — weight toward

pass@k

For a fair fight, use a task suite that resembles your real workload, not a generic benchmark. A harness that wins on someone else's tasks may lose on yours, because harness quality is partly a function of the kind of work it does.

⚠️ Common mistake: Reporting a single aggregate number across a diverse task suite. A harness can win overall while losing badly on the task category you care most about. Break results down by task type, so you see not just which harness is better on average but where each is stronger — which is what actually maps to your decision.

Knowing When a Difference Is Real

Running many trials gives you distributions, but distributions still need interpreting — and the trap is calling a winner on a difference that's within the noise. An agent harness benchmark that reports "A: 82%, B: 79%" and declares A the winner may be reporting nothing at all if the run-to-run variance is 5%. Before you act on a gap, check whether the gap is bigger than the noise.

You don't need heavy statistics for this. Compute a confidence interval on each harness's success rate and see whether they overlap; if they do, you can't distinguish the harnesses on that metric with the data you have, and the honest report is "no measurable difference," not "A wins by 3."

[object Object], ,[object Object],(,[object Object],):
    p = successes / n
    margin = ,[object Object], * ((p * (,[object Object], - p) / n) ** ,[object Object],)   ,[object Object],
    ,[object Object], (p - margin, p + margin)

What this does: Returns an approximate 95% confidence interval for a success rate given the successes and trials. If A's interval is (0.76, 0.88) and B's is (0.73, 0.85), they overlap heavily — you have no basis to prefer A, and running more trials is the only way to sharpen the comparison. This one check prevents the most common benchmark error: shipping a decision built on noise.

The fix when intervals overlap is more trials, not a bolder conclusion. Narrowing an interval requires more data, and it's worth running hundreds of trials on the tasks whose outcome will actually drive your decision, even if that's more than you'd run on the rest.

⚡ Pro tip: Build your task suite from real failures and real workloads, not clever puzzles. The tasks that best predict which harness will serve you in production are the ones drawn from what your agent actually does and where it has actually struggled. A benchmark full of contrived brain-teasers measures brain-teaser performance, which correlates weakly with the boring, high-volume work your users care about.

Curate the suite deliberately: a spread of difficulties, the categories you care about weighted by how much you care, and a healthy number of the messy, ambiguous cases where harness quality actually shows. A suite that's all easy tasks can't distinguish good harnesses from great ones, because they all pass; a suite that's all impossible tasks can't either, because they all fail. The signal lives in the middle, in the tasks a good harness gets right often but not always — those are the ones where a real difference between two harnesses becomes visible.

Save and Reuse This

A good benchmark harness — the multi-trial runner, the metric collectors, the significance tests — is substantial to build and easy to get subtly wrong, and an inconsistent methodology across comparisons makes results incomparable over time. Keep your benchmark task suite and evaluation code versioned alongside your prompts in a library like PromptABCD, so every harness comparison you run uses the same tasks and the same fair-fight discipline — and last month's numbers still mean something next to this month's. A benchmark whose task suite drifts between runs can't tell you whether your harness improved or just faced easier tasks; a stable, versioned suite is what makes your results a real time series instead of a pile of one-off measurements.