Measuring Pass@k for AI Agents (and Why It Misleads)

A team shipped an agent to production on the strength of a number: pass@5 of 90%. It passed nine of ten tasks within five attempts, so they felt safe. In production it failed constantly — users saw errors on more than half their first tries. The number wasn't wrong; their reading of it was. A metric can be perfectly accurate and still point you at the wrong conclusion if you forget what it actually counts. A high pass at k agent eval score can hide terrible single-attempt reliability, and if production only gives the agent one shot, pass@5 is measuring something your users never experience. This case study is about that gap, and how to measure the thing that actually matters.

The Problem the Team Faced

The team built an agent for a task that ran once per user request — no retries, one attempt, take it or leave it. During development they measured pass@k: run each task up to k times, count it a success if any attempt succeeds. Their pass@5 came in at 90%, which looked like a strong, shippable agent.

Then production told a different story. Roughly half of first attempts failed. Users didn't get five tries; they got one, and one try succeeded far less than 90% of the time. The team had optimized and shipped against a metric that assumed retries their production system didn't offer. Every hour of development had pushed a number that described a world their users didn't live in.

The lead engineer's realization was the turning point: "Pass@5 measures 'can it ever do this.' Production measures 'does it do this the first time.' We shipped on the wrong one." That sentence is the whole lesson. Pass@k and single-attempt reliability are different questions, and confusing them puts unreliable agents into production wearing a reassuring number.

Understanding Pass@k

Pass@k is the probability that at least one of k attempts succeeds. It's the standard metric in code-generation research, and it's genuinely useful — for the right question.

The math is worth seeing, because it reveals the trap. If a single attempt succeeds with probability p, then pass@k is:

[object Object], ,[object Object],(,[object Object],):
    ,[object Object],
    ,[object Object], ,[object Object], - (,[object Object], - p_single) ** k

,[object Object],
,[object Object],(pass_at_k(,[object Object],, ,[object Object],))   ,[object Object],
,[object Object],(pass_at_k(,[object Object],, ,[object Object],))   ,[object Object],

What this does: it computes pass@k from the single-attempt success rate. Notice the trap laid bare — a mediocre 37% single-shot agent reaches 90% at k=5, purely because five tries at a coin-flip-ish task usually land once. The impressive pass@5 was arithmetic, not reliability. The higher you set k, the more it flatters an unreliable agent.

⚠️ Common mistake: Reporting pass@k without stating k or the single-attempt rate, then shipping to a system that offers a different number of attempts. Pass@5 is meaningless for a production path that allows one try. Always match the k in your eval to the number of attempts production actually gives the agent — and if production gives one, measure pass@1, however much worse it looks.

⚡ Pro tip: Estimate pass@k from many samples rather than literally running k attempts per task. Run each task, say, twenty times, measure the single-attempt rate directly, and compute pass@k for any k from that. It's cheaper and more stable than running exactly k attempts, and it gives you the single-attempt rate — the number that actually predicts production — for free.

What They Should Have Measured

The fix depended on what production actually did. Since production gave one attempt, the honest metric was pass@1 — the single-attempt success rate, which was a sobering 37%. That number was ugly, but it was true, and a true ugly number beats a flattering false one every time. You can act on a true number; a false one only tells you when to be surprised.

But there's a second, subtler metric the team learned to care about: pass^k — note the caret, not the "at." Where pass@k asks "does at least one of k attempts succeed," pass^k asks "do all k attempts succeed." For reliability, pass^k is often what you want:

[object Object], ,[object Object],(,[object Object],):
    ,[object Object],
    ,[object Object], p_single ** k

,[object Object],(pass_hat_k(,[object Object],, ,[object Object],))   ,[object Object],

What this does: it computes the chance that every one of k attempts succeeds. This is the metric behind reliability-focused agent benchmarks, because a dependable agent should succeed consistently, not just eventually. Where pass@k rewards an agent for getting lucky once, pass^k punishes it for ever failing — which is exactly the pressure you want on an agent people will rely on.

⚡ Pro tip: Match your metric to your production reality. Retries allowed and only final success matters? Pass@k with k set to your real retry budget. One attempt, or every attempt must work? Pass@1 or pass^k. The metric isn't a matter of taste — it's determined by how many chances production gives the agent and whether occasional failure is acceptable.

Results and What Changed

Once the team measured pass@1, they stopped pretending and started improving the thing that mattered. They couldn't ship a 37% agent, so they did the real work — better tool descriptions, stricter output validation, clearer instructions — and watched pass@1 climb toward a rate production could actually stand behind. The ugly number gave them a real target; the flattering one had let them ship a problem.

They also changed how they reported results across the team. Every eval number now came with its k and its single-attempt rate attached, so nobody could accidentally read a pass@5 as a reliability guarantee again. The metric stopped being a single reassuring figure and became an honest description of behavior under a specific number of attempts.

The reframe rippled into how they thought about improvements, too. Under pass@5, the cheapest way to raise the score was to allow more attempts — a change that helps the metric while doing nothing for a user who gets one shot. Under pass@1, the only way to move the number was to make the agent genuinely more reliable on the first try. The metric they chose didn't just measure the agent; it steered what they worked on. Pick pass@k and you're quietly incentivized to add retries; pick pass@1 and you're incentivized to fix the agent. Metrics are targets, and the team learned to choose the target that pointed at the work they actually needed done.

⚡ Pro tip: Report the single-attempt rate alongside any pass@k, always. Pass@k without p is a number that can flatter an unreliable agent into production. With the single-attempt rate visible next to it, everyone can see whether a high pass@k reflects real capability or just a generous retry budget doing the heavy lifting.

How to Apply This to Your Situation

The principle generalizes anywhere you measure agent reliability:

• A fintech engineer whose agent makes one irreversible decision per transaction measures pass^k, because "usually right, occasionally catastrophically wrong" is unacceptable for money and consistency is the whole requirement.
• A coding-tools team whose agent gets several attempts with a human picking the best output legitimately measures pass@k, because their production really does allow retries and only the final result ships.
• A customer-support lead whose bot answers once per user measures pass@1, matching the single shot production gives, rather than a multi-attempt number that would overstate what users experience.

The move is always the same: figure out how many attempts production grants and whether occasional failure is tolerable, then pick the metric that mirrors that reality.

There's a reporting habit that ties this together and prevents the whole class of mistake. Any pass at k agent eval you publish should travel with three numbers, not one: the single-attempt success rate, the value of k, and whether the metric is pass@k (at least one success) or pass^k (all succeed). Those three together are unambiguous — anyone reading them can compute what production will actually experience. A lone percentage can't be reasoned about, because the same 90% can describe a rock-solid agent or a coin-flip that got five tries. Make the k and the attempt model impossible to overlook, and you close the gap between what you measured and what you shipped.

Next Steps

Look at whatever agent metric you're currently trusting and ask two questions: what's k, and how many attempts does production actually give? If the eval's k is bigger than production's real attempt budget, your number is optimistic, possibly dangerously so. Recompute at production's real k — or at pass@1 if that's the truth — and see whether you still feel good about shipping. The recomputation takes five minutes and occasionally saves you from putting a coin-flip into production wearing a 90% badge.

A pass at k agent eval is a sharp tool when its k matches reality and a misleading one when it doesn't. The eval configurations, per-attempt rates, and metric choices you settle on are worth keeping consistent across your whole team, so nobody re-learns this lesson in production. Storing your eval task definitions and their agreed metrics in a library like PromptABCD — tagged with the k and the attempt model each assumes — means everyone reads the same number the same way, and a flattering pass@5 never sneaks an unreliable agent past the gate again.