Measuring the Reliability of an Autonomous Agent

A team shipped an agent that tested at 97% success. Three weeks later, support was fielding complaints that it failed roughly a third of the time. Nobody had lied, nobody had rigged the demo. They'd measured the wrong number, which is the most common autonomous agent reliability mistake there is: they quoted pass@k, the chance the agent succeeds at least once across a few tries, when the number that predicts production is pass^k, the chance it succeeds every time. For an agent that's 70% reliable per attempt, pass@3 is about 97% and pass^3 is about 34%. Same agent. A 63-point gap that lives entirely in which metric you chose to report. That gap is where most autonomous agent reliability claims quietly fall apart.

This guide gives you the metrics that actually predict whether an agent works in production, and how to compute them honestly.

Quick-Start: Measure Autonomous Agent Reliability Honestly

Run every task multiple times and report the consistency view, not the best-case one. This is the single change that closes the gap above.

[object Object], ,[object Object],(,[object Object],):
    ,[object Object],
    successes = ,[object Object],(,[object Object], ,[object Object], _ ,[object Object], ,[object Object],(n) ,[object Object], agent.run(task).passed)
    pass_1 = successes / n
    ,[object Object],
    ,[object Object], math ,[object Object], comb
    pass_k = comb(successes, k) / comb(n, k) ,[object Object], successes >= k ,[object Object], ,[object Object],
    ,[object Object], {,[object Object],: pass_1, ,[object Object],: pass_k,
            ,[object Object],: ,[object Object], ,[object Object], successes == n
                        ,[object Object], ,[object Object], ,[object Object], successes == ,[object Object],
                        ,[object Object], ,[object Object],}

What this does: runs each task many times, reports single-run success and the far stricter pass^k consistency measure, and buckets the task as consistently solved, inconsistently solved, or never solved — the honest picture of how the agent behaves under repetition.

Understanding the Variables of Autonomous Agent Reliability

Two ideas do most of the work, and both push against the flattering number.

pass@k versus pass^k. pass@k is a best-case view (at least one of k tries works) and it makes any agent look great. pass^k is a consistency view (all k tries work) and it's the only honest one for production, where the agent runs unattended and you don't get to pick its best attempt. When someone quotes a single impressive success rate, your first question should be which metric — because the same agent looks production-ready or barely functional depending on the answer.

The gap widens fast as tasks get longer, which is the part that catches teams off guard. A per-step reliability of 95% sounds excellent, but an agent that has to chain twenty steps to finish a task succeeds end-to-end only about 36% of the time, because 0.95 to the twentieth power is roughly 0.36. Reliability compounds against you over a trajectory, so a "95% reliable" agent on a long-horizon task is a coin flip you'll lose more often than you win. This is why per-step metrics flatter and end-to-end pass^k tells the truth.

⚡ Pro tip: Report reliability at the granularity you actually run at. If your agent completes tasks in one step, per-step numbers are fine; if it chains ten or twenty tool calls, only the end-to-end, all-steps-succeed number means anything — and it will be far lower than the per-step figure you were tempted to quote.

Consistency categories. Run each task n times and each task falls into one of three buckets: consistently solved (passes every time), inconsistently solved (passes sometimes), or never solved. The inconsistent bucket is the dangerous one — those are the tasks that pass your demo and fail your users, and an aggregate score smears them into a number that looks fine.

⚡ Pro tip: When your mean success rate stays flat but pass^8 slides over a few releases, your planner is regressing under non-determinism even though the headline number looks stable. Watching the consistency metric, not just the mean, is how you catch a reliability regression before production does.

Step-by-Step: Six SLOs, Not One Score

An aggregate "agent score" is the metric that hides the bug. It reads 0.85 today, 0.83 yesterday, dashboard green — while tool-call success sits at 0.62 and every large refund is going out with the wrong tax line. Agents aren't single-output functions; you can't reduce them to one number any more than you'd SLO an API on a blend of latency and errors. Track six, each mapping to a layer that can break on its own.

1. Task completion. Did the agent achieve the goal, verified on the end state (the database row, the sent email), not on the agent's own claim of success? Below ~80% usually means the planner or tool registry is broken.

2. Tool-call success. A stack of sub-checks: did it pick the right tool (or correctly call none on an irrelevant input), and were the arguments both schema-valid and semantically correct? A date that validates as 2026-01-01 but is the wrong date passes the schema and fails the task.

3. Recovery rate. When a tool call fails or returns garbage, does the agent recover, or does it spiral? Recovery separates agents that survive real-world flakiness from ones that only work when everything cooperates.

4. p99 latency. Not the average — the tail. The slow runs are where users give up and where loops hide.

5. Guardrail trip rate. How often safety gates fire. A rising rate means inputs are shifting, an attack is probing, or behavior is drifting.

6. Trace-grounded score. A judged score over the trajectory — the sequence of steps — not just the final answer, so you catch an agent that reached the right result through a wrong and fragile path.

Each of these maps to a layer that fails independently and has a different fix, which is the whole argument for tracking them separately. A task-completion problem points at the planner; a tool-call-success problem points at your tool descriptions or argument extraction; a recovery problem points at your retry and error-handling logic; a latency problem points at the tail. Collapse them into one score and every diagnosis becomes "the number went down," which tells you nothing about which layer to fix.

⚡ Pro tip: Give each SLO its own error budget and alert independently. When tool-call success breaches while the composite still looks green, you want a page that names the specific layer — not a slowly-sagging aggregate that everyone watches drift for a week before anyone can say what's actually wrong.

SLOS = {
    ,[object Object],: ,[object Object],, ,[object Object],: ,[object Object],, ,[object Object],: ,[object Object],,
    ,[object Object],: ,[object Object],, ,[object Object],: ,[object Object],, ,[object Object],: ,[object Object],,
}
,[object Object], ,[object Object],(,[object Object],):
    ,[object Object], {k: (,[object Object], ,[object Object], passes(k, metrics[k], SLOS[k]) ,[object Object], ,[object Object],) ,[object Object], k ,[object Object], SLOS}

What this does: holds each reliability dimension to its own SLO with its own budget, so a break in any single layer surfaces as a specific breach instead of being averaged away into a healthy-looking composite.

Pro-Level Variations

False-completion rate. Track how often the agent claims success while the end state says otherwise. This is one of the most dangerous agent failures — quiet, confident wrongness — and it's invisible to any metric that trusts the agent's own report. Verify against reality, always. The failure has a subtler cousin worth watching too: an agent that silently abandons part of the task while completing the rest — asked to update three systems, it updates two, reports done, and the third drift goes unnoticed until something downstream breaks. Measuring completion against the full intended end state, not just "did it return without erroring," is the only way to catch partial completion masquerading as success.

Slice by workflow and risk. A 90% overall completion rate can hide a 60% rate on your highest-risk workflow. Slice reliability by task type and by risk tier, because the aggregate is exactly where a dangerous segment goes to hide.

Watch the offline-online relationship. The value of your offline evals is whether they predict production. When a good offline score reliably precedes a good online outcome, the offline signal has earned trust — and you re-check that relationship as inputs, policies, and traffic change, because it can quietly decay.

⚡ Pro tip: Build your golden dataset from real production failures, not hypothetical tasks. An eval set drawn from actual failures measures the reliability that matters; one drawn from your imagination measures how well you predicted your own blind spots — which is exactly the thing you're bad at.

Troubleshooting Common Issues

"Our success rate is high but users complain." You're reporting pass@k or an aggregate mean. Switch to pass^k and slice by workflow; the complaints are almost certainly concentrated in an inconsistent bucket or a low-scoring segment the average is hiding.

⚠️ Common mistake: Trusting the agent's self-reported success. An agent that says "done" is making a claim, not stating a fact — measure completion on the end state it left behind, or your false-completion rate is whatever the agent feels like telling you.

"Reliability looks fine offline, bad in production." Your offline set doesn't contain the inputs production does. Feed real failures back into the golden set continuously so the gap between offline and online closes over time.

Your Turn

Pick one workflow and run it ten times per task this week. Report pass^k alongside pass@k, bucket each task by consistency, and split your one score into the six SLOs. The number that comes back will be less flattering than your old dashboard — and far more predictive of what your users are about to experience.

Keep your SLO definitions, pass^k harnesses, and golden datasets versioned in a shared library like PromptABCD, so every agent you measure gets held to an honest autonomous agent reliability bar — the one that predicts production, not the one that looks good in a slide.