Measuring Loop Efficiency: Steps to Completion

Most teams measuring agent efficiency track the wrong number. They watch average steps to completion, see a healthy-looking 4.8, and conclude the agent is efficient — while a fifth of their users suffer through fifteen-step marathons the average quietly buries. Good agent loop efficiency metrics aren't about the mean. They're about the distribution, the tail, and the reason a run took the steps it took. This is a case study of a team that learned that the hard way.

The Problem the Platform Team Faced

The team ran an agent handling internal engineering queries, and their dashboard showed average steps to completion holding steady around 4.8. By that number, the agent was efficient and healthy. Leadership was happy. But support tickets told a different story: a steady trickle of "the agent took forever" and "it never gave me an answer" complaints that the metrics said shouldn't exist.

The disconnect was the average. When they finally plotted the full distribution of steps to completion, the shape was ugly — a tall spike at 2 to 4 steps and a long, fat tail stretching past 15. The healthy average was the sum of a lot of great runs and a meaningful minority of terrible ones, and the terrible ones were exactly the runs users complained about. The mean had been hiding the only runs that mattered.

⚡ Pro tip: Never report agent efficiency as a single average. At minimum, track the median, the 95th percentile, and the completion rate. The median tells you the typical experience, the p95 tells you how bad the bad runs get, and the completion rate tells you how often the agent gives up entirely — three numbers the mean smears into one misleading figure.

The Wrong Approach

The team's first response was to optimize the average down. They tuned prompts and front-loaded some context, and the average dropped from 4.8 to 4.1. On the dashboard, progress. In the tickets, nothing changed — because their optimizations made the already-fast runs slightly faster and did nothing for the tail. They'd improved the metric they were watching and not the experience anyone was having.

This is the core trap of averaging: it rewards you for helping the runs that were already fine. Shaving a step off a 3-step run moves the average as much as fixing a 15-step disaster, but only one of those changes a user's day. The team spent three weeks optimizing the mean and their complaint volume didn't budge, because they were measuring the thing that was already working.

Their second misstep was treating all long runs as identical. A 15-step run because the task was genuinely complex is fine; a 15-step run because the agent got stuck re-searching the same thing is a bug. Lumping them together as "long runs" meant they couldn't tell legitimate depth from pathological thrashing, and any fix aimed at "long runs" would wrongly punish the legitimate ones.

⚡ Pro tip: Before you optimize any efficiency metric, ask what behavior it rewards at the extremes. A metric that rewards fewer steps rewards an agent that gives up. A metric that rewards higher completion rewards an agent that never admits it can't help. Every single metric, optimized in isolation, drives a pathology — which is exactly why you track a small set that balances against each other rather than one number someone can game.

The Correct Approach

The fix started with measuring the right things. They replaced the single average with a set of agent loop efficiency metrics designed to expose the tail and explain it.

[object Object], ,[object Object],(,[object Object],):
    steps = ,[object Object],(r.steps ,[object Object], r ,[object Object], runs)
    completed = [r ,[object Object], r ,[object Object], runs ,[object Object], r.completed]
    ,[object Object], {
        ,[object Object],: percentile(steps, ,[object Object],),
        ,[object Object],: percentile(steps, ,[object Object],),
        ,[object Object],: ,[object Object],(completed) / ,[object Object],(runs),
        ,[object Object],: ,[object Object],(,[object Object], ,[object Object], s ,[object Object], steps ,[object Object], s > ,[object Object],) / ,[object Object],(steps),
        ,[object Object],: avg(r.wasted_steps / r.steps ,[object Object], r ,[object Object], completed),
    }

What this does: It reports the median and p95 to show the distribution's shape, the completion rate to catch give-ups, the share of runs in the long tail, and a wasted-step rate — steps that produced no new information — so the team can separate legitimately long runs from thrashing ones.

The

wasted_step_rate

[object Object], ,[object Object],(,[object Object],):
    ,[object Object],
    ,[object Object], step.result_hash ,[object Object], step.prior_result_hashes

What this does: It flags a step as wasted when its tool result duplicates one already seen, giving a concrete, countable signal for the thrashing that drives pathological long runs — the exact signal an average step count throws away.

A refinement they added later made the wasted-step signal even sharper: distinguishing wasted steps caused by the agent (re-asking a question it already answered) from wasted steps caused by a tool (returning the same empty result no matter what). Both show up as duplicate results, but they need different fixes — the first is a prompt or memory problem, the second is a tool problem. Splitting the wasted-step rate by likely cause turned a "something's thrashing" alarm into a "this specific tool is thrashing" diagnosis, which is what actually let them fix it fast.

⚡ Pro tip: Attach the wasted-step signal to individual runs, not just aggregate dashboards, so you can pull up the worst offenders and read their traces directly. Aggregate metrics tell you a tail exists; a sorted list of your highest-wasted-step runs shows you exactly what the thrashing looks like, and the root cause is usually obvious within two or three traces.

Results and What Changed

Once they measured wasted steps and the tail directly, the fix became obvious. Nearly all the pathological long runs shared a pattern — the same wasted-step signature from a tool that returned ambiguous empty results. They fixed that one tool's error signaling, and the p95 dropped from 15 steps to 6 within a week. The average barely moved, because the average had never been the problem. But the completion rate rose from 91% to 99%, and the "took forever" tickets stopped.

The lasting change was cultural. The team retired the average from their dashboard entirely and replaced it with the distribution and the wasted-step rate. Now when someone proposes an efficiency change, they judge it on whether it shrinks the tail and cuts wasted steps, not whether it nudges a mean that was already fine. Their agent loop efficiency metrics finally measured the thing users actually felt.

⚠️ Common mistake: Using average steps as your primary efficiency metric. The average is dominated by your common easy cases and structurally blind to the tail, which is precisely where your worst user experiences and your real bugs live. Track the distribution and a wasted-step signal instead; the mean will look fine right up until a user churns over a run it hid.

How to Apply This to Your Situation

Start by plotting your full distribution of steps to completion, not just the average. The shape tells you immediately whether you have a tail problem, and almost everyone does. Then add a wasted-step signal — the simplest version is flagging steps whose tool result duplicates an earlier one — so you can separate hard runs from broken ones.

Track completion rate alongside steps, because an agent can look efficient precisely because it gives up early, and a low step count paired with a low completion rate is a red flag, not a success. Report median and p95 together, and judge every efficiency change by its effect on the tail.

⚡ Pro tip: Pick one tail metric — p95 steps or the tail share — and put it on an alert with a threshold. Averages are for dashboards nobody watches; a paged alert on the tail is what actually catches a thrashing regression the day it ships, instead of three weeks later when the support tickets finally correlate. Alert on the number that represents your worst experiences, not your typical one.

Three teams applied this well. A sales-automation team found their "efficient" agent had a 12% give-up rate hidden behind a great average, and fixing completion mattered far more than steps. A legal-research team used the wasted-step rate to catch an agent re-reading the same documents, invisible in aggregate. And a devops team tied their p95 steps to an alert, so a thrashing regression paged them instead of hiding in a healthy mean.

Next Steps

Retire the average as your headline metric. Plot the distribution, add median and p95, track completion rate, and build a wasted-step signal to tell legitimate depth from thrashing. Then aim every efficiency effort at the tail, because that's where your users and your bugs actually are, and it's the part your old average was structurally built to hide.

The metric definitions and the wasted-step detection are reusable across every agent you monitor. A library like PromptABCD is a useful home for your measurement snippets and the prompts you pair with them, so your next agent ships with efficiency metrics that tell the truth instead of a comforting average that doesn't. The shift from watching a mean to watching a distribution is small in code and large in what it reveals — most teams find their first honest tail plot uncomfortable, which is exactly the point.