When to Give an Agent More Autonomy

Picture this: your refund agent has run clean for two months. Every refund it proposes, a human approves, and lately they've all been rubber-stamps — nobody's changed one in weeks. So someone on the team asks the obvious question: can we stop reviewing every single refund? It's the right question, and the way most teams answer it is exactly wrong. They decide to increase agent autonomy because the reviews got annoying, not because the evidence earned it — and those are very different reasons that lead to very different outcomes.

Let's walk through how one team got this right, as a case study, because "when to give an agent more autonomy" is a decision that deserves data, not a gut call.

The Problem the Ops Team Faced

The team ran a customer-refund agent gated at synchronous approval: the agent proposed every refund, a human clicked yes. It worked well. Too well, in a sense — the approval queue had become a formality, humans were approving in seconds without really looking, and the whole point of the agent (saving human time) was being undercut by a review step that had stopped adding value. They needed to decide whether, and how, to loosen the gate.

The pressure to just flip it off was real. The reviews felt pointless, the team was busy, and "it's been fine for two months" is a seductive argument that feels like data without actually being data. But "fine so far" and "safe to run unwatched" are not the same claim, and conflating them is how teams get surprised.

The Wrong Approach

The tempting move was to flip the whole agent to full autonomy in one step, justified by vibes: it feels ready, the reviews are annoying, let's free everyone up. This fails for three reasons the team thankfully caught before shipping it.

First, "feels ready" isn't a measurement. Rubber-stamped approvals aren't evidence the agent is right — they're evidence the humans stopped checking, which tells you nothing about the agent's actual accuracy. Second, flipping the whole agent ignores that different actions carry different risk: a $10 refund and a $5,000 refund are not the same decision, and one autonomy setting for both is wrong for at least one of them. Third, no plan for catching a regression: full autonomy with no sampling means the first sign of a problem is a customer complaint or a finance discrepancy, which is the most expensive possible way to learn your agent drifted.

⚠️ Common mistake: Increasing an agent's autonomy because oversight became tedious rather than because the data earned it. Annoyance is a real signal that something should change — but the fix might be better tooling or batched approvals, not blanket autonomy. Let the correction-rate data, not the tedium, decide whether trust was actually earned.

The Correct Way to Increase Agent Autonomy

The team replaced the gut call with explicit, evidence-based promotion criteria, applied per action class rather than to the whole agent.

[object Object], ,[object Object],(,[object Object],):
    ,[object Object],
    ,[object Object], stats.n < ,[object Object],:                        ,[object Object], ,[object Object],
    ,[object Object], stats.human_correction_rate > ,[object Object],:   ,[object Object], ,[object Object],
    ,[object Object], stats.pass_k(k=,[object Object],) < ,[object Object],:             ,[object Object], ,[object Object],
    ,[object Object], ,[object Object], action_class.bounded_blast_radius: ,[object Object], ,[object Object],
    ,[object Object], ,[object Object],       ,[object Object],

What this does: refuses to increase autonomy for an action class until it has a large clean sample, a near-zero human-correction rate, proven consistency under repetition, and a bounded worst case — and even then promotes by a single rung, not straight to unattended.

Four criteria had to all hold before any promotion. Sufficient volume, so the track record is statistically real and not a lucky streak. A near-zero human-correction rate, the direct measure of whether the agent's outputs were actually right (not just approved). Consistency under repetition, measured with pass^k, because an action class that passes 8-of-8 independent runs is genuinely reliable in a way a single-run success rate can't show. And a bounded blast radius, so even a rare miss is survivable.

The correction-rate criterion deserves special emphasis, because it's the one that exposes the rubber-stamp trap. The team couldn't use "approval rate" as evidence — a 100% approval rate is exactly what rubber-stamping produces, and it says nothing about correctness. So they went back and had a reviewer genuinely re-examine a sample of the agent's past refunds, carefully this time, to establish the real correction rate rather than the rubber-stamped one. That distinction — between "how often did a human click yes" and "how often was the agent actually right" — is the whole difference between evidence and its illusion. If your only data is a queue of fast approvals, you don't have evidence yet; you have to go create it.

Crucially, promotion moved the class up one rung — from "human approves each" to "auto with notify-and-undo" — not straight to unattended autonomy. And they promoted the low-risk classes first: small refunds under $50 graduated, while large refunds stayed gated.

⚡ Pro tip: Never use approval rate as evidence for promotion — use a freshly-measured correction rate. A high approval rate is indistinguishable from rubber-stamping, so before you promote a class, have someone genuinely re-review a sample and measure how often the agent was actually right. That number, not the click-through rate, is your evidence.

Results and What Changed

Small-refund autonomy freed most of the human review time without incident, because the promotion was backed by two months of genuine correction-rate data rather than a feeling. The notify-and-undo rung meant that even the promoted class had a safety net — a human saw every auto-refund and could reverse it within a window — so "autonomous" never meant "invisible."

The large-refund class stayed gated, and that turned out to matter: a month later a pricing change caused the agent to miscalculate a batch of high-value refunds. Because those were still human-reviewed, the reviewer caught it immediately. Had the team flipped everything to full autonomy on vibes, that batch would have gone out wrong. The graded, evidence-based approach didn't just free up time — it kept the human exactly where the risk still lived.

The episode also validated the demotion machinery. When the pricing change landed, the small-refund class's correction rate ticked up too — not catastrophically, but enough to cross the threshold — and the automatic demotion dropped it back to synchronous approval on its own, before anyone noticed the pricing issue manually. The team's first signal that something had changed upstream was the agent's own trust falling. That's the payoff of building demotion in from the start: the system that grants autonomy is also the system that revokes it the moment the evidence stops supporting it, with no human having to be watching at the right moment.

⚡ Pro tip: Promote autonomy per action class, never per agent. The same agent can be fully autonomous on tagging tickets and fully gated on issuing large refunds. A single autonomy dial for the whole agent forces you to set it wrong for either the safe actions or the risky ones.

⚡ Pro tip: Make autonomy bidirectional and automatic. The same criteria that promote a class should demote it: if the correction rate creeps up after a model change or a data shift, the class drops back a rung on its own. Trust that can only ratchet up is how you end up over-trusting a system that quietly got worse.

How to Apply This to Your Situation

Before you loosen any gate, write down the four criteria and check them against real data: volume, correction rate, pass^k consistency, and bounded blast radius. Promote one rung at a time, lowest-risk class first, and keep a safety net (notify-and-undo, or sampling) on newly promoted classes. Wire the demotion path so trust can fall as automatically as it rises.

Sequence it so each promotion teaches you something before the next. Start with the single lowest-risk, highest-volume class — the one where you have the most data and the least downside — and promote only that one. Watch it at the new rung for a few weeks with the safety net on, confirm the correction rate holds now that a human isn't approving each action, and only then consider the next class. Promoting one class at a time isn't slower in any way that matters; it's how you keep every promotion individually reversible, so a mistake teaches you a cheap lesson on one class instead of an expensive one across all of them at once.

⚡ Pro tip: After any model or prompt change, temporarily demote your most autonomous classes and let them re-earn the rung on fresh correction data. The trust the old version earned doesn't transfer to the new one — verify it cheaply before a regression reaches an unattended action.

Next Steps

Pick the one action class where oversight has become a rubber-stamp and gather its real numbers this week. If the volume, correction rate, and consistency clear the bar and the blast radius is bounded, promote it one rung with a safety net — and set up the demotion trigger before you do. Let evidence, not tedium, be the thing that finally decides to increase agent autonomy.

Keep your promotion criteria, per-class thresholds, and demotion triggers versioned in a shared library like PromptABCD, so every decision to increase agent autonomy is made against the same evidence bar — instead of against whoever's most tired of clicking approve that week.