Budget Caps for Autonomous Agents

Most advice on the autonomous agent budget cap is wrong in the same way: it tells you to set a dollar limit and move on. A single dollar cap is the one that fails first, in the most expensive way, and a team I'll describe learned exactly how. The real answer is layered, and the layers matter more than the numbers. This is the story of a budget cap that didn't hold and the redesign that did.

An autonomous agent budget cap is a limit that stops an agent when it has spent enough - but "enough" turns out to have several dimensions, and capping only one leaves the others wide open. The team in this case study capped dollars per run, felt protected, and got a surprise bill anyway. Understanding why is the whole lesson.

The problem the team faced

A SaaS company ran autonomous agents to handle user support requests - one agent run per incoming ticket. They'd read about cost runaway and done the responsible thing: set a budget cap of two dollars per run. No single agent could spend more than two dollars. They considered the cost problem solved and shipped.

For weeks it was fine. Then a bad deploy introduced a bug that caused a subset of tickets to trigger agent runs in a retry loop - the same ticket spawning run after run. Every individual run stayed comfortably under its two-dollar cap. But thousands of runs fired in an hour, and the bill for that hour was enormous. The per-run cap did precisely what it promised and protected nothing that mattered.

The wrong approach

The original design had exactly one budget cap, at exactly one scope:

[object Object], ,[object Object],(,[object Object],):
    run_agent(ticket, max_spend=,[object Object],)   ,[object Object],
,[object Object],

What this does: it caps spending within a single agent run but places no limit on how many runs happen, how frequently, or what they sum to - so any bug that multiplies runs bypasses the cap entirely while technically never violating it.

The flaw is a scope mismatch. The cap protected against one run being expensive. The actual failure was many runs being individually cheap. A budget cap at the run scope is blind to everything above the run - and "a thousand cheap runs" is above the run. The team had built a fence around each tree and left the forest unguarded.

⚠️ Common mistake: Relying on a single per-run budget cap. It protects against one expensive run and does nothing against many cheap ones - which is the more common and more dangerous failure, because it usually comes from a bug that multiplies runs, and bugs multiply things fast.

The correct approach

The redesign added budget caps at multiple scopes and on multiple dimensions, so a runaway on any axis hit a limit:

BUDGETS = {
    ,[object Object],:   ,[object Object],,     ,[object Object],
    ,[object Object],:  ,[object Object],,    ,[object Object],
    ,[object Object],:  ,[object Object],,   ,[object Object],
}

,[object Object], ,[object Object],(,[object Object],):
    ,[object Object], spend_today(user) > BUDGETS[,[object Object],]:
        ,[object Object], escalate(,[object Object],, ticket)
    ,[object Object], spend_last_hour() > BUDGETS[,[object Object],]:
        halt_all_agents(); alert_oncall()      ,[object Object],
        ,[object Object], escalate(,[object Object],, ticket)
    ,[object Object], run_agent(ticket, max_spend=BUDGETS[,[object Object],])

What this does: it checks spending at three scopes before each run - the individual run, the user's daily total, and the whole fleet's hourly total - and trips a circuit breaker that halts every agent if aggregate spend spikes, so a runaway on any axis is caught by the cap at that axis.

The key addition is the fleet-wide circuit breaker. The per-hour aggregate cap is what would have caught the retry-loop bug: thousands of runs summing past two hundred dollars in an hour trips the breaker, halts everything, and pages a human - long before the bill got huge. The per-user cap catches a narrower version, and the per-run cap still handles the single-expensive-run case. Three scopes, three different failures covered.

⚡ Pro tip: An autonomous agent budget cap needs at least three scopes - per run, per user, and an aggregate fleet-wide circuit breaker. The per-run cap you think of first is the one least likely to save you, because the expensive failures are usually about run count, not run cost.

Results and what changed

When a similar multiplication bug reappeared months later, the fleet circuit breaker tripped within minutes, halted the agents, and alerted the on-call engineer. The total damage was a rounding error compared to the original incident. The bug still happened - caps don't prevent bugs - but the blast radius was bounded to minutes and a small sum instead of an hour and a large one.

The team also added budget caps on dimensions, not just scopes. They'd been capping dollars, but a run could also burn excessive time (blocking other work) or tokens against a rate limit. Capping all three - dollars, wall-clock, and token count - meant a runaway on any dimension hit a limit, not just the one they'd happened to think of first.

⚡ Pro tip: Cap dollars, time, and tokens separately - they're different resources and a runaway can spike any one of them. An agent can stay under its dollar cap while blowing your rate limit or blocking a queue for an hour, so a dollar-only budget cap leaves two other doors open.

What should an agent do when it hits a budget cap?

A budget cap raises a question most teams answer badly: what happens at the moment the cap trips? The lazy answer - hard-stop and drop everything - throws away whatever work the agent had done and often leaves things in a half-finished state. A good autonomous agent budget cap degrades gracefully instead of crashing into the limit.

The principle is to spend the last of the budget finalizing, not on more work. When an agent nears its cap, it should stop starting new work and instead consolidate what it has - return partial results, summarize progress, and hand off cleanly - so the spend already made isn't wasted. A run that hits its cap mid-task and returns "here's what I completed and what remains" is far more useful than one that vanishes at the limit with nothing to show.

⚡ Pro tip: Reserve a slice of the budget for a clean finish. When spend crosses, say, 90% of the cap, switch the agent from working to wrapping up - returning partial results and a handoff summary. A graceful stop preserves the value of everything spent so far; a hard stop discards it.

There's a reporting dimension too. A tripped budget cap is information: it means the task cost more than budgeted, which either means the budget was too low or the task was bigger or buggier than expected. Every cap hit should be logged and reviewed, not silently swallowed. A cap that trips often is telling you something about your tasks or your agent that you want to hear.

⚡ Pro tip: Treat every budget-cap hit as a signal to investigate, not a routine outcome. Frequent cap hits mean your estimate is wrong or your agent is inefficient - either way, a cap firing regularly is data about a problem, and swallowing it silently wastes that data.

How to apply this to your situation

List the scopes at which your agents run - individual run, user, team, whole fleet - and put a budget cap at each. Then list the resources they consume - money, time, tokens, maybe API quota - and cap each of those too. The matrix of scopes times resources is your real budget protection; a single dollar-per-run cap is one cell of it.

A question the layered approach raises: how do you set the actual numbers? Guessing invites both false trips and useless caps. The method that works is to measure first - run a representative sample of real tasks with generous caps and record the actual distribution of cost per run, per user per day, and per hour across the fleet. Set each cap comfortably above the normal maximum you observed, so legitimate work never trips it, but far below the level that would hurt, so a runaway does. A cap set around ten times the observed p99 leaves room for genuinely large tasks while still catching a true explosion, which is usually orders of magnitude beyond that. Setting caps from measured distributions rather than round numbers is what keeps them from either firing on normal work or sitting so high they never fire at all - and you should revisit the numbers whenever your task mix or model pricing changes, because both shift the distribution the caps were tuned to.

For a consumer app with many users, the per-user and fleet caps matter most, because your risk is volume. For an internal tool with few users but expensive tasks, the per-run and time caps matter most. For any multi-agent system, the fleet circuit breaker is essential, because branching agents are exactly the case where run count explodes.

Next steps

Audit your agents for the scopes and resources you're not capping. Almost everyone caps dollars-per-run and stops there, which leaves the failure that actually gets people - many cheap runs - completely open. Add the aggregate circuit breaker first; it's the single highest-value cap you're probably missing.

The budget cap configurations - the scope thresholds, the circuit-breaker logic, the multi-resource limits - are reusable across every agent you deploy. Keeping them in PromptABCD means your next agent inherits layered budget protection by default, instead of learning about scope mismatch the way this team did, through a bill for an hour that should have cost pennies instead of a fortune.