Cost per Loop Iteration: Budgeting Token Spend

Most teams budget agent loop token cost wrong, and it's not their fault — they look at the total spend per run and try to bring it down, when the real payoff is in understanding where the tokens go within each iteration. A team I worked with was convinced their agent was expensive because it took too many steps. It wasn't. It took a reasonable number of steps, each of which re-sent a mountain of context. Once they saw the cost per iteration instead of the cost per run, they cut spend by around sixty percent without removing a single step. Here's the case.

Cost per loop iteration is the token spend of a single turn: the context sent in, plus the output generated. Across a run, these iterations sum to the total — but the total hides the fact that later iterations often cost far more than earlier ones, because the context sent keeps growing. Budgeting by the total misses this entirely.

The Problem the Team Faced

The team's research agent cost roughly triple what they'd modeled. Their instinct, reasonable on the surface, was that it was taking too many steps — so they focused on making it finish faster, cutting steps to cut cost.

But the step count was fine. When they finally instrumented cost per iteration, the real picture appeared: step one cost a little, step two a bit more, and by step twelve each iteration cost many times what step one had — because every turn re-sent the entire growing transcript, including raw tool outputs from every prior step. The agent wasn't taking too many steps. Each step was dragging all the previous steps' bulk along with it, so the cost grew with every iteration even though the useful work per step stayed constant.

This is the trap that makes agent loop token cost so consistently misread: the run total is an average that hides a curve. Two runs can post the same total, but one spent it evenly across cheap steps while the other spent almost nothing early and enormous amounts late. The total treats those as identical when they call for opposite fixes — and it's always the late, context-heavy steps quietly dominating the bill that the total conceals. You cannot manage a cost you're only ever seeing in aggregate.

The Wrong Approach

They first tried to cut cost by reducing steps — a shorter loop, fewer iterations, a faster finish.

Solve the task in as few steps as possible. Be efficient.
Do not take unnecessary actions.

What this does: pushes the model to use fewer steps, which trims the count of iterations but does nothing about the cost of each one — leaving the real driver, growing per-iteration context, completely untouched.

It barely moved the bill, and it hurt quality. Fewer steps meant less thorough research, and the cost per remaining step was still climbing because the context problem was untouched. They were optimizing the wrong variable — cutting the number of iterations while ignoring that each iteration's cost was the thing spiraling. You can't fix a per-iteration cost problem by changing the iteration count.

⚠️ Common mistake: Budgeting agent cost by run total and optimizing step count. The total tells you how much you spent, not where it went — and the where is usually growing per-iteration context, not the number of iterations. Instrument cost per step before you optimize; teams routinely cut steps (hurting quality) when the real fix is holding per-step context flat (hurting nothing).

The Correct Prompt

The fix wasn't fewer steps — it was flat per-iteration cost. Keep the context sent each turn roughly constant instead of letting it grow, by trimming and compacting what rides along.

[object Object], ,[object Object],(,[object Object],):
    ,[object Object],
    state.messages = trim_and_compact(state.messages, target_tokens=,[object Object],)
    ,[object Object],
    reply = model.call(state.messages)
    ,[object Object], c ,[object Object], reply.tool_calls:
        slim = extract_relevant(run_tool(c), max_items=,[object Object],)
        state.add_observation(c.,[object Object],, slim)
    log_iteration_cost(state.step, reply.usage.total)     ,[object Object],
    ,[object Object], reply

What this does: holds the per-iteration context near a target size by compacting old turns and storing only relevant tool-output slices, and logs each step's cost — so later iterations cost about the same as early ones instead of ballooning.

[object Object],
,[object Object],
,[object Object],
,[object Object],

What this does: shows the effect of flat-context management — early iterations are similar, but the runaway growth in later steps disappears, which is where the bulk of the savings comes from.

Results and What Changed

Total spend fell by around sixty percent, on the same number of steps doing the same research. The savings came entirely from the later iterations, which had been carrying enormous re-sent context and now stayed lean. Same agent, same task, same thoroughness — just no longer paying to re-send every prior step's raw output on every turn.

The team's mental model shifted permanently. They stopped asking "how many steps did it take?" and started asking "what did each step cost, and why?" That reframing surfaced other savings too — an oversized system prompt re-sent every turn, verbose tool outputs — all invisible in the run total and obvious in the per-iteration view.

The broader lesson is that managing agent loop token cost is a measurement problem before it's an optimization problem. The team's sixty-percent win didn't come from a clever trick; it came from finally looking at the right number. Almost every team that thinks its agent is "just expensive" has never plotted cost against step index, and almost every one that does finds the same climbing curve hiding the same fixable cause. The optimization is easy once you can see it; seeing it is the actual work.

⚡ Pro tip: Break each iteration's tokens into input versus output. If input tokens dominate and grow with step number — the usual case — your problem is re-sent context, fixable with compaction. If output tokens dominate, the model is over-generating, a different fix. The input/output split points you straight at the right lever.

⚡ Pro tip: Instrument cost per iteration, not just per run. Log the token count of every model call with its step number, and plot it. A flat or slowly rising line is healthy; a steeply climbing one means growing context is quietly inflating your spend, and the run total will never show you that shape.

How to Apply This to Your Situation

Start by measuring. Log per-iteration token cost across a few real runs and plot it against step number. If the line climbs steeply, your cost is in growing context, and the fix is context management — trimming, compaction, relevant-slice storage — not fewer steps. If the line is flat but the run is just long, then step count is your lever. The plot tells you which problem you actually have.

This is why "measure before you optimize" is more than a platitude here — the two problems look identical in the bill and demand opposite fixes. Cut steps when the real issue is growing context and you damage quality while barely denting cost. Manage context when the real issue is genuinely too many steps and you add machinery that changes nothing. The plot is what disambiguates, and it takes an afternoon to build. An afternoon of measurement routinely saves a week of optimizing the wrong variable.

Then attack the biggest per-iteration contributors. Usually it's re-sent raw tool outputs and an ever-growing transcript; sometimes it's an oversized system prompt paid on every single turn. Each is invisible in the total and clear in the per-step breakdown.

⚡ Pro tip: Watch for prompt-caching opportunities on the fixed parts of your context. If your system prompt and tool definitions are identical every turn, a provider that caches repeated prefixes can charge far less for that re-sent bulk. The per-iteration view is what reveals how much of each step is the same fixed prefix — and therefore how much a cache could save.

⚡ Pro tip: Remember you pay for the system prompt every iteration. A 2,000-token system prompt on a 15-step run is 30,000 tokens spent before the agent does anything — trimming it to 800 tokens saves 18,000 tokens per run, every run. Fixed per-turn costs multiply by step count, so shaving them pays off far more than it looks.

Next Steps

Instrument cost per iteration on your agent this week, plot it against step number, and look at the shape. That single graph usually reveals whether your spend is a step-count problem or a per-iteration-context problem — and they need completely different fixes. Measure before you optimize.

The per-iteration cost logging and the flat-context management helpers are reusable across every agent you run. I keep them saved and versioned in PromptABCD next to the loop code, so a new agent's cost is visible and controlled per step from the start — instead of a run total that hides a per-iteration cost quietly tripling as the context grows. The graph is the whole intervention — everything downstream follows from finally being able to see the curve.