Confidence Scoring to Decide When to Stop

Here's a number that surprised the team I was helping: 41% of their agent's wrong answers came from stopping too early, and another 28% came from stopping too late and talking itself out of a correct answer. Nearly seventy percent of failures were timing failures — the agent knew enough, or didn't, but the loop had no way to tell. Their stopping rule was a fixed step count, and a fixed step count is blind to how hard the current task actually is.

The fix was agent confidence scoring: instead of stopping after N steps, the loop asks "how sure am I?" at each step and stops when confidence crosses a threshold. This is a walkthrough of that rollout — what broke, what they tried, and the version that finally shipped.

The Problem the ML Team Faced

The agent answered technical questions for an internal developer platform. It searched docs, read code, and synthesized answers. The team had set

max_steps=6

But questions aren't uniform. A "what's the default timeout?" lookup needs one step. A "why does this request fail intermittently under load?" investigation might need nine. The fixed ceiling served neither well. Simple questions burned all six steps because nothing told the agent to stop early. Hard questions got guillotined at step six with a half-formed answer. The team was tuning one number to fit two opposite problems, and it fit neither.

Worse, the failures were invisible in aggregate metrics. Average step count looked healthy at 5.2. Only when they bucketed by question type did the bimodal disaster show up — a cluster of easy questions wastefully maxing out, and a cluster of hard questions truncated.

⚡ Pro tip: Never trust an average step count as a health metric. Bucket runs by task type or difficulty first. Averages hide exactly the two failure modes — premature stopping and pointless grinding — that a stopping-criteria problem produces, because they cancel out in the mean.

The Wrong Approach

The team's first instinct was to raise the ceiling to twelve, giving hard questions room. This made hard questions somewhat better and easy questions dramatically worse — now trivial lookups could ramble for eleven steps, tripling cost and, oddly, lowering accuracy on easy questions because the agent kept "finding" reasons to second-guess a correct early answer. Extra steps aren't free even when they're available; a model given room to keep going will often use it to reason itself into a worse place.

The second attempt was a per-category ceiling: lookups get three steps, investigations get ten, routed by a classifier. This helped, but it was brittle. The classifier mis-routed maybe 15% of questions, and every mis-route inherited the wrong ceiling. It also didn't generalize — every new question type needed a new bucket and a new tuned number. They were hand-maintaining a lookup table that difficulty should have been deciding dynamically.

The Correct Prompt

The version that worked asked the model to self-report confidence and justify it, then used that signal to drive stopping. The key was making confidence concrete and grounded, not a vague number.

After each step, before deciding whether to continue, output:

CONFIDENCE: <0-100>
BASIS: <one sentence: what specific evidence supports your answer,
        and what, if anything, is still missing or uncertain>

Stop and give your final answer when CONFIDENCE >= 85 AND nothing
material is missing. If confidence is below 85, take another step
that targets the specific missing piece named in BASIS.

What this does: It forces the model to name the evidence behind its confidence and the specific gap that's keeping it uncertain, which both produces a usable stop signal and steers the next step toward closing the real gap instead of wandering.

The

BASIS

[object Object], ,[object Object],(,[object Object],):
    ,[object Object], step ,[object Object], ,[object Object],(max_steps):
        out = model_step(state)          ,[object Object],
        ,[object Object], out.confidence >= conf_threshold ,[object Object], out.nothing_missing:
            ,[object Object], out.answer
        state = update_state(state, out.action, run_tool(out.action))
    ,[object Object], force_answer(state)            ,[object Object],

What this does: It reads the model's self-reported confidence each step and stops as soon as the threshold is met with no missing evidence, falling back to the step ceiling only as a last resort rather than as the primary control.

Results and What Changed

Over three weeks, wrong answers dropped 44%. Easy questions now stopped at their natural point — median steps for lookups fell from 6 to 2. Hard investigations that used to truncate at 6 now ran to 8 or 9 when confidence justified it, and their accuracy climbed accordingly. Total token spend fell 19% despite hard questions running longer, because the huge volume of easy questions stopped wasting steps.

The most useful side effect was observability. Because every step now logged a confidence score and a basis, the team could see exactly where the agent felt shaky. Clusters of low-confidence runs pointed straight at documentation gaps — questions the agent couldn't answer confidently because the source material genuinely didn't cover them. Agent confidence scoring became a product signal, not just a control mechanism.

⚡ Pro tip: Log the confidence trajectory, not just the final value. A run that climbs steadily to 90 is healthy; a run that oscillates between 60 and 80 for six steps is thrashing, and one that jumps to 95 in a single step may be overconfident. The shape of the curve tells you more than the endpoint.

One cost worth naming: asking for a confidence score and basis every step adds tokens and a little latency to each iteration. In practice it paid for itself many times over here, because the steps it saved on easy questions vastly outnumbered the small per-step overhead. But it isn't free, and on a very high-volume, low-margin agent you may want the confidence check only every other step, or only once the agent believes it has an answer. Measure the overhead against the steps saved before assuming agent confidence scoring is a pure win — for this team it was, but the arithmetic depends on your task mix.

⚡ Pro tip: If per-step confidence output is too expensive, run it lazily: let the agent act normally, and only when it signals "I think I'm done" do you trigger the confidence-and-basis check to decide whether it really is. This gets you the stopping benefit at a fraction of the token cost, since you pay for scoring once near the end instead of on every step.

How to Apply This to Your Situation

Start by having your agent emit a confidence score and a one-line basis each step, and just log them without changing behavior. Watch the distribution for a week. You'll learn what "confident enough" actually looks like in your domain before you wire it to stopping.

Then set your threshold empirically, not by gut. Find runs you know were correct and see what confidence they reported when they got the answer right; set the threshold near the low end of that band. Too high and the agent never feels sure enough and grinds; too low and it stops half-baked.

Consider using different thresholds for different action types rather than one global number. An agent that's about to give a read-only answer can stop at 80; the same agent about to trigger an irreversible action should demand 95, because the cost of a confident-but-wrong destructive act is far higher than a confident-but-wrong sentence. Tying the bar to consequences, not just to the model's feeling, is where confidence scoring stops being a nice metric and starts being a real safety control. The single number that worked for stopping is rarely the right number for acting.

One more refinement that paid off for this team: they had the agent report confidence separately for each distinct claim in a multi-part answer, rather than one score for the whole response. A question with four parts often has three parts the agent is sure of and one it's shaky on. A single blended score hides that, and the agent either over-verifies the sure parts or ships the shaky one. Per-claim confidence let it target the one weak spot and leave the rest alone.

⚡ Pro tip: When an agent reports low confidence repeatedly on the same topic across many runs, that's not an agent problem — it's a data problem. Route those clusters to whoever owns your knowledge base. Persistent low confidence is a map of exactly where your source material has holes.

⚠️ Common mistake: Trusting raw self-reported confidence as calibrated probability. Models are often overconfident and their "90" may empirically mean "right 70% of the time." Don't treat the number as a true probability. Use it as a relative signal, calibrate the threshold against known-correct runs, and if you need real probabilities, map the raw scores to observed accuracy with a small validation set.

Three teams applied this differently. A cybersecurity triage agent used confidence scoring to decide when to escalate to a human instead of when to stop — low confidence routed the case to an analyst. A financial-analysis agent required two independent high-confidence passes before releasing a number, trading cost for safety on high-stakes outputs. And a customer-support agent used the basis line to auto-flag answers where the missing piece was "I couldn't find this in the knowledge base," turning uncertainty into a content-gap backlog.

Next Steps

Add a confidence-and-basis output to your loop, log it before you act on it, calibrate the threshold against known-correct runs, then let it drive stopping with the step ceiling demoted to a safety net. Expect the biggest wins on the two ends of your difficulty distribution.

Confidence prompts are fiddly to word well, and small changes to the basis instruction shift behavior a lot. Once you've tuned wording that calibrates cleanly for your domain, keep it. A prompt library like PromptABCD is a good home for your confidence-scoring blocks and threshold notes, so your next agent inherits a stopping rule that already works instead of starting blind.