Retry Logic in an AI Harness: Safe by Default

A payments agent I reviewed once charged a customer's card twice. The first

charge

Before: The Naive Retry Wrapper

Here's the retry logic almost every harness starts with:

[object Object], ,[object Object],(,[object Object],):
    ,[object Object], attempt ,[object Object], ,[object Object],(tries):
        ,[object Object],:
            ,[object Object], fn(**args)
        ,[object Object], Exception:
            ,[object Object],          ,[object Object],
    ,[object Object], ,[object Object],

What this does: it runs a tool, and on any exception, it runs it again — up to three times. It looks defensive and responsible. It's the code that double-charged the card, because it retries everything, including operations that already succeeded and operations that should never run twice.

Why It Fails

The naive wrapper makes two errors, and both are about failing to distinguish between kinds of failure.

First, it retries the un-retryable. A

400 Bad Request

Second, and far more dangerous, it retries operations with side effects that already happened. A charge, an email send, a file delete, a database insert — if the operation completed but the response was lost to a timeout, retrying does the thing again. The naive wrapper can't tell "the operation failed" from "the operation succeeded but I didn't hear back," so it assumes failure and doubles the action.

⚠️ Common mistake: Retrying non-idempotent operations without an idempotency key. Any tool that changes external state — charges, sends, creates, deletes — can double-execute when a retry fires after a lost response. This is the failure that costs real money and real trust, and a blanket retry wrapper walks straight into it every time the network hiccups at the wrong moment. The tell is subtle: the operation looks like it failed from your side while having succeeded on the server's side, and only a shared key lets the two sides agree on what really happened.

After: Classified, Idempotent Retry

The fix sorts errors into three buckets and treats each correctly:

[object Object], time, random

RETRYABLE = {,[object Object],, ,[object Object],, ,[object Object],, ,[object Object],}
FATAL = {,[object Object],, ,[object Object],, ,[object Object],, ,[object Object],}

,[object Object], ,[object Object],(,[object Object],):
    ,[object Object], ,[object Object], idempotent:
        ,[object Object],
        args = {**args, ,[object Object],: args.get(,[object Object],) ,[object Object], new_key()}
    ,[object Object], attempt ,[object Object], ,[object Object],(tries):
        ,[object Object],:
            ,[object Object], fn(**args)
        ,[object Object], ToolError ,[object Object], e:
            kind = classify(e)
            ,[object Object], kind ,[object Object], FATAL:
                ,[object Object], ,[object Object],      ,[object Object],
            ,[object Object], kind ,[object Object], ,[object Object], RETRYABLE:
                ,[object Object], ,[object Object],
            sleep = ,[object Object],(,[object Object], ** attempt + random.random(), ,[object Object],)  ,[object Object],
            time.sleep(sleep)
    ,[object Object], ,[object Object],

What this does: it refuses to retry fatal errors that can't succeed, backs off exponentially with jitter for genuinely transient ones, and — crucially — attaches an idempotency key to any non-idempotent operation so a retried charge dedupes at the server instead of running twice. The double-charge becomes structurally impossible, because the second request carries the same key as the first and the payment processor recognizes it as a duplicate.

Breaking Down the Retry Decision

Three questions decide whether and how to retry, and answering them in order keeps you out of trouble.

Is this error transient? Timeouts, rate limits, and 5xx responses are worth retrying — the operation might succeed once the blip passes. A 4xx almost never is; the request itself is wrong. Retrying a fatal error is a slower way to reach the same failure.

Is this operation idempotent? Reads, pure computations, and lookups can retry freely — running them twice changes nothing. Writes and side effects cannot, unless the downstream system dedupes them. This is the axis the naive wrapper ignored entirely.

If it's not idempotent, can I make the retry safe? Usually yes, with an idempotency key the server uses to recognize duplicates. Most payment and messaging APIs support this exact pattern for exactly this reason. Generate the key once, before the first attempt, and reuse it across retries.

A fourth question quietly decides how much any of this helps: can I see my retries? Retries that happen invisibly are retries you can't reason about. A harness should record every attempt — which tool, which error kind, which attempt number, how long the backoff was — so that when a run is slow or expensive, you can tell whether it's the model thinking or the harness silently retrying a flaky dependency eight times. I've debugged agents that looked like they had a reasoning problem and turned out to be spending 90% of their wall-clock time in backoff sleeps against a degraded API. Without retry-level logging, that's invisible; with it, it's a single glance at the trace. Instrument the retries, and a whole category of "why is this agent so slow" mysteries resolves itself.

[object Object], ,[object Object],(,[object Object],):
    ,[object Object],(,[object Object],)

What this does: it emits one line per retry with the tool, attempt number, error kind, and how long the harness waited. Aggregated over a run, these lines show you exactly where your time and money went — and a sudden run of them against one tool is your earliest signal that a dependency is failing.

⚡ Pro tip: Generate the idempotency key outside the retry loop, at the moment you first decide to call the tool. If you generate it inside, each retry gets a fresh key and the server sees three distinct requests — which defeats the whole mechanism. One key per logical operation, reused across every attempt of that operation.

When to Stop Retrying Entirely

Classification tells you whether a single call should retry. Circuit breaking tells you when to stop trying at all — and mature agent harness retry logic needs both. If a downstream service is fully down, retrying every call against it just multiplies your failures and your latency. A circuit breaker notices the pattern and fails fast until the service recovers.

[object Object], ,[object Object],:
    ,[object Object], ,[object Object],(,[object Object],):
        ,[object Object],.fails = ,[object Object],
        ,[object Object],.open_until = ,[object Object],
        ,[object Object],.threshold = threshold
        ,[object Object],.cooldown = cooldown

    ,[object Object], ,[object Object],(,[object Object],):
        ,[object Object], time.time() >= ,[object Object],.open_until

    ,[object Object], ,[object Object],(,[object Object],):
        ,[object Object], ok:
            ,[object Object],.fails = ,[object Object],
        ,[object Object],:
            ,[object Object],.fails += ,[object Object],
            ,[object Object], ,[object Object],.fails >= ,[object Object],.threshold:
                ,[object Object],.open_until = time.time() + ,[object Object],.cooldown

What this does: it counts consecutive failures against a service, and once they cross a threshold, it "opens" — refusing further calls for a cooldown window instead of retrying into a wall. After the cooldown it allows a probe call; success resets the counter, failure re-opens the breaker. This stops one dead dependency from consuming your entire retry budget and your whole time allowance.

The breaker also gives the model better information. Instead of a string of identical timeout messages, it gets a clear "this service is unavailable, try a different approach" — which lets the agent route around the outage rather than pounding on it. A retry policy that only ever says "try again" traps the agent in a loop against something that isn't coming back soon.

⚡ Pro tip: Trip the breaker per-dependency, not globally. One flaky API shouldn't stop your agent from calling three healthy ones. Keep a breaker keyed by service name so an outage in the payment provider doesn't block the unrelated email tool — isolate the failure to the thing that's actually failing.

A platform engineer running an agent that orchestrates five external APIs uses per-service breakers so a single vendor's outage degrades one capability gracefully instead of taking the whole agent offline — the difference between "one feature is temporarily down" and "the agent is broken."

Variations for Different Contexts

The right retry policy shifts with what the tool does:

• A fintech engineer wrapping payment tools makes idempotency keys mandatory and caps retries low — a duplicate charge is far worse than a failed one, so the policy leans toward giving up cleanly and surfacing the failure to a human.
• A data engineer running idempotent warehouse queries retries freely with long backoff, because re-running a

  SELECT

  Copy
  is harmless and transient warehouse hiccups are common. Here, aggressive retry is the right call.
• A DevOps engineer whose agent calls deployment APIs distinguishes sharply:

  get_status

  Copy
  retries liberally,

  trigger_deploy

  Copy
  retries only with a deduplication token, so a lost response can't launch two deploys.

Same retry machinery, three policies — set by the cost of a duplicate action versus the cost of a failed one.

⚡ Pro tip: Add jitter to your backoff even when it feels unnecessary. Without it, a fleet of agents that all hit the same rate limit will retry in perfect lockstep and hammer the API at the same instants — a self-inflicted thundering herd. The

random.random()

Save and Reuse This

Good agent harness retry logic is mostly about refusing to retry the wrong things. Classify the error before you retry — fatal errors don't get a second chance. Check idempotency before you retry a side effect — and if the operation changes external state, carry an idempotency key so a lost response can't double-execute. Add backoff with jitter so transient failures resolve without a stampede. The naive "just try again" wrapper gets all three wrong, and one of those wrongs charges a card twice.

The tool descriptions that tell the model which operations are safe to repeat, and the error-classification rules that drive your retries, are worth keeping in one place. Storing those retry policies and descriptions in a prompt library like PromptABCD — tagged by which tools they govern — means the next side-effecting agent you build inherits the idempotency discipline you already worked out, instead of rediscovering it through a duplicate transaction. Retry logic is easy to get subtly wrong once; a saved, proven policy keeps you from getting it wrong twice.