Resuming an Interrupted Agent Run

Most guides to agent reliability are wrong about what to do when a run fails partway: they tell you to retry the run. Restarting a three-hour run from scratch because it died at minute 170 isn't reliability — it's waste, and worse, it re-executes every side effect the run already committed. The teams that actually run agents at scale don't restart; they resume. A resume agent run harness picks up an interrupted run at the exact step it stopped, without redoing what it already did. This is a case study of one team that made the switch and everything they had to get right to do it safely.

The contrarian core here: retry and resume look similar and behave completely differently. Retry throws away progress and repeats side effects. Resume preserves progress and continues. For anything long-running or side-effectful, resume is the only correct answer — and it's harder than it looks, because the interesting failures happen mid-action.

The Problem This Team Faced

The team ran data-processing agents that took twenty minutes to two hours per run, each performing dozens of writes to external systems along the way. Their reliability strategy was retry-on-failure: if a run crashed, restart it.

It half-worked and half-created new problems. Restarting a run that had already sent 30 of 50 emails sent the first 30 again. Restarting one that had already updated inventory re-applied the updates, double-counting stock. And restarting a two-hour run that died near the end burned two more hours of compute to redo work that had already succeeded. Their "reliability" mechanism was generating duplicate side effects and doubling their costs.

⚠️ Common mistake: Using retry as your reliability strategy for runs with side effects. Retry assumes the run is a pure function you can safely re-execute — but an agent that writes to the world is not pure, and re-executing it repeats those writes. The more successful work a run did before failing, the more damage a naive retry does.

The Wrong Approach

Their first attempt to fix the duplicates was to make the run detect where it had gotten to by inspecting the world — "have I already emailed this person?" — before each action. It was fragile and slow.

[object Object], ,[object Object],(,[object Object],):
    ,[object Object], item ,[object Object], items:
        ,[object Object], already_emailed(item):     ,[object Object],
            ,[object Object],
        send_email(item)

What this does: Before each email, queries whether that person was already emailed, skipping them if so. It technically avoids duplicates, but it adds a round-trip per item on every restart, it only works for actions you can check after the fact, and it falls apart for actions with no clean "did I do this?" query. Reconstructing progress by interrogating the world is a losing game — the world doesn't reliably remember what your run was doing.

The deeper flaw was that this only patched one tool. Every side-effectful action needed its own bespoke "did I already do this?" check, and any the team forgot silently double-fired. They were reconstructing state from the outside when the run should have been recording it from the inside.

The Correct Approach

The fix was to make the run journal its own progress as it went, so resuming means reading the journal — not interrogating the world. The harness already checkpointed state at each step; the change was to also record, durably, each side effect before committing it and mark it done after, so a resumed run knows exactly what completed, what was in flight, and what remains.

[object Object], ,[object Object],(,[object Object],):
    action = state.pending_action
    entry = journal.get(action.,[object Object],)
    ,[object Object], entry ,[object Object], entry.status == ,[object Object],:
        ,[object Object], advance(state)                 ,[object Object],
    ,[object Object], entry ,[object Object], entry.status == ,[object Object],:
        ,[object Object],
        ,[object Object], action.is_idempotent ,[object Object], verify_committed(action):
            journal.mark(action.,[object Object],, ,[object Object],)
            ,[object Object], advance(state)
    journal.mark(action.,[object Object],, ,[object Object],)        ,[object Object],
    execute(action)
    journal.mark(action.,[object Object],, ,[object Object],)      ,[object Object],
    ,[object Object], advance(state)

What this does: Consults a durable journal keyed by a stable action ID. If the action already committed, it's skipped. If it was started but not committed — the crashed-mid-action case — the run reconciles rather than blindly redoing. Only genuinely new actions execute, bracketed by "started" and "committed" markers. A resumed run reads this journal and knows precisely where it was, no queries to the outside world required.

The subtle case — the one the team kept hitting — is a crash between "started" and "committed," where you can't tell from the journal alone whether the action actually happened. That's where idempotency keys carry the day.

[object Object], ,[object Object],(,[object Object],):
    ,[object Object],
    provider.send(to=item.email, body=item.body, idempotency_key=idem_key)

What this does: Passes a stable idempotency key derived from the action so the email provider itself refuses to send a duplicate if the same key arrives twice. Now the ambiguous "did it send?" case is safe either way — resuming and re-calling with the same key is a no-op if it already sent. The journal narrows the uncertainty; idempotency keys eliminate the damage from what's left.

Results and What Changed

Duplicate side effects went to zero. A run that died at email 30 of 50 resumed at email 31 — not email 1 — and the provider's idempotency key covered the one email that might have been in flight during the crash. The double-counted inventory problem vanished for the same reason.

The cost change was just as stark. Their worst case went from "redo two hours of work" to "redo one step," because resume continues from the last checkpoint instead of the beginning. A resume agent run harness turned failures from expensive, damaging events into cheap, invisible ones — most interruptions now resolve without anyone noticing, because the run simply continues.

⚡ Pro tip: Record intent before the action and completion after, always in that order. If you record only after, a crash mid-action leaves no trace it was ever attempted, and you can't tell "never started" from "started and maybe finished." The before-and-after bracket is what makes the ambiguous middle case detectable instead of invisible.

How to Apply This to Your Situation

Start by sorting your agent's actions into three buckets: read-only (safe to redo, no journaling needed), idempotent writes (safe to redo with an idempotency key), and non-idempotent writes (dangerous to redo, need journaling and idempotency). Most of the work goes into that last bucket, and identifying it honestly is half the battle — teams routinely assume an action is idempotent when it isn't.

Then add the journal. It can share your checkpoint store; it's just a durable, per-run record of which actions started and committed. Bracket every side-effectful action with started/committed markers, and pass an idempotency key to every external write that supports one. On resume, replay the journal to skip committed actions and reconcile started-but-uncommitted ones.

⚡ Pro tip: Derive idempotency keys from the action's meaning, not from a random value generated at call time. A key of

email:{run_id}:{recipient}

When Resuming Isn't Enough: Compensating Actions

Sometimes a run can't be resumed forward — it has to be partly undone. If an agent got halfway through a multi-step transaction and then hit an unrecoverable error, leaving the work half-applied is often worse than either finishing or reversing it. A mature resume agent run harness knows the difference between "continue from here" and "this can't continue, so unwind what was done."

The pattern borrows from distributed systems: for each committed action, record how to compensate for it — the inverse operation — so a run that can't move forward can walk its journal backward, undoing committed steps in reverse order.

[object Object], ,[object Object],(,[object Object],):
    ,[object Object], entry ,[object Object], ,[object Object],(journal.committed()):
        undo = COMPENSATIONS.get(entry.action_type)
        ,[object Object], undo:
            undo(entry.args)                  ,[object Object],
            journal.mark(entry.,[object Object],, ,[object Object],)

What this does: Walks the committed actions newest-first and applies each one's registered inverse — reversing a charge, deleting a created record — marking each as compensated. A run that created a mess it can't finish cleaning can at least return the world to a consistent state instead of leaving it stranded midway. Not every action has a clean inverse, which is itself useful to know in advance.

The honest caveat: compensation is best-effort, and some actions genuinely can't be undone — an email that was sent is sent. That's exactly why the earlier disciplines matter so much. The more you can push irreversible actions behind approval gates and toward the end of a run, the smaller the window where an interruption leaves something you can neither finish nor reverse.

⚠️ Common mistake: Assuming every failed run should be compensated. Compensation is for runs that are genuinely stuck and shouldn't proceed; most interruptions just need a plain resume, which is cheaper and safer. Reach for compensation only when forward progress is truly impossible — reversing work you could have simply continued wastes effort and adds its own risk of error.

Next Steps

Resuming safely depends entirely on solid state persistence underneath it — the journal and checkpoints are what a resume reads from, so a resume agent run harness is only as reliable as the state it saved. The two are inseparable: get persistence right and resume becomes straightforward; get it wrong and no resume logic can save you.

Keep your journaling conventions, idempotency-key formats, and reconciliation rules versioned alongside your prompts in a library like PromptABCD, so the hard-won knowledge of how to resume this kind of action safely travels with the agent — and the next long run someone builds doesn't relearn the duplicate-email lesson the expensive way.

⚡ Pro tip: Test your resume path by killing runs on purpose, at random steps, in a staging environment. A resume path that's never exercised until a real crash is a resume path you don't actually know works — and the first real crash is the worst time to discover a bug in the code that's supposed to save you. Make interruption a routine test, not a production surprise.