Checkpointing Agent State Between Iterations

Picture this: you're running an agent that takes twenty minutes to work through a large batch job, and nineteen minutes in, the process crashes. Without checkpointing, you start over — twenty minutes and the tokens gone, and if it crashes again you're stuck in a loop of expensive restarts. With agent loop checkpointing, the crash costs you the last step and nothing more; you resume nineteen minutes in and finish. That gap — lose everything versus lose one step — is what checkpointing buys, and it's cheaper to build than most people assume.

Checkpointing means saving the agent's state after each iteration so you can restore it and continue. It's the mechanism underneath resumability, human-approval pauses, and crash recovery. This guide shows you what to save, how to save it cheaply, and how to resume without corruption.

The reason checkpointing feels harder than it is comes down to one worry: won't saving state every step make the loop slow and expensive? For most agents, no — and the intuition is off because the thing you're saving is small relative to the thing you're already spending. A model call sends thousands of tokens and takes a second or more; serializing a compact state object and writing it takes milliseconds. Checkpointing rides in the shadow of costs you're already paying. Once you see that the save is cheap next to the step it follows, the reluctance to checkpoint every iteration mostly evaporates.

Quick-Start (Copy This Right Now)

[object Object], ,[object Object],(,[object Object],):
    state = Checkpoint.load(run_id) ,[object Object], Checkpoint.new(task)
    ,[object Object], ,[object Object], state.done ,[object Object], state.step < state.max_steps:
        reply = model.call(state.messages)
        state.apply(reply)
        ,[object Object], c ,[object Object], reply.tool_calls:
            state.record(c.,[object Object],, run_tool(c))
        state.step += ,[object Object],
        state.save(run_id)          ,[object Object],
    ,[object Object], state.result

What this does: loads any existing checkpoint for the run (or starts fresh), advances one step, and saves the updated state after every completed iteration — so an interruption at any point leaves a consistent checkpoint to resume from.

The load-at-top, save-at-bottom pattern is the whole idea: every run begins by trying to resume, and every step ends by checkpointing. A crash anywhere leaves a clean point to restart from.

Understanding the Variables

Three questions define good agent loop checkpointing.

What to save — the minimum state needed to continue: the transcript or its compaction, the current step, the plan, completed actions, and any accumulated results. When to save — after each completed step, at a consistent boundary, never mid-action. And how to resume — load the checkpoint, verify it's consistent, and continue the loop as if it had never stopped. Get these three right and interruption becomes a non-event.

The subtle one is what to save. Too little and you can't actually resume — you're missing the plan or the completed-actions list and the agent redoes work. Too much and checkpointing gets slow and expensive. You want exactly the state that determines the next step, and nothing decorative.

A useful test for whether something belongs in the checkpoint: would the agent behave differently on resume if this were missing? The plan, the completed-actions list, the accumulated results all pass — lose them and the agent redoes work or contradicts itself. A verbose reasoning transcript from step two usually fails — the agent's next move doesn't depend on the exact words it thought earlier, only on the conclusions those words reached. Checkpoint the conclusions, not the deliberation, and your saves stay small without losing anything the resume actually needs.

⚡ Pro tip: Save the state, not the whole history, when you can. If your loop uses compaction, checkpoint the compacted state plus recent turns rather than the full raw transcript. A lean checkpoint is faster to write every step and just as complete for resuming, since resuming needs current state, not the entire narrative.

Step-by-Step: Agent Loop Checkpointing

Build it in four moves.

First, define a serializable state object — everything needed to continue, and nothing that can't be saved. Second, load at the top of the loop, so every run starts by attempting to resume. Third, save at the bottom of each step, at a clean boundary. Fourth, make saves atomic, so a crash during a save doesn't corrupt the checkpoint.

[object Object], ,[object Object],(,[object Object],):
    tmp = ,[object Object],
    write(tmp, serialize(state))        ,[object Object],
    os.replace(tmp, ,[object Object],)   ,[object Object],

What this does: writes the checkpoint to a temporary file and atomically renames it into place, so the real checkpoint is only ever a complete, valid state — a crash mid-write leaves the previous good checkpoint untouched.

[object Object], ,[object Object],(,[object Object],):
    ,[object Object], ,[object Object], exists(,[object Object],):
        ,[object Object], ,[object Object],
    state = deserialize(read(,[object Object],))
    ,[object Object], state.is_consistent(), ,[object Object],    ,[object Object],
    ,[object Object], state

What this does: loads and validates a checkpoint before the loop trusts it, so a corrupt or partial state is caught at resume time instead of silently continuing from a broken point.

⚠️ Common mistake: Checkpointing mid-action or non-atomically. If you save state while a tool call is half-done, or write the checkpoint in place so a crash can leave a half-written file, you resume into a corrupt state that's worse than starting over — because it looks valid and fails subtly. Always checkpoint at a step boundary, and always write atomically. A checkpoint you can't trust is not a checkpoint.

Pro-Level Variations

Checkpointing strategy shifts with the cost of a step.

A data-pipeline team running expensive, hours-long agents checkpoints aggressively — every step, because each step represents real money they never want to repeat. A latency-sensitive consumer agent checkpoints more selectively — every few steps or on meaningful milestones — because a synchronous save on every fast step adds felt latency. A high-throughput batch system checkpoints to a fast store and asynchronously, so saving never blocks the loop's forward progress.

Same checkpointing mechanism, three cadences — tuned to how much a lost step actually costs against how much a save slows things down.

⚡ Pro tip: Keep more than the latest checkpoint — retain the last few. If the most recent checkpoint turns out to encode a bad state (a step that poisoned the run), a one-deep history lets you roll back to a known-good point instead of being stuck resuming into the same failure. A tiny ring buffer of recent checkpoints turns "corrupt latest state" from a dead end into a rewind.

⚡ Pro tip: Checkpoint asynchronously when saves are slow. Fire the save to a background writer and let the loop proceed, so checkpointing adds no latency to the critical path. The small risk — losing the very last step if a crash beats the async write — is usually worth the speed, and you can make the final answer's checkpoint synchronous for safety.

Troubleshooting Common Issues

If resuming redoes completed work, your checkpoint is missing the completed-actions state — you saved the transcript but not the record of what's already done. Add the completed list to what you checkpoint.

If checkpointing slows the loop noticeably, your state is too big or your writes are synchronous on a slow store. Trim to essential state, or move saves to a background writer.

If resumes occasionally start from a corrupt state, your saves aren't atomic — a crash caught one mid-write. Switch to write-temp-then-rename so a checkpoint is only ever complete, never a half-written file a crash left behind.

⚡ Pro tip: Test resume by killing the process at random steps, not just at the end. A checkpoint system that only ever resumes cleanly from the final step is untested where it matters. Crash it at step three, step seven, mid-batch — and confirm each resume continues correctly. The bugs live in the awkward middle, not the tidy end.

Your Turn

Take a long-running agent and add checkpointing: define a serializable state object, load at the top, save atomically at each step boundary, and verify on load. Then prove it works by killing the process mid-run and resuming — the agent should continue as if nothing happened. That test is the whole point, so run it before you trust the system.

It's worth making that the actual acceptance criterion, not an afterthought: a checkpoint system you haven't crash-tested is a checkpoint system you don't know works, and the failure mode — discovering the resume is broken during a real outage — is the worst possible time to find out. Wire a "kill at random step" flag into your test harness and let it run a dozen times. Resumability that survives that gauntlet is resumability you can actually rely on when a real crash arrives unannounced.

The checkpoint object and the atomic save/load helpers are reusable across every long-running agent. I keep them saved and versioned in PromptABCD, so a new agent survives crashes from day one — instead of the fragile default where a process dying at minute nineteen throws away the whole run and the tokens that paid for it. Crash recovery stops being a heroic incident and becomes a line in a log.