Persisting Agent State in the Harness

What actually needs to be saved for an agent run to survive a crash — and what should never be written to disk at all? That question sits at the center of agent state persistence, and getting the answer wrong in either direction hurts. Save too little and a resumed run is missing the context it needs to continue correctly. Save too much — or the wrong things — and you've got secrets in your database, a state schema you can't evolve, and blobs so large they slow every checkpoint. This post draws the line carefully, because the line is where most persistence bugs live.

The reason this is subtle is that an agent's "state" isn't one thing. It's the conversation, yes, but also counters, pending actions, tool results, and metadata about the run itself — and each of those has different rules about how it should be stored, versioned, and secured.

What Is Agent State Persistence?

Agent state persistence is durably saving everything a run needs to be reconstructed and continued from a point in time. It's what makes a run survivable — able to outlast a crash, a deploy, a worker reassignment, or a pause for human input. Without it, a run exists only in the memory of one process, and that process is mortal.

The unit of persistence is a checkpoint: a complete, self-contained snapshot of the run at a step boundary. "Self-contained" is the operative word. A good checkpoint contains everything needed to resume with no reference to the process that created it, because that process may be long gone by the time someone resumes.

[object Object],
,[object Object], ,[object Object],:
    run_id: ,[object Object],
    step: ,[object Object],
    schema_version: ,[object Object],          ,[object Object],
    messages: ,[object Object],               ,[object Object],
    used_tokens: ,[object Object],             ,[object Object],
    pending_action: ,[object Object], | ,[object Object],  ,[object Object],
    tool_cursor: ,[object Object],            ,[object Object],

What this does: Defines a checkpoint as a snapshot carrying the conversation, the step and token counters, any action the run was about to take, and a

schema_version

schema_version

Why It Matters

The immediate payoff is survivability, but the deeper reason to take agent state persistence seriously is that state is the thing you can't regenerate. You can re-run a model call; you can't reconstruct the exact conversation that led to step 40 if you never saved it. State is the record of a path through a space too large to retrace, which is exactly why losing it is so costly.

There's also a correctness dimension. A partially-saved checkpoint — messages updated but the counter not, or the pending action written but the tool cursor stale — resumes into an inconsistent state that produces subtly wrong behavior. Persistence isn't just "write it down"; it's "write a consistent snapshot atomically," so a resumed run is always a run that could have existed.

⚠️ Common mistake: Persisting the conversation but not the run's counters and cursors. On resume, the run has the messages but has forgotten it already spent 180K tokens or already processed 8 of 10 batch items — so it blows its budget or reprocesses work. The conversation is necessary but not sufficient; the bookkeeping around it has to travel too.

Choosing What to Persist

The discipline is to persist everything needed to resume and nothing that's either regenerable or dangerous.

Persist the conversation, the counters (tokens, step, spend), any pending or in-flight action, and cursors that track progress through multi-part work. Persist enough metadata — run ID, status, timestamps — to find and manage the run later.

Do not persist secrets, raw credentials, or full personal data. This is where agent state persistence meets secrets management: the state store is durable and widely read, so a credential in a checkpoint is a credential leaked to backups, replicas, and anyone with database access. Store secret references, never values, exactly as you do everywhere else.

Do not persist things you can cheaply regenerate. Large tool outputs are often better stored by reference — write the blob to object storage, keep a pointer in the checkpoint — than inlined, which keeps checkpoints small and fast to read and write.

[object Object], ,[object Object],(,[object Object],):
    ,[object Object], m ,[object Object], cp.messages:
        m[,[object Object],] = scrub_secrets(m[,[object Object],])      ,[object Object],
    large = extract_large_blobs(cp)                      ,[object Object],
    store.put(cp.run_id, cp.step, asdict(cp), blobs=large)

What this does: Scrubs secrets out of the conversation before writing, offloads oversized tool outputs to blob storage keeping only references in the checkpoint, and stores the result keyed by run and step. The checkpoint stays lean and secret-free, and the heavy data lives where heavy data belongs.

⚡ Pro tip: Store checkpoints keyed by

(run_id, step)

run_id

Versioning State So You Can Change It

The reality nobody warns you about: your state schema will change, and when it does, there will be in-flight runs checkpointed under the old shape. If your loader assumes the current shape, every one of those runs breaks on resume the moment you deploy the change. This is why

schema_version

[object Object], ,[object Object],(,[object Object],):
    raw = store.get(run_id, step)
    v = raw.get(,[object Object],, ,[object Object],)
    ,[object Object], v < CURRENT_VERSION:
        raw = MIGRATIONS[v](raw)     ,[object Object],
        v += ,[object Object],
    ,[object Object], Checkpoint(**raw)

What this does: Reads the checkpoint's version and runs it forward through migration functions until it matches the current shape, so an old checkpoint loads correctly even after the schema evolved. A run started last week under version 2 resumes today under version 4 without special-casing. The migration chain is small insurance against a large class of resume failures.

⚡ Pro tip: Write the migration before you ship the schema change, and test it against a real old checkpoint. The tempting shortcut — "I'll migrate old runs later" — means the moment you deploy, every in-flight run is broken until "later" arrives. A migration written alongside the change makes schema evolution boring instead of an incident.

Choosing and Testing Your State Store

Where checkpoints live matters as much as what's in them, and the right choice follows from one property: durability under concurrent access. A relational database is the safe default — transactions give you atomic checkpoint writes for free, and

SELECT ... FOR UPDATE SKIP LOCKED

Whatever you pick, the store for agent state persistence should be one you'd trust with your most important data, because that's what it holds — the irreplaceable record of every in-flight run. The in-memory cache that's fast and occasionally evicts things is exactly wrong here; an eviction is a lost run.

The part teams skip is testing that persistence actually works, and the only honest test is a real interruption. Save a checkpoint mid-run, kill the process, load the checkpoint in a fresh process, and confirm the run continues correctly to completion.

[object Object], ,[object Object],():
    state = run_until_step(prompt, stop_at=,[object Object],)
    save_checkpoint(store, state)
    ,[object Object], state                              ,[object Object],
    restored = load_checkpoint(store, run_id, step=,[object Object],)
    final = run_from(restored)             ,[object Object],
    ,[object Object], final.done ,[object Object], final.answer     ,[object Object],

What this does: Runs partway, checkpoints, discards the in-memory state to mimic a crash, reloads purely from the store, and confirms the run finishes correctly from the restored state. If this test passes, your persistence is real; if it only passes when you don't discard the state, you have a hidden dependency on in-memory data that a real crash will expose at the worst time.

⚡ Pro tip: Run the crash-recovery test in CI at a few different step boundaries, not just one. A checkpoint that resumes fine from step 5 can still be broken at step 40 if some field only appears later in a run. Testing resume at several points catches the "we forgot to persist X, but only long runs touch X" class of bug before a long run finds it in production.

Common Mistakes

The mistake that quietly corrupts runs is non-atomic checkpoint writes. If saving a checkpoint involves multiple writes and a crash lands in the middle, you resume from a half-written state. Write the checkpoint as a single atomic operation — one row, one object, one transaction — so a checkpoint either fully exists or doesn't exist at all.

The second is unbounded conversation growth. A long run's message history grows every step, and naive persistence rewrites the whole thing each time, making checkpoints slower as the run goes. For very long runs, persist messages incrementally or summarize older turns, so checkpoint cost stays flat instead of climbing with run length.

The third is treating the state store as an afterthought with weak durability. This is the one store where losing a write means losing irreplaceable progress. It deserves your durable, backed-up, replicated storage — not the cache you reach for because it's fast.

Conclusion

Agent state persistence is the discipline of saving a consistent, secret-free, versioned snapshot that lets any worker resume a run from a step boundary. Persist the conversation and its bookkeeping, offload big blobs, never persist secrets, version everything, and write atomically. Do that and a crash becomes a pause; skip it and a crash becomes lost work you can't recreate.

Because the shape of your state — what's in a checkpoint, how it's versioned, what's scrubbed — is a design that every resumable agent shares, keep it versioned alongside your prompts in a library like PromptABCD, so the checkpoint format you got right once is the one your next agent inherits, migrations and all.