Building a Harness for Long-Running Agents

Picture this: you're building an agent that migrates a codebase, and a single run legitimately takes four hours — hundreds of model calls, thousands of tool invocations, the occasional pause for a human to approve a risky change. Your first version ran it inside a web request, the request timed out at 30 seconds, and everything past that point evaporated. A long running agent harness is a different animal from the request-response agents most guides teach, and the differences aren't cosmetic — they change the fundamental architecture. This guide walks through building one that survives hours, restarts, and dead workers.

The core realization is that a long-running run cannot live inside the thing that started it. A web request, a Lambda invocation, a single process — all of these have lifetimes measured in seconds to minutes, and your run needs to outlive all of them. Once you internalize that, the architecture follows, and most of the "hard" parts of long-running agents turn out to be consequences of that one decision rather than separate problems to solve.

Quick-Start (Copy This Right Now)

Here's the shape that unlocks everything else: separate the run (a durable record) from the worker (an ephemeral process that advances it one step at a time).

[object Object], ,[object Object],(,[object Object],):
    ,[object Object], ,[object Object],:
        run_id = queue.claim(lease_seconds=,[object Object],)     ,[object Object],
        ,[object Object], run_id ,[object Object], ,[object Object],:
            time.sleep(,[object Object],); ,[object Object],
        state = store.load(run_id)
        ,[object Object],:
            state = advance_one_step(state)        ,[object Object],
            store.save(run_id, state)              ,[object Object],
            ,[object Object], state.done:
                queue.complete(run_id)
            ,[object Object],:
                queue.release(run_id)              ,[object Object],
        ,[object Object], Exception ,[object Object], e:
            queue.release(run_id, delay=backoff(state))

What this does: A worker claims a run with a lease (not a permanent pop), advances it by one step, saves the new state, and releases the run back to the queue for the next step. The run is a durable record that any worker can pick up; the worker holds it only briefly. Kill the worker mid-step and the lease expires, another worker claims the run, and it continues from the last checkpoint. Nothing is tied to a single process's lifetime.

⚡ Pro tip: Make each "step" small — one model call plus its tool calls, then checkpoint. The smaller the step, the less work you lose when a worker dies, and the more evenly work spreads across your worker pool. A long-running agent harness that does an hour of work between checkpoints loses an hour on every crash; one that checkpoints every step loses seconds.

What a Long Running Agent Harness Needs

Three decisions define a long-running harness, and they interact. A long running agent harness lives or dies on how these three settings relate to each other, so it's worth reasoning about them together rather than tuning each in isolation.

Step granularity. How much work happens between checkpoints. Finer steps mean less lost work and better load-balancing but more storage writes. One model turn per step is a good default — it's a natural boundary, and it's usually where the interesting state changes anyway.

Lease duration. How long a worker holds a run before the system assumes the worker is dead and reclaims it. Too short and slow-but-healthy steps get reclaimed mid-work; too long and a genuinely dead worker leaves a run stuck for that whole duration. Set it to a comfortable multiple of your typical step time.

Concurrency and fairness. With many long runs and finite workers, you need a policy for which run advances next. Naive FIFO lets one giant run monopolize a worker for hours; interleaving steps across runs keeps every run progressing. This matters more the longer your runs get.

Step-by-Step: Surviving a Dead Worker

The scenario that breaks naive designs is a worker that dies holding a run. Here's how the lease pattern handles it end to end.

Step one: Workers claim by lease, which records an expiry, not a permanent assignment.

[object Object], ,[object Object],(,[object Object],):
    ,[object Object], ,[object Object],.db.execute(,[object Object],, (lease_seconds, ,[object Object],.worker_id))

What this does: Atomically grabs one run whose lease has expired (or was never leased), sets a new lease, and returns it — using

SKIP LOCKED

leased_until < now()

Step two: A live worker renews its lease periodically during a long step.

[object Object], ,[object Object],(,[object Object],):
    ,[object Object],.db.execute(,[object Object],
                    ,[object Object],, (run_id, ,[object Object],.worker_id))

What this does: Extends the lease while the worker is genuinely still working, so a legitimately slow step doesn't get reclaimed out from under a healthy worker. The heartbeat is how the system tells "slow but alive" apart from "dead" — a worker that stops heartbeating loses its lease; one that keeps beating keeps its run.

⚡ Pro tip: Cap how many times a run can be reclaimed. A run that dies, gets reclaimed, and dies again at the same step isn't unlucky — it's hitting a deterministic bug, and infinite reclaiming just burns workers on a step that will never succeed. After a few reclaims at the same step, fail the run to a dead-letter state a human can inspect.

Pro-Level Variations

For runs that pause on human approval, don't hold a lease while waiting — that's a worker frozen for hours. Move the run to a

waiting

pending

For very long runs, add progress checkpoints the user can watch and a way to cancel cleanly. Cancellation on a long-running agent harness means setting a flag the worker checks between steps, not killing a process — a killed process might die mid-tool-call and leave state half-written.

For runs that must survive a full deploy, make sure a

SIGTERM

Troubleshooting Common Issues

Runs get stuck in

running

leased_until

The same step runs twice. A worker completed a step but died before marking it done, so another worker redid it. This is expected in an at-least-once system — the fix isn't to prevent it but to make each step idempotent, so redoing it is harmless.

Throughput collapses under load. Usually lease contention — every worker fighting for the same head-of-queue run.

SKIP LOCKED

⚠️ Common mistake: Running a long agent inside the request that triggered it, then "fixing" timeouts by raising the request timeout. A ten-minute request timeout is still a request, still tied to one process, still lost on a deploy or a crash. The fix isn't a longer leash — it's cutting the leash entirely by making the run durable and the worker disposable.

Watching Runs You Can't Watch in Real Time

A run that lasts hours creates a monitoring problem that short runs don't: nobody is sitting there watching it, so it can go wrong for an hour before anyone notices. A long running agent harness needs monitoring built around the run record, not around a live connection, because there's no live connection to hang a dashboard off.

The run record itself is the monitoring surface. Because every step updates a durable row — step count, last-updated time, tokens spent, status — you can query for trouble across your whole fleet without touching a single running process. A run whose

updated_at

running

[object Object], ,[object Object],(,[object Object],):
    ,[object Object], store.query(,[object Object],, (quiet_minutes,))

What this does: Finds runs that claim to be running but haven't checkpointed recently — the signature of a stall. Because the state is durable and queryable, this is a simple database query you can run on a schedule and alert on, rather than something you'd need to instrument inside every worker. The durability you added for resumability doubles as your monitoring backbone.

⚡ Pro tip: Cap total cost per run, not just steps and time. A run can stay under your step limit and still spend a fortune if each step processes huge contexts. Track cumulative tokens on the run record and fail runs that blow a spend ceiling — for a job that might run for hours, an unbounded cost is a much scarier failure than an unbounded duration.

Your Turn

Take your longest-running agent and make one change today: move its state into durable storage and have a worker advance it one step at a time instead of running it start-to-finish in one process. Even before you add leases and heartbeats, that single split makes the run survivable. It's the change that unlocks all the others — once the run is a durable record instead of a live process, leases, resumption, and monitoring all become natural extensions rather than rewrites.

Keep your run-state schema, lease conventions, and worker loop versioned alongside your prompts in a library like PromptABCD, so the durable-run architecture you built once becomes the default every long-running agent inherits, instead of each new one rediscovering why a four-hour job can't live in a thirty-second request.