Recording and Replaying Agent Sessions for Debugging

How do you reproduce an agent bug that only happened once, in production, three days ago? You can't just re-run it — the model samples differently, the tools return different data, and the exact conditions are gone. This is the question that pushes teams toward an agent session replay harness: record every input and output of a run once, then replay it deterministically as many times as you need, offline and for free. Let's tear down the "just re-run it" approach and rebuild around record-and-replay.

Before: Reproducing Bugs by Re-Running

Here's how most teams try to reproduce an agent bug:

[object Object], ,[object Object],(,[object Object],):
    ,[object Object],
    ,[object Object], attempt ,[object Object], ,[object Object],(,[object Object],):
        result = live_agent.run(task)
        ,[object Object], result.failed:
            ,[object Object], result      ,[object Object],
    ,[object Object], ,[object Object],

What this does: it re-runs the live agent over and over, hoping to trigger the same failure. It hits real tools, spends real tokens, and depends on the model happening to sample the same problematic path again. For an intermittent bug, this can take dozens of expensive attempts — and often never reproduces at all, because the exact tool responses that triggered it are gone. You end up debugging from logs alone, which is the hard way.

Why That Approach Fails

Re-running a live agent to reproduce a bug fights against everything that makes agents nondeterministic.

The model won't necessarily repeat the choice that caused the failure. If the bug came from a specific unlucky sampling path, you might run it fifty times without hitting it again. You're gambling that randomness repeats, which is not how randomness works. And even when you do reproduce it, you can't be sure it's the same failure — a superficially similar error might have a different cause, sending you down the wrong debugging path entirely.

The tools won't return the same data. The failure might have been triggered by a specific API response — a particular error, a specific edge-case value — that the live tool no longer returns. Re-running queries the current state of the world, not the state that caused the bug. The one input you most need to reproduce the failure is the one input you can no longer get.

And it's slow and expensive. Every reproduction attempt is a full live run with real API costs and real latency. Chasing an intermittent bug this way can burn hours and real money, with no guarantee of success. I've watched a team spend most of a day and never reproduce a failure they'd clearly seen in their logs — because the conditions were simply gone.

⚠️ Common mistake: Treating an agent bug like a deterministic-code bug you can reproduce on demand. A traditional bug reproduces when you re-run the code; an agent bug lived in a specific combination of model sampling and tool responses that re-running doesn't recreate. Without a recording of that exact combination, you're not reproducing the bug — you're hoping to stumble into it again.

After: The Agent Session Replay Harness

The fix is to record every interaction during the original run, then replay those exact recordings. Record once:

[object Object], ,[object Object],:
    ,[object Object], ,[object Object],(,[object Object],):
        ,[object Object],.events = []

    ,[object Object], ,[object Object],(,[object Object],):
        ,[object Object], ,[object Object],(,[object Object],):
            reply = model.complete(messages, **kw)
            ,[object Object],.events.append({,[object Object],: ,[object Object],, ,[object Object],: messages, ,[object Object],: reply})
            ,[object Object], reply
        ,[object Object], recorded

    ,[object Object], ,[object Object],(,[object Object],):
        ,[object Object], ,[object Object],(,[object Object],):
            out = fn(**kw)
            ,[object Object],.events.append({,[object Object],: ,[object Object],, ,[object Object],: name, ,[object Object],: kw, ,[object Object],: out})
            ,[object Object], out
        ,[object Object], recorded

What this does: it wraps the model and every tool so that each call's inputs and outputs are captured to an event log during a live run. The result is a complete recording — a "cassette" — of exactly what the model saw and what every tool returned. This runs in production with negligible overhead and gives you the raw material for perfect reproduction.

Then replay reads from the recording instead of calling anything live:

[object Object], ,[object Object],:
    ,[object Object], ,[object Object],(,[object Object],):
        ,[object Object],.events = ,[object Object],(events)

    ,[object Object], ,[object Object],(,[object Object],):
        ,[object Object],
        ,[object Object], i, e ,[object Object], ,[object Object],(,[object Object],.events):
            ,[object Object], e[,[object Object],] == kind ,[object Object], e[,[object Object],] == request:
                ,[object Object], ,[object Object],.events.pop(i)[,[object Object],]
        ,[object Object], LookupError(,[object Object],)

What this does: during replay, every model call and tool call is answered from the recording rather than executed live, matched by the request content. The agent runs through the exact same sequence of responses that produced the original bug — deterministically, offline, in milliseconds, at zero cost. The bug reproduces every single time because you're feeding it the precise conditions that caused it. What was a game of chance becomes a certainty.

Breaking Down the Replay Harness

Three design choices make a replay harness reliable, and getting them right is what separates a toy from a tool you'll depend on for years.

Match on request content, not call order. If your replayer matches purely by sequence — first call gets first recording — it breaks the moment the agent's path changes slightly. Matching on the hashed content of each request makes replay resilient to reordering and lets you replay partial or modified runs.

Record the rendered prompt, not just the user input. The model's behavior depends on the fully assembled prompt — system instructions, tool schemas, history. Record that, because a bug often lives in the assembled prompt rather than the visible input.

Support "what-if" replay. The real payoff beyond reproduction is testing fixes. Swap one recorded tool response for a different value and replay — now you can see how the agent would have behaved if the tool had returned something else, without any live run. This turns a recording into an experiment.

A fourth choice determines how much your recordings are worth over time: record at a stable boundary. If you record the raw HTTP responses of your tools, your cassettes break every time a tool's wire format shifts. If you record at the tool function boundary — the arguments in, the structured result out — your recordings survive changes to how the tool fetches its data, because they capture the contract the agent actually depends on. The closer you record to the agent's own interface, the longer your recordings stay valid, and the less often you have to re-capture a library of sessions you spent months building.

This stability matters because a replay harness compounds in value. The first recording reproduces one bug. A year of recordings becomes a regression suite covering every failure you've ever seen, an A/B testing corpus of real traffic, and an onboarding tool that shows new engineers exactly how the agent behaves in tricky situations. But only if the recordings from a year ago still replay — which is entirely a function of choosing a boundary that doesn't churn.

⚡ Pro tip: Turn every reproduced production incident into a permanent replay test. Once you've recorded the session that caused a bug, that recording becomes a regression test that runs forever, offline and free, guaranteeing the bug can't silently return. Your recordings library slowly becomes a museum of every failure you've ever fixed — and a wall against every one of them recurring. Each cassette you add makes the wall a little higher.

Variations for Different Contexts

The replay pattern adapts to what a team needs most:

• A platform engineer records all production agent sessions and retains the ones that failed, building a replay-based regression suite from real incidents instead of imagined ones.
• A research team uses what-if replay to A/B test prompt changes against recorded sessions, measuring how a new system prompt would have handled last month's real traffic without spending a token on live runs.
• A support-automation lead replays a customer's exact failed session to a developer's laptop, so the bug that happened to one user at 2 a.m. can be debugged offline at 10 a.m. with the precise conditions intact.

Same recording mechanism, three different payoffs — reproduction, experimentation, and offline debugging.

One more use worth calling out: replay makes agent behavior teachable. A recorded session of the agent handling a genuinely hard case — a gnarly multi-step recovery, a subtle tool-ordering decision — is a better onboarding artifact than any document, because a new engineer can step through exactly what the model saw and what it did at each turn. The recording is the agent explaining itself, frozen in a form you can inspect at leisure instead of racing past in a live run.

⚡ Pro tip: Store recordings with the agent version and prompt version that produced them. When you replay an old cassette against new agent code, you want to know whether a divergence means your fix worked or your recording is stale. The version tags turn "this replays differently now" from a mystery into a clear answer about what changed between then and now.

⚡ Pro tip: Redact sensitive data at record time, not replay time. Recordings capture whatever the tools returned, which may include personal data or secrets. Scrub those fields as you record so the cassette is safe to store and share — a recording full of real customer data is a liability, and cleaning it after the fact means it existed unprotected in between.

Save and Reuse This

An agent session replay harness converts the impossible — reproducing a one-time, three-days-ago bug — into the routine. Record every model and tool interaction once, match replays on request content, and you get deterministic, free, offline reproduction plus the ability to test fixes against real recorded conditions. The "just re-run it" approach can't compete, because it's gambling on randomness repeating, and randomness doesn't take requests.

The recordings you capture and the what-if scenarios you build from them are lasting assets — a growing library of exactly how your agent behaved in every situation that mattered. Keeping the scenarios and the prompts that drove them organized in a library like PromptABCD means your regression suite and your experiments stay reusable across agent versions. Record the failure once, and it never surprises you again.