Deterministic Replay for Harness Debugging

A team spent three days trying to reproduce a bug where their agent occasionally deleted the wrong records. They couldn't. Every time they re-ran the failing input, the agent did something different — sometimes right, sometimes wrong, never the same twice — because the model samples differently each run and the tools returned different live data. The bug was real and completely unreproducible, which meant it was undebuggable. Harness deterministic replay is the technique that solves exactly this: record everything non-deterministic about a run, then replay it byte-for-byte to debug the failure as many times as you need. This guide shows you how to build it.

The core problem is that an agent run has several sources of non-determinism — the model's sampled output, live tool results, timestamps, random values — and any one of them makes a run irreproducible. Deterministic replay records the outputs of all these sources during a real run, then feeds those exact recorded values back during replay, so the run unfolds identically every time. The failure that happened once, unpredictably, becomes a failure you can summon on demand — which is the precondition for actually fixing it.

Quick-Start (Copy This Right Now)

The heart of it is a recorder that captures every non-deterministic result during a live run, and a player that returns those exact results during replay.

[object Object], ,[object Object],:
    ,[object Object], ,[object Object],(,[object Object],):
        ,[object Object],.events = []

    ,[object Object], ,[object Object],(,[object Object],):
        result = fn(*args)                      ,[object Object],
        ,[object Object],.events.append({,[object Object],: kind, ,[object Object],: serialize(result)})
        ,[object Object], result

,[object Object], ,[object Object],:
    ,[object Object], ,[object Object],(,[object Object],):
        ,[object Object],.events, ,[object Object],.i = events, ,[object Object],

    ,[object Object], ,[object Object],(,[object Object],):
        event = ,[object Object],.events[,[object Object],.i]; ,[object Object],.i += ,[object Object],
        ,[object Object], event[,[object Object],] == kind, ,[object Object],
        ,[object Object], deserialize(event[,[object Object],])     ,[object Object],

What this does: During a live run the

Recorder

Player

assert

⚡ Pro tip: Record at the boundary of every non-deterministic call — model, tools, clock, randomness — not just the model. Teams remember to record model outputs and forget that

time.now()

What Harness Deterministic Replay Must Capture

Faithful harness deterministic replay depends on capturing every source of variation, so it's worth enumerating them.

Model outputs. The big one. Record the full model response for each call — content, tool calls, everything — so replay reproduces the exact reasoning path.

Tool results. Live tools return different data over time; a search today isn't a search last week. Record each tool result so replay sees the world as it was during the original run, not as it is now.

Time and randomness. Timestamps, UUIDs, random samples — anything from the clock or a random source varies per run. Route these through the recorder too, or they'll drift on replay and push the run down a different path.

External state reads. Anything the run reads from a changing external source is non-deterministic from the run's perspective and belongs in the recording.

Step-by-Step: Wiring Replay Into a Run

Step one: Route every non-deterministic call through the recorder/player interface, so switching between record and replay is one flag.

[object Object], ,[object Object],(,[object Object],):
    io = Player(tape) ,[object Object], tape ,[object Object], Recorder()
    state = State(prompt)
    ,[object Object], ,[object Object], state.done:
        resp = io.wrap(,[object Object],, model.call, state.messages)
        ,[object Object], call ,[object Object], resp.tool_calls:
            result = io.wrap(,[object Object],, dispatch, call)
            state.add(result)
    ,[object Object], state, io

What this does: Uses a

Player

Recorder

wrap

Step two: Persist the tape with every run, or at least with every failure, so you can replay after the fact.

[object Object], ,[object Object],(,[object Object],):
    ,[object Object], ,[object Object],(io, Recorder):
        tape_store.save(state.run_id, io.events)   ,[object Object],

What this does: Saves the recorded events for a completed run so it can be replayed later. Keeping tapes for failed runs turns every production failure into a reproducible test case — the three-day unreproducible bug from the intro becomes a tape you replay in seconds.

Step three: To debug, load the failing run's tape and replay it under a debugger, stepping through the exact sequence that failed.

⚡ Pro tip: Store the tape of any run that hits an error automatically. The whole point is to reproduce failures, and failures are exactly the runs you didn't know to record in advance. Recording every run is ideal if you can afford the storage; recording every failure is the minimum that makes replay pay off.

Pro-Level Variations

Turn tapes into regression tests. A recorded run of a scenario you care about becomes a fixture: replay it in CI and assert the agent still behaves the same. When a prompt or code change alters behavior, the replay diverges and the test fails — catching behavioral regressions that ordinary tests miss.

Use partial replay to test fixes. Replay a failing run up to the point of the bug, then let it run live from there with your fix in place, to confirm the fix changes the outcome. You're reproducing the exact conditions that triggered the failure, then testing the patch against them.

Redact secrets in tapes. Tapes contain everything a run saw, including tool results that might hold sensitive data, so run the same scrubber over tapes that you use for logs before persisting them.

Troubleshooting Common Issues

Replay diverges immediately. Usually an un-recorded non-deterministic source — a clock or random call that isn't going through the recorder. Find the first

wrap

Tapes are huge. Full model responses and tool results add up. Compress tapes, and for high-volume systems keep full tapes only for failures while keeping lightweight metadata for the rest.

Replay works but the bug doesn't reproduce. The bug may depend on a source you didn't record — concurrent state, an external system's behavior. Widen what the recorder captures until the failure reproduces reliably.

⚠️ Common mistake: Re-calling the model during replay instead of returning the recorded response. If replay actually calls the model again, it samples fresh output and you're not replaying — you're running a new run that happens to start the same. Replay must return the recorded result and never touch the real model, or it isn't deterministic at all.

Designing Tapes You Can Trust Later

A tape is only useful if you can still replay it weeks later, after the code has changed — and that's where naive record-replay quietly breaks. If harness deterministic replay matches recorded events to calls purely by order, then any code change that adds, removes, or reorders a non-deterministic call makes every old tape diverge. The tape that captured last week's failure won't replay against this week's code, which is exactly when you need it most.

The fix is to match events by a stable key, not by position. Give each recorded event a key derived from what it represents — the tool name plus a hash of its arguments, the model call plus the message count — so replay looks up "the result for this call" rather than "the next result in the list." Order-independent matching survives the code changes that order-dependent matching can't.

[object Object], ,[object Object],(,[object Object],):
    ,[object Object], ,[object Object],

,[object Object], ,[object Object],:
    ,[object Object], ,[object Object],(,[object Object],):
        ,[object Object],.by_key = {e[,[object Object],]: e[,[object Object],] ,[object Object], e ,[object Object], events}
    ,[object Object], ,[object Object],(,[object Object],):
        k = key_for(kind, args)
        ,[object Object], k ,[object Object], ,[object Object], ,[object Object],.by_key:
            ,[object Object], fn(*args)            ,[object Object],
        ,[object Object], deserialize(,[object Object],.by_key[k])

What this does: Keys each recorded result by the kind and a hash of its arguments, then during replay looks up results by that key instead of by position. A call the tape has seen replays from the recording; a genuinely new call (because the code changed) falls through to live execution instead of crashing. This is what lets an old tape replay against evolved code — the unchanged calls replay, the changed ones run.

The trade-off is honesty about what changed: falling through to a live call means that part of the run isn't truly reproduced. That's usually fine for debugging a specific failure, but for regression tests you want strict matching that fails loudly on divergence — so make the matching mode a choice. Debug loosely; test strictly.

⚠️ Common mistake: Keeping tapes forever without versioning the tape format. The day you change what a tape records, every old tape becomes unreadable by the new player unless you version the format and handle old versions. Stamp each tape with a format version, exactly as you version persisted state, so a tape recorded today still replays after the recorder evolves.

Your Turn

Add a recorder to your agent that captures model and tool results, and save the tape whenever a run errors. The next time a run fails mysteriously, you'll have a tape you can replay under a debugger as many times as you need — turning "we can't reproduce it" into "let's step through exactly what happened." That shift, from unreproducible to reproducible, is often the entire difference between a bug you can fix and one you can only pray about.

Keep your recorder, tape format, and replay harness versioned alongside your prompts in a library like PromptABCD, so every agent you build is debuggable by replay from day one — and the three-day-unreproducible-bug story stays a story about someone else's agent.