How to Run Repeatable Agent Tests Without the Flakes

A team I worked with had a continuous-integration pipeline that was red almost half the time — and not because their code was broken. Their agent tests were flaky: the same test would pass, then fail, then pass again, with no code change in between. Their developers had learned to just hit "re-run" until it went green, which meant the tests had stopped catching anything real. Repeatable agent testing is the discipline that fixes this, and it comes down to controlling the three things that make agents nondeterministic. Here's the case study of how that team went from 47% flaky to reliably green.

The Problem the Team Faced

The team built an agent that automated parts of their customer onboarding. They'd written a reasonable-looking test suite: give the agent a scenario, check that it did the right thing. The tests worked when they wrote them. Then they started failing at random.

The failures had a pattern, once someone looked closely. A test asserting the agent's exact output text would fail because the model phrased its answer slightly differently on this run. A test that depended on a live tool would fail because the tool returned different data than last time. A test involving "today's date" would fail whenever it ran across midnight. None of these were bugs in the agent. They were the test suite assuming determinism from a system that had none.

The lead engineer described the cost: "We can't tell a real failure from a flake anymore, so we've stopped trusting the whole suite." That's the endgame of flaky tests — not that they fail, but that they train the team to ignore failure. A test suite you re-run until it passes is theater, and expensive theater at that.

What Was Actually Going Wrong

Agent nondeterminism comes from exactly three sources, and this team's flakes traced to all three.

Model sampling. Language models sample from a probability distribution. Run the same prompt twice and you can get different tokens, different phrasing, sometimes different tool choices. Any test asserting exact output text is at the mercy of this.

Tool responses. If a test lets the agent call a live tool — a real API, a real database — the response can change between runs. Yesterday the query returned four rows; today it returns five. The agent behaved correctly both times; the test assumed a fixed answer.

Time and randomness. Anything the agent does that touches the current time, a random seed, or a generated ID varies by definition. A test that hardcodes an expected timestamp fails the moment the clock moves.

Repeatable agent testing means pinning all three. Miss any one and the flakes continue, which is why the team's earlier partial fixes hadn't stuck — they'd frozen the model temperature but left live tools in the loop.

It's worth naming why these three and not more. Everything else an agent does is deterministic given those three inputs: the same prompt, the same tool outputs, and the same clock produce the same logic every time, because the agent's own code is ordinary code. Nondeterminism only enters through the model's sampling, the outside world the tools reach, and the passage of time. Control those three doorways and the entire run becomes reproducible — which is why the fix is a checklist of exactly three items, not an open-ended hunt.

⚡ Pro tip: When a test is flaky, don't retry it — quarantine it and find which of the three axes it left unpinned. A retry hides the flake; it doesn't fix it, and every hidden flake erodes trust in the suite a little more. The flake is information: it's telling you exactly which source of nondeterminism you forgot to control.

The Fix: Freeze All Three Axes

The team rebuilt their tests to control each source explicitly.

[object Object], ,[object Object],(,[object Object],):
    ,[object Object], Agent(
        model=Model(temperature=,[object Object],, seed=,[object Object],),          ,[object Object],
        tools=FrozenTools(recorded_tool_responses),    ,[object Object],
        clock=FixedClock(frozen_time),                 ,[object Object],
    )

What this does: it builds an agent with temperature and seed fixed so sampling is stable, tools that return recorded responses instead of hitting live systems, and a clock frozen to a known instant. With all three pinned, the same test produces the same run every time — which is the entire point of a test.

But pinning sampling isn't quite enough, and this is the part the team learned the hard way. Even at temperature zero, model output isn't guaranteed identical across model versions or hardware. So they stopped asserting on exact output strings and started asserting on invariants:

[object Object], ,[object Object],():
    agent = make_test_agent(FIXTURES[,[object Object],])
    result = agent.run(,[object Object],)
    ,[object Object],
    ,[object Object], ,[object Object], ,[object Object], result.tools_called
    ,[object Object], result.tools_called.index(,[object Object],) < \
           result.tools_called.index(,[object Object],)
    ,[object Object], result.state[,[object Object],] ,[object Object], ,[object Object],

What this does: instead of checking that the agent said a specific sentence, it checks that the agent called

create_account

⚡ Pro tip: Assert on behavior, not prose. "Did it call the right tools in the right order and reach the right state" is stable across model versions; "did it produce this exact string" is a flake generator. The best agent tests read like a description of correct behavior, not a transcript of one particular run.

Results and What Changed

The flaky-test rate went from 47% to near zero. Not because the agent got more deterministic — models still sample — but because the tests stopped depending on the parts that vary. Green meant working; red meant broken. The team started trusting the suite again, which meant it started catching real regressions again.

The deeper win was speed. Because the tests no longer hit live tools, they ran in seconds instead of minutes, and they ran offline with no API cost. A suite that's fast, free, and deterministic gets run constantly; a slow, flaky, expensive one gets skipped. Repeatable agent testing didn't just fix reliability — it made the tests cheap enough to actually use on every commit, which is the only place a test suite earns its keep.

⚡ Pro tip: Keep one small suite of non-frozen "live" tests that do hit real tools, run on a schedule rather than on every commit. Frozen tests prove your agent's logic is correct; a handful of live tests prove your tool integrations still work against the real world. You need both, but only the frozen ones belong in the fast feedback loop.

How to Apply This to Your Situation

The three-axis freeze generalizes across domains:

• A fintech engineer freezes market-data tool responses and the clock so tests of a trading agent are reproducible, then runs a separate nightly live suite to confirm the real data feeds still work.
• A healthcare-software developer pins tool responses to recorded patient-record fixtures so onboarding-agent tests never touch real records and never flake on live data changes.
• A logistics platform team freezes routing-API responses and asserts on the sequence of decisions the agent made, catching a regression where the agent started skipping a required validation step.

In each case the move is identical: pin sampling, pin tools, pin time, and assert on invariants rather than exact text.

⚠️ Common mistake: Pinning two of the three axes and assuming you're done. This is the trap the team fell into first — they set temperature to zero, saw some flakes disappear, and declared victory while live tools and the system clock kept generating the rest. Nondeterminism is an AND, not an OR: a test is only repeatable when all three sources are controlled. One unpinned axis is enough to keep a test flaky, and a "mostly repeatable" test is just a flaky test that fails less often — which is arguably worse, because it lulls you into trusting it right up until it fails in front of a customer.

There's a subtler version of the same mistake: pinning the axes but then writing an assertion that secretly depends on an unpinned detail. A test that freezes tool responses but asserts on a generated request ID, or that pins the clock but checks a duration measured in real wall-clock time, has smuggled nondeterminism back in through the assertion. When you freeze the three axes, audit your assertions too — make sure nothing you're checking varies for a reason you didn't control.

Next Steps

Look at your own flakiest agent test today and ask which of the three axes it leaves unpinned — it's almost always at least one. Freeze it, switch its assertions from output strings to behavioral invariants, and watch the flake disappear. Then do the same for the next one. Within a week or two the whole suite flips from something people re-run until it's green to something they believe the first time.

Repeatable agent testing turns a suite you've learned to ignore into one you can trust, and trust is the only thing that makes a test suite worth having. The fixtures, frozen scenarios, and invariant checks you build are reusable across every agent version. Keeping the test scenarios and their expected-behavior descriptions organized in a library like PromptABCD — tagged by the capability each one guards — means your test suite survives rewrites intact, and the hard lesson about asserting on behavior instead of prose doesn't have to be relearned one flaky pipeline at a time.