How to Test AI Agents Before Production

Picture this: you're a QA lead at a fintech startup, launch is Friday, and the agent passed every demo you threw at it. So why does your stomach hurt? Because you've tested software before, and you know a green demo means almost nothing when the same input can pass twice and fail the third time. Agents aren't deterministic, and testing AI agents like they are is how confident teams ship quiet disasters.

This guide is hands-on. You'll get a harness you can copy today, the few variables that decide whether your tests mean anything, and the failure categories that never show up in a happy-path demo.

Quick-Start (Copy This Right Now)

[object Object], statistics

,[object Object], ,[object Object],(,[object Object],):
    results = []
    ,[object Object], c ,[object Object], cases:
        outcomes = [check(agent.run(c[,[object Object],]), c) ,[object Object], _ ,[object Object], ,[object Object],(runs)]
        pass_rate = ,[object Object],(outcomes) / runs
        results.append({,[object Object],: c[,[object Object],], ,[object Object],: pass_rate})
    flaky = [r ,[object Object], r ,[object Object], results ,[object Object], ,[object Object], < r[,[object Object],] < ,[object Object],]
    ,[object Object], {
        ,[object Object],: statistics.mean(r[,[object Object],] ,[object Object], r ,[object Object], results),
        ,[object Object],: flaky,     ,[object Object],
    }

What this does: it runs every test case several times instead of once and reports which cases are flaky — passing sometimes and failing others — which is the failure mode a single run hides completely.

Run this against your cases before anything else. The

flaky_cases

Understanding the Variables

Three things decide whether your suite tells the truth.

Runs-per-case is the one people skip and shouldn't. Running each case once gives you a number that feels like a result but is really a single sample from a distribution. Five runs starts to show you the spread; ten shows it clearly. The cost is more tokens, and it's worth every one, because the alternative is discovering the variance in production.

The check function is where most suites quietly cheat. Exact string matching is too strict for open-ended output and too loose for structured output. For factual answers, check for the required facts, not the exact wording. For structured actions, check the fields. For tone or quality, you'll need a model-based judge — calibrated against human labels first, or it just launders your assumptions.

The case set is your coverage. A suite of twenty happy-path questions tells you the agent works when everything goes right, which you already knew from the demo. The cases that matter are the ugly ones.

⚡ Pro tip: Seed your case set from real user logs the moment you have any. Ten real, messy questions teach you more than a hundred you imagined, because users phrase things in ways you never would and reach for the agent in situations you never designed for.

Step-by-Step: Testing AI Agents Before Production

Here's the order that catches the most, fastest.

First, build a golden set — inputs paired with what a correct outcome looks like. Keep it small and real. Twenty to fifty well-chosen cases beat a thousand generated ones, because you'll actually maintain fifty and you'll trust every one.

Second, add adversarial cases on purpose. Malformed inputs, contradictory requests, attempts to push the agent off-task, questions just outside its scope. These are the cases production will send you whether you test them or not.

adversarial = [
    {,[object Object],: ,[object Object],, ,[object Object],: ,[object Object],},
    {,[object Object],: ,[object Object],, ,[object Object],: ,[object Object],},
    {,[object Object],: ,[object Object],, ,[object Object],: ,[object Object],},
    {,[object Object],: ,[object Object],, ,[object Object],: ,[object Object],},
]

What this does: it defines a small set of deliberately hostile inputs so you can confirm the agent degrades gracefully — refusing, clarifying, or staying in scope — instead of doing something unsafe when a real user sends the same thing.

Third, test the trajectory, not just the answer. Assert on how many tool calls the agent made, whether it stayed in scope, and whether it recovered cleanly from a failed step. An agent that reaches the right answer after four redundant API calls is failing a test you haven't written yet.

Fourth, gate the pipeline. Wire the suite into CI so a change that drops the average pass rate or introduces a flaky case can't merge. A test suite nobody runs is a document; a test suite that blocks a bad merge is protection.

⚡ Pro tip: Pin your model version in tests. When a provider updates the model underneath you, your agent's behavior can shift without a single line of your code changing. If your suite doesn't pin the version, you can't tell a regression you caused from one that arrived on its own overnight.

Pro-Level Variations

Different teams stretch the harness in useful directions.

A healthcare team building an intake agent runs every case through a required-fields checker, treating any missing field as a hard failure regardless of how good the rest of the response reads. Completeness is their quality bar, so that's what the suite enforces.

An e-commerce team testing a recommendation agent runs a bias sweep — the same request across many product categories and customer profiles — to catch cases where the agent quietly favors one brand or mishandles one segment. A single happy-path test would never surface it.

A developer-tools company testing a code agent runs the agent's output through an actual sandbox and executes it. The test isn't "does the code look right," it's "does the code run and pass its own tests," which is the only bar that matters for generated code.

⚡ Pro tip: Track flakiness as its own metric over time, separate from pass rate. A suite whose average holds steady at 95% while its flaky-case count creeps up is a suite quietly rotting. Rising variance is an early warning that something upstream — a prompt, a tool, the model — is getting less stable.

Troubleshooting Common Issues

When every case passes but production still breaks, your case set doesn't match reality. Pull last week's failures and turn each into a permanent case. Your suite should grow every time production surprises you.

When results swing wildly between runs, you've found real non-determinism, not a broken test. Lower the temperature for tasks that should be deterministic, tighten the prompt, or add a validation step — but first confirm the variance is the agent's and not your check function being inconsistent.

When the suite is too slow to run often, split it. A fast smoke set of ten cases runs on every commit; the full set runs nightly. A suite that's too slow to run is a suite that stops running, and an unrun suite protects nothing.

⚠️ Common mistake: Testing AI agents once and calling them tested. Agents drift as models update, prompts change, and tools evolve underneath them. Testing is not a launch gate you pass once — it's a standing process that runs for the life of the agent, or it runs never.

What Makes Testing AI Agents Different From Unit Tests

If you come from traditional software, the instinct is to write assertions like

assert output == expected

A unit test checks a deterministic function: same input, same output, forever. An agent is a distribution — the same input produces a range of outputs, most good, some not. So the meaningful question shifts from "did it pass?" to "how often does it pass, and how bad are the misses?" A test that's green once tells you the agent can produce a good answer, not that it reliably does. That's exactly why the quick-start harness runs each case multiple times: you're sampling a distribution, not checking a value.

The second difference is that correctness is often a range, not a point. Two different summaries can both be right; two different tool-call orders can both reach the correct result. Assertions that demand one exact answer will fail perfectly good runs and train you to ignore your own suite. You have to assert on properties — the required facts are present, the action stayed in scope, no forbidden field leaked — rather than on an exact string.

The third difference is that your system under test can change without your code changing. A model update, a shifted provider default, a tweaked system prompt three layers up — any of these can move behavior while your diff shows nothing. Traditional tests assume the code is the only variable; agent tests can't.

⚡ Pro tip: Write assertions as properties, not exact matches, from day one. "Response contains the correct balance and no other account's data" survives legitimate variation in phrasing; "response equals this exact paragraph" fails the moment the model rewords a sentence. Property-based checks are the difference between a suite you trust and one you learn to mute.

Your Turn

Start with the quick-start harness and five real cases run five times each. That single change — from one run to five — will show you flakiness you didn't know you had, and flakiness is what production feels as unreliability.

As your golden set and adversarial cases accumulate, keep the test prompts and judge criteria somewhere the whole team can reach. Groups that store their evaluation and test prompts in a shared library like PromptABCD stop rewriting the same checks for every agent and start compounding their coverage. A good test suite is the most reusable thing you'll build — every case you add protects every agent that comes after, and every real failure you fold back in makes the whole fleet a little harder to break.