Simulating Agent Teams Before Deployment

How do you know an agent team will hold up under real conditions before you actually ship it to real users? You can unit-test agents and still have no idea how the team behaves under a hundred concurrent requests, with a flaky dependency, and a mix of easy and hard tasks. Multi agent simulation answers that question cheaply - you run the team against synthetic-but-realistic conditions and watch what breaks, before a customer is the one who finds it. Here's a case study of a team that adopted simulation after a painful launch, and exactly how they built it.

The launch that motivated them went fine in a demo and fell over within an hour of real traffic. Nothing in their testing had exercised concurrency, sustained load, or the specific mix of task types real users sent. Simulation was how they made sure it never happened again.

The Problem This Team Faced

The team built an agent system for processing insurance claims - intake, classification, fraud-check, and decision agents working together. Their pre-launch testing was thorough at the unit level and nonexistent at the system level under load. They tested each agent on sample claims. They never tested the whole team on a realistic stream of claims arriving concurrently.

Production exposed everything the tests missed. Under concurrent load, the shared state between the classification and fraud-check agents developed race conditions. A burst of claims caused the fraud-check agent's rate-limited external API to throttle, and the team had no backpressure. And the real distribution of claims - mostly simple, occasionally very complex - created latency spikes on the complex ones that the uniform test claims never revealed. Each of these was findable before launch, but only by simulating realistic conditions, which they hadn't.

The Wrong Approach

Their first instinct after the bad launch was to just add more unit tests and a bigger sample of test claims.

[object Object],
,[object Object], ,[object Object],():
    ,[object Object], claim ,[object Object], load_sample_claims(,[object Object],):
        result = system.process(claim)   ,[object Object],
        ,[object Object], result.decision ,[object Object], (,[object Object],, ,[object Object],, ,[object Object],)

What this does: It runs 500 claims through the system and checks each produces a valid decision. It runs them sequentially, one at a time, so it never exercises concurrency, never creates the race conditions or throttling that concurrent load produces, and never reveals how the team behaves when many claims arrive at once. It's a bigger happy-path test, not a simulation.

The gap is that processing claims one at a time, no matter how many, tells you nothing about behavior under concurrent load - which is the only mode that matters in production. The bugs were all emergent properties of concurrency and load, and a sequential test, however large, has neither. They needed to reproduce production conditions, not just production volume.

⚡ Pro tip: Volume is not load. Running a thousand tasks sequentially exercises none of the concurrency, contention, or backpressure behavior that a hundred tasks arriving simultaneously does. If your pre-launch testing runs tasks one at a time, you have tested throughput of a single lane, not the traffic jam - and the traffic jam is what breaks agent teams.

The Correct Pattern

The fix was a proper multi agent simulation harness: generate a realistic stream of tasks with production-like timing and distribution, run them concurrently against the team, inject realistic failures, and measure how the system behaves.

[object Object], asyncio, random

,[object Object], ,[object Object], ,[object Object],(,[object Object],):
    results, start = [], time.time()
    ,[object Object], ,[object Object], ,[object Object],():
        task = sample_task(task_mix)   ,[object Object],
        t0 = time.time()
        r = ,[object Object], system.process(task)
        results.append({,[object Object],: time.time() - t0, ,[object Object],: r.status})
    tasks = []
    ,[object Object], time.time() - start < duration_s:
        ,[object Object],
        ,[object Object], asyncio.sleep(random.expovariate(arrival_rate))
        tasks.append(asyncio.create_task(fire()))
    ,[object Object], asyncio.gather(*tasks)
    ,[object Object], results

What this does: It fires tasks at the system concurrently with randomized, bursty arrival timing and a realistic mix of task types - reproducing how real traffic actually arrives rather than a tidy sequential stream. The collected latencies and statuses reveal the concurrency bugs, throttling, and tail-latency spikes that sequential testing structurally cannot surface.

The two details that made it realistic were the arrival pattern and the task mix. Real traffic arrives in bursts, not at a steady rate, so they used exponentially-distributed inter-arrival times that produce natural bursts. And real claims follow a skewed distribution - mostly simple, a long tail of complex - so they sampled task types from the real distribution instead of a uniform one. Both details matter because the bugs lived specifically in the bursts and the complex tail.

[object Object],
TASK_MIX = {,[object Object],: ,[object Object],, ,[object Object],: ,[object Object],, ,[object Object],: ,[object Object],}
,[object Object],
FAILURE_INJECTION = {,[object Object],: ,[object Object],, ,[object Object],: ,[object Object],}

What this does: It defines a task distribution matching real production data and a failure-injection rate matching real dependency behavior. Simulating with the real skew and real failure rates is what surfaces the tail-latency and backpressure problems - a uniform distribution with no failures would run clean and teach you nothing about production.

Results and What Changed

The simulation reproduced all three production failures in a safe environment within its first serious run. The concurrency race in shared state showed up as inconsistent classifications under load. The API throttling showed up as a cascade of failures when the simulated throttle fired. And the tail-latency spikes on complex tasks showed up clearly in the latency distribution. All three were now findable and fixable before deployment, on a laptop, for the cost of some tokens.

The larger shift was cultural: simulation became a gate. No agent team shipped without passing a simulation run that reproduced realistic load, mix, and failures. That gate turned "we think it'll be fine" into "we watched it handle worse than production and hold up," which is a completely different level of confidence and one you can only buy with simulation.

⚡ Pro tip: Simulate worse than production, not just equal to it. Run at higher arrival rates, higher failure-injection rates, and a heavier complex-task tail than you expect. A system that survives conditions worse than reality has real headroom; a system tuned to exactly expected conditions has none, and production always eventually exceeds your expectations. Deliberate overload in simulation is cheap insurance.

What Do You Measure in a Simulation Run?

A simulation that produces a pass/fail is wasting most of its value. The point is the distribution of what happened, because agent teams fail at the tail, not the average. So you measure percentiles, not means, and you watch how they change as load rises.

The core metrics are latency percentiles (p50, p95, p99), throughput (tasks completed per second), error rate under load, and - specific to agents - how gracefully the system degrades when the failure injection fires. A healthy system shows latency rising smoothly with load and recovering after a burst; an unhealthy one shows a cliff, where latency stays flat until some threshold and then explodes as a queue backs up irrecoverably.

[object Object], ,[object Object],(,[object Object],):
    lat = ,[object Object],(r[,[object Object],] ,[object Object], r ,[object Object], results)
    n = ,[object Object],(lat)
    ,[object Object], {
        ,[object Object],: lat[n // ,[object Object],],
        ,[object Object],: lat[,[object Object],(n * ,[object Object],)],
        ,[object Object],: lat[,[object Object],(n * ,[object Object],)],
        ,[object Object],: ,[object Object],(r[,[object Object],] != ,[object Object], ,[object Object], r ,[object Object], results) / n,
    }

What this does: It reduces a simulation run to the numbers that actually predict production behavior - the latency tail and the error rate - rather than an average that hides both. The p99 is the number to watch: it's what your unluckiest users experience, and it's where agent teams show stress long before the average moves.

The most valuable thing a simulation reveals is the shape of degradation as you push load up. Run the simulation at several load levels and plot p99 against arrival rate. A gentle upward slope means headroom; a knee where p99 suddenly rockets means you've found the capacity limit, and now you know it before production does. That knee is the single most actionable output of the whole exercise, because it tells you exactly when to add capacity.

⚡ Pro tip: Run the simulation at increasing load levels and find the "knee" - the point where p99 latency stops rising gently and shoots up. That knee is your real capacity limit, and knowing it lets you set autoscaling thresholds and alerts below it rather than discovering it live when a traffic spike drives you past it. Finding the knee in simulation is far cheaper than finding it in production.

How to Apply This to Your Situation

Start by capturing your real (or best-estimated) task distribution and arrival pattern. The realism of a simulation is entirely determined by these inputs - a simulation with a uniform distribution and steady arrivals tests a system that doesn't exist. Get the mix and the burstiness right first.

Then build the concurrent harness, add failure injection matching your dependencies' real failure modes, and run at and above expected load. Watch the latency distribution and the failure behavior, not just the average - the average hides the tail and the bursts, which is exactly where the bugs are.

⚠️ Common mistake: Simulating with a uniform task distribution and steady arrival rate because it's easier to write. This tests a system under conditions that never occur in production and gives false confidence, because the bugs live specifically in the skewed distribution and the bursty arrivals that a uniform, steady simulation smooths away. The realism is the entire value - an unrealistic multi agent simulation is worse than none, because it tells you you're safe when you aren't.

Next Steps

Capture your task mix and arrival pattern, build a concurrent harness that reproduces them, inject realistic failures, and make a passing simulation run a gate before any deployment.

Keep your simulation scenarios and synthetic task generators versioned so you can re-run the same stress conditions on every release. I store the scenario definitions and task-generation prompts in PromptABCD, because a well-tuned set of realistic scenarios - the right mix, the right bursts, the right failures - is genuinely valuable and hard to get right, and reusing proven scenarios across releases means each new version faces the same tough, realistic gauntlet instead of an ad-hoc test that varies every time.

⚡ Pro tip: Refresh your simulation's task mix from real production data periodically, because the mix drifts. The distribution of tasks your system saw at launch is not the distribution it sees six months later as usage patterns change, and a simulation calibrated to stale data slowly stops reflecting reality. Re-sampling the mix from recent production traffic keeps the gauntlet honest and catches the new failure modes that shifted usage introduces.