Concurrency in an Agent Harness: A Case Study

Most advice about running agents concurrently focuses on the wrong problem. It tells you how to spin up parallel workers and bound them with a semaphore — the easy part — while barely mentioning the failure that actually bites: shared mutable state leaking between concurrent runs. Agent harness concurrency goes wrong not because parallelism is hard to start, but because a harness that worked perfectly for one agent at a time quietly corrupts itself when ten run at once. This case study is about a team that learned that the expensive way, and the one-line design flaw at the center of it — a flaw that passed every test they had right up until they turned on parallelism.

The Problem a Team Faced

A team built a batch-processing feature: an agent that classified support tickets, run across thousands of tickets a night. Sequentially it worked flawlessly. To hit their nightly window, they made it concurrent — a pool of workers, each running the same agent on a different ticket.

That's when the results went strange. Occasionally a ticket would get a classification that clearly belonged to a different ticket. Ticket A's agent would reference details from ticket B. The outputs were being cross-contaminated, but only sometimes, and never reproducibly. Sequential runs were clean; concurrent runs leaked, and the difference between the two was the whole clue nobody could read yet.

The lead engineer's first assumption was a race condition in the database. It wasn't. The bug was in the harness itself, and it was the kind of thing that's invisible until concurrency exposes it. "We wrote the harness assuming one run at a time," she said, "and nothing in it said so out loud." That silent assumption is the heart of most agent concurrency bugs.

The Wrong Approach

Here's the shape of the harness that leaked. Read the class carefully:

[object Object], ,[object Object],:
    messages = []          ,[object Object],

    ,[object Object], ,[object Object],(,[object Object],):
        ,[object Object],.model = model
        ,[object Object],.tools = tools

    ,[object Object], ,[object Object],(,[object Object],):
        ,[object Object],.messages.append({,[object Object],: ,[object Object],, ,[object Object],: task})
        ,[object Object], _ ,[object Object], ,[object Object],(,[object Object],):
            reply = ,[object Object],.model.complete(,[object Object],.messages, ,[object Object],.tools)
            ,[object Object],
        ,[object Object], extract(,[object Object],.messages)

What this does — and this is the bug — is declare

messages

Agent

messages

The insidious part is that this passes every sequential test. The bug is dormant until concurrency wakes it, which is exactly when it's hardest to reproduce — intermittent, timing-dependent, and gone the moment you slow things down to debug. A print statement or a debugger, by serializing execution, makes the very interleaving that causes the bug stop happening, so the tools you'd reach for to find it are the tools that hide it. That's what made this take days: every attempt to observe it made it disappear.

⚠️ Common mistake: Storing per-run state anywhere shared across runs — class attributes, module-level globals, a single reused client object with mutable state. A harness written and tested one-run-at-a-time can hide this indefinitely, because the bug only appears when runs overlap. Before you make anything concurrent, audit every piece of state your harness touches and ask: is this per-run, and is it actually isolated per run? Shared mutable state is the single most common source of concurrency corruption in agent harnesses.

The Correct Approach: Isolated State for Agent Harness Concurrency

The fix is to make every run own its state completely — nothing shared, nothing mutable across runs.

[object Object], ,[object Object],:
    ,[object Object], ,[object Object],(,[object Object],):
        ,[object Object],.model = model
        ,[object Object],.tools = tools

    ,[object Object], ,[object Object],(,[object Object],):
        messages = [{,[object Object],: ,[object Object],, ,[object Object],: task}]   ,[object Object],
        ,[object Object], _ ,[object Object], ,[object Object],(,[object Object],):
            reply = ,[object Object],.model.complete(messages, ,[object Object],.tools)
            ,[object Object],
        ,[object Object], extract(messages)

What this does: it moves

messages

run

With state isolated, safe concurrency is straightforward:

[object Object], asyncio

,[object Object], ,[object Object], ,[object Object],(,[object Object],):
    sem = asyncio.Semaphore(max_concurrent)
    ,[object Object], ,[object Object], ,[object Object],(,[object Object],):
        ,[object Object], ,[object Object], sem:
            ,[object Object], ,[object Object], make_agent().run_async(task)   ,[object Object],
    ,[object Object], ,[object Object], asyncio.gather(*(one(t) ,[object Object], t ,[object Object], tasks))

What this does: it runs tasks concurrently but caps how many run at once with a semaphore, and it builds a fresh agent per task so no state is shared. The semaphore bounds concurrency to respect rate limits; the fresh-agent-per-task guarantees isolation. Both matter, and the isolation is the one people forget.

Notice the

make_agent()

Results and What Changed

Once state was isolated, the cross-contamination vanished completely. The team reran the batch that had produced mysterious mismatches and got clean, correct results every time. The fix was a few characters; finding it had taken days, because the symptom pointed everywhere except the actual cause.

The deeper lesson reshaped how they built harnesses. They adopted a rule: a harness must be safe to run concurrently by default, even when today's use is sequential, because "we'll never run this in parallel" has a way of becoming false. Writing state as per-run from the start costs nothing and removes an entire class of future bug. Their new harnesses had no shared mutable state anywhere, by policy.

The rule paid off in an unexpected place: testing. A harness with no shared mutable state is also a harness whose runs are independent, which means their eval suite could run its tasks in parallel and trust the results — the same isolation that fixed the production bug made their tests faster. Correctness under concurrency and speed under concurrency turned out to be the same property, bought once. That's the quiet payoff of designing for isolation early: it's not just insurance against a future bug, it's a capability you get to use immediately.

⚡ Pro tip: Choose async over threads for agent concurrency. Agent work is overwhelmingly IO-bound — waiting on model and tool API calls — which is exactly where async shines, letting one process juggle hundreds of in-flight agents cheaply. Threads work too but carry more overhead and more room for subtle sharing bugs. Async's explicit

await

⚡ Pro tip: Bound concurrency to your rate limit, not your CPU. The constraint on parallel agents is almost never compute — it's the provider's requests-and-tokens-per-minute cap. Size your semaphore to fit under that shared limit, and coordinate it with your rate limiter, or you'll launch a hundred agents that all get throttled at once.

How to Apply This to Your Situation

The isolate-state principle holds across every concurrent agent workload:

• A data engineering team running thousands of extraction agents nightly makes every run fully state-isolated and bounds concurrency to their token limit, finishing in their window without cross-contamination.
• A customer-support platform handling many simultaneous live conversations gives each conversation a fully independent agent instance, so no two users' contexts can ever touch.
• A research group running large parallel evaluations isolates every eval run's state, so a batch of a thousand concurrent evaluations produces the same results it would one at a time — which is the entire point of an eval.

Across all three, the payoff of isolation is the same: concurrency stops being a source of correctness risk and becomes pure speed. You get the throughput of running everything at once with the reliability of running everything alone, which is the outcome that made parallelism worth reaching for in the first place.

The move is always the same: no shared mutable state, fresh state per run, and a semaphore sized to your rate limit rather than your cores.## Next Steps

Audit your harness today for state that outlives a single run — class attributes, module globals, reused mutable clients. Ask of each one: what happens if two runs touch this at the same time? If the answer is "corruption," move it to per-run scope before you ever go concurrent. It's a cheap change now and an expensive, intermittent bug later.

⚡ Pro tip: Test concurrency deliberately, don't wait for production to find the bug. Write a test that runs the same agent on ten different inputs concurrently and asserts each output matches what that input produces alone. Shared-state leaks show up immediately under this test and are invisible under sequential ones — so the test is the difference between finding the bug on your laptop and finding it in a customer's cross-contaminated results.

Safe agent harness concurrency comes down to isolation: every run owns its state, and nothing mutable is shared. The concurrency patterns and the semaphore-sizing rules you settle on are reusable across every batch and parallel workload you build. Keeping those patterns — and the prompts your concurrent agents run — organized in a library like PromptABCD means your next parallel workload starts isolated and rate-aware by default, instead of hiding a shared-state bug that waits for load to reveal itself.