Turning a Prototype Harness Into Production Code

Roughly 80% of the agent code teams ship to production started life as a 40-line

while

I've rebuilt this exact path more times than I can count, and the same gaps show up every time. The prototype assumes the network never fails, the model always returns valid JSON, tools never hang, and only one user ever runs at once. Production assumes the opposite of all four. This post walks through what actually changes when you cross that line — with code you can drop into your own loop.

What Is a Production Agent Harness?

A harness is the code that sits between your model and your tools: it sends the prompt, parses the response, dispatches tool calls, feeds results back, and decides when to stop. A prototype harness does the happy path. A production agent harness adds everything that keeps the happy path from being the only path it can survive.

Think of it as the difference between a car that runs and a car that passes a crash test. Both drive. Only one is safe when something hits it. The features that separate them — timeouts, retries, resource caps, structured logging, graceful shutdown — are invisible in a demo and load-bearing in production.

Here's a prototype loop, the kind almost everyone starts with:

[object Object], ,[object Object],(,[object Object],):
    messages = [{,[object Object],: ,[object Object],, ,[object Object],: prompt}]
    ,[object Object], ,[object Object],:
        resp = client.messages.create(model=,[object Object],,
                                       messages=messages, tools=tools)
        ,[object Object], resp.stop_reason != ,[object Object],:
            ,[object Object], resp
        ,[object Object], block ,[object Object], resp.content:
            ,[object Object], block.,[object Object], == ,[object Object],:
                result = TOOLS[block.name](**block.,[object Object],)
                messages.append({,[object Object],: ,[object Object],, ,[object Object],: result})

What this does: Loops the model, runs any tool it asks for, and feeds the result back until the model stops asking for tools. It's correct — and it's a liability. There's no timeout, no retry, no iteration cap, no way to stop it cleanly, and a single bad tool call takes the whole thing down.

Why It Matters

The cost of skipping this work is not theoretical. An agent that loops forever burns tokens at real dollars per minute. A tool that hangs holds a request open until something upstream times out and returns a 502 to your user. A model that returns malformed arguments throws an unhandled exception that kills the worker mid-run, losing all the state that led up to it.

I've watched a single unbounded loop rack up a four-figure API bill overnight because a tool kept returning "please try again" and the model kept trying. The fix was three lines. The lesson was that production is mostly about the failures you didn't imagine during the demo.

Adding the Guardrails That Actually Matter

The first thing to add is a hard iteration cap and a per-run token budget. Both are trivial and both save you from the worst outcomes.

MAX_STEPS = ,[object Object],
MAX_TOKENS = ,[object Object],

,[object Object], ,[object Object],(,[object Object],):
    messages = [{,[object Object],: ,[object Object],, ,[object Object],: prompt}]
    used_tokens = ,[object Object],
    ,[object Object], step ,[object Object], ,[object Object],(MAX_STEPS):
        ,[object Object], time.monotonic() > deadline:
            ,[object Object], RunTimeout(,[object Object],)
        resp = call_with_retry(messages, tools)
        used_tokens += resp.usage.input_tokens + resp.usage.output_tokens
        ,[object Object], used_tokens > MAX_TOKENS:
            ,[object Object], BudgetExceeded(used_tokens)
        ,[object Object], resp.stop_reason != ,[object Object],:
            ,[object Object], resp
        messages.append({,[object Object],: ,[object Object],, ,[object Object],: resp.content})
        messages.append({,[object Object],: ,[object Object],, ,[object Object],: dispatch(resp)})
    ,[object Object], StepLimitExceeded(MAX_STEPS)

What this does: Caps the run at 25 model turns and 200K tokens, enforces a wall-clock deadline passed in by the caller, and raises typed exceptions the caller can catch and classify. Nothing here runs forever, and every failure mode has a name.

Notice the deadline is passed in rather than hardcoded. That's deliberate — a background batch job and a user-facing chat need very different limits, and the harness shouldn't decide that for them. The caller owns the budget; the harness enforces it.

⚡ Pro tip: Set the step limit lower than you think you need, then raise it based on real percentiles. Most legitimate runs finish in single-digit steps. A run that hits 25 is almost always stuck, not working hard. Watching where real runs land tells you far more than guessing.

The second thing to add is retry logic that knows the difference between a transient failure and a permanent one. Retrying a 429 with backoff is correct. Retrying a 400 because your request was malformed just wastes time and hides the bug.

[object Object], ,[object Object],(,[object Object],):
    ,[object Object], i ,[object Object], ,[object Object],(attempts):
        ,[object Object],:
            ,[object Object], client.messages.create(model=,[object Object],,
                                           messages=messages, tools=tools)
        ,[object Object], RateLimitError:
            time.sleep(,[object Object],(,[object Object], ** i, ,[object Object],) + random.random())
        ,[object Object], (BadRequestError, AuthenticationError):
            ,[object Object],  ,[object Object],
    ,[object Object], MaxRetriesExceeded()

What this does: Retries rate-limit and overload errors with exponential backoff plus jitter, but immediately re-raises permanent errors like a malformed request or a bad key. The jitter matters — without it, every worker that got throttled at the same instant retries at the same instant and hammers the API in sync.

Making Tool Calls Safe to Run

The single biggest source of production incidents I see is a tool that hangs. The model asks for a database query, the database is slow, and the tool blocks forever. Every tool needs its own timeout, and every tool needs to fail as a result the model can see, not an exception that kills the loop.

[object Object], ,[object Object],(,[object Object],):
    results = []
    ,[object Object], block ,[object Object], resp.content:
        ,[object Object], block.,[object Object], != ,[object Object],:
            ,[object Object],
        ,[object Object],:
            ,[object Object], timeout(tool_timeout):
                out = TOOLS[block.name](**block.,[object Object],)
            results.append({,[object Object],: ,[object Object],, ,[object Object],: block.,[object Object],,
                            ,[object Object],: ,[object Object],(out)})
        ,[object Object], Exception ,[object Object], e:
            results.append({,[object Object],: ,[object Object],, ,[object Object],: block.,[object Object],,
                            ,[object Object],: ,[object Object],,
                            ,[object Object],: ,[object Object],})
    ,[object Object], results

What this does: Runs each tool under a 15-second timeout and, critically, converts any failure into a tool result marked

is_error

⚠️ Common mistake: Letting tool exceptions bubble up to the top of the loop. When a tool throws and you don't catch it inside

dispatch

Three Places This Bites in the Real World

A fintech team running an agent that reconciles transactions found their prototype worked on 50 test records and fell over on the first real batch of 8,000 — one row had a null the tool didn't expect, the tool threw, and the entire batch failed instead of skipping the bad row. Wrapping tool errors as results fixed it in an hour.

A customer-support SaaS shipped an agent that occasionally looped forever when the knowledge-base search returned nothing useful and the model kept rephrasing the same query. A step cap of 12 turned a runaway into a clean "I couldn't find that" response.

A data-engineering group at a logistics company ran agents as scheduled batch jobs and had no idea why some silently produced nothing. They had no logging. Adding a structured log line per step — step number, tool called, tokens used, latency — turned an unexplained black box into a debuggable pipeline in an afternoon.

⚡ Pro tip: Log one structured event per loop iteration, not one giant blob at the end. When a run fails at step 14, you want to replay steps 1 through 13 exactly as they happened. A per-step log lets you do that. An end-of-run summary tells you it broke but not where.

Running Many Agents at Once Without Crossing Wires

The prototype runs one agent at a time, so it never confronts concurrency. Production runs dozens simultaneously, and the number-one concurrency bug is shared mutable state — a

messages

[object Object],
,[object Object], ,[object Object],:
    messages: ,[object Object],
    used_tokens: ,[object Object], = ,[object Object],
    step: ,[object Object], = ,[object Object],
    run_id: ,[object Object], = ,[object Object],

,[object Object], ,[object Object], ,[object Object],(,[object Object],):
    state = RunState(messages=[{,[object Object],: ,[object Object],, ,[object Object],: prompt}],
                     run_id=,[object Object],(uuid.uuid4()))
    ,[object Object], state.step < MAX_STEPS:
        ,[object Object],
        ...

What this does: Packages all per-run data into a single

RunState

run_id

The other concurrency concern is backpressure. Your model provider has rate limits, and firing 200 runs at once just means 195 of them get throttled and retry into a thundering herd. A bounded worker pool — a semaphore capping concurrent in-flight model calls — smooths the load and keeps you under the limit without dropping work on the floor.

⚡ Pro tip: Size your concurrency limit to your rate limit, not your CPU count. Agent runs are almost entirely I/O-bound waiting on the model, so you can run far more of them than you have cores — but only up to the point where you start hitting 429s. Find that ceiling empirically and set the semaphore just below it.

Common Mistakes When Hardening a Harness

The mistake I see most is treating graceful shutdown as optional. When your deploy platform sends

SIGTERM

SIGKILL

The second mistake is sharing mutable state across concurrent runs. The prototype has one user, so a module-level

messages

The third is optimizing the model before fixing the plumbing. Teams spend weeks tuning prompts while their harness still has no timeouts. The prompt is rarely why production breaks. The plumbing almost always is.

⚡ Pro tip: Add a

run_id

Conclusion

Turning a prototype into a production agent harness is unglamorous work — caps, timeouts, retries, isolated state, structured logs, clean shutdown. None of it shows up in a demo. All of it shows up in your incident count. The good news is that each piece is small, and together they're the difference between an agent you trust in production and one you babysit.

As you harden the loop, the prompts and tool schemas that drive it become assets worth versioning on their own. Keeping those in a dedicated library like PromptABCD means the prompt that survived three rounds of production hardening is the exact one your next service reuses — not a slightly-wrong copy someone pasted from memory.