Error Taxonomy: Classifying Harness Failures

When was the last time your agent failed and you could tell, instantly, what kind of failure it was — a model problem, a tool problem, a bug in your harness, or a bad input from the user? For most teams the answer is "never, they all look the same," and that's the exact problem agent harness error classification solves. When every failure is just a generic exception, you can't route them, can't retry the right ones, can't alert on the ones that matter, and can't tell a flaky tool from a broken harness. This teardown takes a harness with no error structure and rebuilds it with a taxonomy that drives real behavior.

The core idea: errors aren't all the same, and treating them the same throws away the single most useful piece of information you have about a failure — its category. A rate limit and a null-pointer bug are both "exceptions," but everything you'd want to do about them is different.

Before: The Weak Prompt

Here's the error handling most harnesses start with — catch everything, log it, maybe retry, move on.

[object Object], ,[object Object],(,[object Object],):
    ,[object Object],:
        ,[object Object], agent.execute(prompt)
    ,[object Object], Exception ,[object Object], e:
        log.error(,[object Object],)
        ,[object Object], retry(prompt)          ,[object Object],

What this does: Catches any exception, logs a generic message, and retries. It treats a transient rate limit, a malformed user input, a hung tool, and a genuine bug in your own code as if they were the same event. They are not, and handling them identically is wrong for at least three of the four — the only one a blanket retry serves correctly is the transient model error, and it happens to be the least urgent of the bunch.

Why It Fails

Retrying indiscriminately is the first problem. A rate limit should be retried with backoff. A malformed input should not — it'll fail identically every time, and retrying just wastes calls and delays the inevitable error. A bug in your harness definitely shouldn't be retried; it needs a human. Blanket retry does the wrong thing for most failure types.

The second problem is invisibility. When every failure logs "run failed," your error metrics are a single meaningless number. You can't see that tool errors spiked while model errors stayed flat, because you never distinguished them. The most important signal in your failures — the pattern across categories — is erased before it's recorded.

The third is misrouting. Different failures belong to different owners. A model overload is the provider's problem and resolves itself; a tool timeout might be an infrastructure issue; a validation error is a user problem; a harness exception is your bug. With no classification, every failure lands in the same undifferentiated pile, and the ones that need urgent attention hide among the ones that don't. The practical effect is that on-call engineers either get paged for every transient rate limit until they mute the alert entirely, or they mute it and then miss the real bug — both outcomes flow directly from refusing to distinguish the categories.

⚠️ Common mistake: Catching

Exception

After: The Improved Prompt

The rebuilt harness defines a taxonomy of error types and makes each one carry its own retry policy and severity, so classification drives behavior automatically.

[object Object], ,[object Object],(,[object Object],):
    retryable = ,[object Object],
    severity = ,[object Object],

,[object Object], ,[object Object],(,[object Object],):        ,[object Object],
    retryable = ,[object Object],
    severity = ,[object Object],

,[object Object], ,[object Object],(,[object Object],):         ,[object Object],
    retryable = ,[object Object],                   ,[object Object],
    severity = ,[object Object],

,[object Object], ,[object Object],(,[object Object],):   ,[object Object],
    retryable = ,[object Object],                  ,[object Object],
    severity = ,[object Object],

,[object Object], ,[object Object],(,[object Object],):     ,[object Object],
    retryable = ,[object Object],
    severity = ,[object Object],              ,[object Object],

What this does: Defines a hierarchy where each error type declares whether it's retryable and how severe it is. A

ModelError

ValidationError

InternalError

Breaking Down Each Element

Each error class maps to a distinct owner and response.

ModelError

ToolError

ValidationError

InternalError

[object Object], ,[object Object],(,[object Object],):
    ,[object Object],:
        ,[object Object], agent.execute(prompt)
    ,[object Object], HarnessError ,[object Object], e:
        metrics.increment(,[object Object],, tags={,[object Object],: ,[object Object],(e).__name__,
                                              ,[object Object],: e.severity})
        ,[object Object], e.retryable ,[object Object], under_retry_budget():
            ,[object Object], retry(prompt)
        ,[object Object],

What this does: Classifies at the catch site by type, records the failure with its type and severity as metric dimensions, and retries only if the error declares itself retryable and there's budget. One handler now does the right thing for every category, and every failure is counted under its true type — so your dashboards finally show which kind of failure is happening.

⚡ Pro tip: Make the error type a dimension on every failure metric. "Errors are up" is useless; "ToolError is up 5x while everything else is flat" points straight at the problem. The taxonomy isn't just for routing — it's what turns your error dashboard from a single scary line into an actionable diagnosis of exactly what's failing.

Variations for Different Contexts

For user-facing agents, add a

UserError

ValidationError

For agents with critical tools, subclass

ToolError

For multi-agent systems, add error classes for cross-agent failures — a sub-agent that failed, a coordination timeout — so failures in one agent don't masquerade as bugs in another. Classification scoped to the architecture keeps blame accurate.

⚠️ Common mistake: Classifying errors but then not wiring the classification to behavior — defining nice error types and still retrying everything and alerting on nothing. The taxonomy is only worth building if

retryable

severity

Turning Classification Into a Feedback Loop

The payoff of agent harness error classification compounds once the typed failures start flowing into your metrics, because now your errors form a dataset you can learn from instead of a wall of noise you tune out. Over weeks, the distribution across types tells you where to invest: a steady baseline of

ModelError

ToolError

InternalError

That last point is worth dwelling on. Because

InternalError

[object Object], ,[object Object],(,[object Object],):
    counts = metrics.query(,[object Object],, group_by=,[object Object],, window=window)
    ,[object Object], {t: {,[object Object],: c, ,[object Object],: ROUTING[t]} ,[object Object], t, c ,[object Object], counts.items()}
    ,[object Object],
    ,[object Object],

What this does: Aggregates failures by type over a window and pairs each with its routing action, producing a report that says not just what failed but what to do about each category. This is the operational view classification unlocks — a triage list instead of a body count. The counts point you at the problem; the routing tells you who owns it.

⚡ Pro tip: Review your error-type distribution weekly, not just when something breaks. The classes that are slowly trending up — a tool that's failing 2% more each week — are the incidents you can prevent instead of respond to. A flat error dashboard hides these trends; a per-type one surfaces them while they're still small enough to fix calmly.

Save and Reuse This

An error taxonomy is a design that every agent in your fleet should share, because inconsistent error classes across services mean your cross-agent dashboards and alerting can't line up — one service's "ToolError" is another's generic exception, and the aggregate is noise. Keep your error hierarchy and its retry-and-severity policies versioned alongside your prompts in a library like PromptABCD, so every agent classifies failures the same way, your fleet-wide error metrics actually compose, and the critical bug never again hides in the transient noise. A shared taxonomy is what lets you build one alerting rule, one dashboard, and one on-call runbook that work across every agent you run — instead of a different, incompatible error scheme per service that no aggregate view can ever reconcile.