Handling Model Output That Won't Parse in Your Harness

Most advice about handling an agent output parse error is wrong in the same way: it tells you to add a JSON-repair library and move on. Reach for the library, patch the broken braces, keep going. That's backwards. Repairing malformed output silently guesses at what the model meant, and a guess in your parser is a bug you'll never see coming. The better approach starts by asking a question the repair library skips entirely: why didn't it parse? Because the three reasons need three different fixes, and repair addresses none of them well.

What Is an Agent Output Parse Error?

An agent output parse error happens when the harness expects structured output — usually JSON matching a schema — and the model returns something that doesn't fit. The parser can't turn the text into the object the next step needs, so the loop stalls or, worse, proceeds on garbage.

The reflex is to treat every parse failure as the same event: "the JSON is broken, fix it." But lumping them together is exactly why so many agents handle failures poorly. A truncated response and a mis-schema'd response look similar to a

json.loads

Why It Matters More Than It Looks

Parse errors sit on the critical path of every structured agent. If your recovery is bad, you get one of two bad outcomes: the agent burns retries re-emitting output that fails the same way, or the repair library "fixes" the output into something plausible but wrong and the mistake flows downstream silently. The first wastes money. The second corrupts data. Neither is acceptable in anything you'd deploy.

Here's the reframe that fixes both: parse errors come in three distinct kinds, and diagnosing the kind is most of the cure.

• Truncation. The response was cut off mid-output because it hit the token limit. The JSON is valid up to the point it stops. This isn't a reasoning failure — the model needed more room.
• Malformation. The model emitted broken syntax — a trailing comma, an unquoted key, prose wrapped around the JSON. The model can fix this if you tell it what broke.
• Schema mismatch. The JSON parses fine but violates your rules — a string where a number belongs, a missing required field. The model misunderstood the shape you wanted.

Each kind has a different right answer, and a single generic "retry" only accidentally solves one of them.

⚡ Pro tip: Before you build any repair logic, log the raw output of every parse failure for a week. You'll find your failures cluster heavily into one of the three kinds — usually the same one — and you can fix the specific cause instead of building a general repair machine you don't need. Most teams discover 80% of their failures are one kind with one cheap fix.

How to Diagnose and Recover From Each Kind

Detection is cheaper than you'd expect. Here's a classifier that routes each failure to the right recovery:

[object Object], json

,[object Object], ,[object Object],(,[object Object],):
    stripped = raw.strip()
    ,[object Object],
    ,[object Object], stripped ,[object Object], stripped[-,[object Object],] ,[object Object], ,[object Object], ,[object Object], ,[object Object], stripped.count(,[object Object],) > stripped.count(,[object Object],):
        ,[object Object], ,[object Object],
    ,[object Object],:
        obj = json.loads(stripped)
    ,[object Object], json.JSONDecodeError:
        ,[object Object], ,[object Object],
    ,[object Object],:
        schema.model_validate(obj)
        ,[object Object], ,[object Object],
    ,[object Object], Exception:
        ,[object Object], ,[object Object],

What this does: it checks for an unclosed structure first (a strong truncation signal), then tries to parse, then validates against the schema — returning which of the three kinds it hit. That label is what lets your harness respond intelligently instead of blindly retrying.

The recovery for each kind is specific:

[object Object], ,[object Object],(,[object Object],):
    ,[object Object], kind == ,[object Object],:
        ,[object Object], call_model(max_tokens=,[object Object],)             ,[object Object],
    ,[object Object], kind == ,[object Object],:
        ,[object Object], call_model(feedback=,[object Object],)
    ,[object Object], kind == ,[object Object],:
        ,[object Object], call_model(feedback=,[object Object],)

What this does: truncation gets a bigger token budget rather than a re-prompt, because re-prompting a truncated-but-correct response just wastes a turn. Malformation gets a "JSON only" nudge. Schema mismatch gets the schema restated. Same failure surface, three targeted cures.

That truncation branch is the insight most guides miss entirely. A truncated response isn't wrong — it's incomplete. Retrying it at the same token limit reproduces the exact same cutoff. The fix is more budget, and you can only apply that fix if you detected truncation as distinct from malformation in the first place.

⚡ Pro tip: For truncation specifically, ask the model to continue rather than restart when the output is expensive to regenerate. Feed back the partial JSON and prompt "continue from where you stopped." On long structured outputs this halves the token cost of recovery compared to regenerating from scratch.

Real Scenarios Where the Distinction Pays Off

The three-way split earns its keep across very different agents:

• A marketing analyst running an agent that generates twenty ad variants as one JSON array kept hitting failures. Diagnosis: pure truncation — twenty variants overflowed the token limit. The fix was a bigger budget and smaller batches, not a repair library.
• A backend engineer whose extraction agent returned JSON wrapped in explanatory prose was seeing malformation. A one-line "return only the JSON object" in the tool description cut the failures by most of their volume.
• A clinical-data specialist whose agent returned valid JSON with dosages as strings had a schema-mismatch problem. Restating the schema with an explicit

  "type": "number"

  Copy
  on the dosage field fixed it — repair would have "helpfully" cast the string and hidden the ambiguity.

Three agents, three parse-error kinds, three unrelated fixes. A generic repair step would have papered over all three without solving any.

Preventing Parse Errors Before They Happen

Diagnosing and recovering is the cure. Prevention is cheaper, and three habits eliminate most parse errors before your classifier ever runs.

Use native tool calling, not free-text JSON. When the provider's API returns arguments as a validated structured field, you skip the entire "find the JSON in the prose" problem — malformation from wrapping text simply can't occur. If you're prompting for raw JSON in the response body, you've opted into a class of failures the structured API doesn't have.

Set a token budget that fits your largest expected output, plus headroom. Most truncation is a budget you set too low for the job. If your agent returns arrays that sometimes hold thirty items, size the budget for forty. Truncation you prevented is worth ten truncations you recovered from.

Show the model an example, not just a schema. A schema tells the model the rules; a filled-in example shows it the shape. One concrete example in the tool description does more to reduce schema-mismatch errors than three paragraphs of field descriptions, because models pattern-match to examples more reliably than they parse specifications.

TOOL_DESC = ,[object Object],

What this does: it pairs a concrete example with the rules, so the model has both a pattern to copy and the constraints to respect. In practice the example carries most of the weight — the model sees the exact shape it should produce and reproduces it.

⚡ Pro tip: When you find a recurring parse failure, add the corrected form as a second example in the tool description rather than adding more retry logic. Teaching the model the right shape once, at the source, retires the failure permanently — while retry logic pays the cost of that failure on every single run it happens.

Three teams that cut failures at the source: a support-tooling engineer who switched from prompted JSON to native tool calling and watched malformation errors nearly disappear; a analytics lead who sized token budgets to the largest report and ended chronic truncation; and a data platform engineer who added one worked example per tool and cut schema mismatches sharply.

Common Mistakes

⚠️ Common mistake: Silently repairing malformed output and continuing. A repair library that closes an unbalanced brace is guessing at structure, and its guess can invert meaning — moving a value into the wrong field, or truncating an array at the break point. If you must repair, log both the original and the repaired version and treat any repaired output as lower-confidence. Better still, feed the failure back and let the model — which knows what it meant — produce correct output itself.

A few more traps:

• Retrying truncation at the same token limit. It'll fail identically. Detect truncation and raise the budget.
• No cap on parse retries. A model that can't satisfy the schema will loop forever. Bound it and route persistent failures to human review.
• Discarding the raw output on failure. The raw text is your best diagnostic. Keep it — it tells you which of the three kinds you're actually fighting.
• Treating a slow parse as a broken one. A large valid output takes longer to generate and parse; don't let a timeout on a big-but-correct response get miscategorized as malformation and thrown away. Size your timeouts to the output you actually expect.

Conclusion

Handling an agent output parse error well isn't about repairing broken JSON — it's about diagnosing why it broke and applying the one fix that matches. Truncation wants budget. Malformation wants a clean re-prompt. Schema mismatch wants the schema restated. Collapse those three into a single "repair and continue" and you'll waste tokens on some failures and corrupt data on others.

The recovery prompts that do this well — the "JSON only" nudges, the schema restatements, the "continue from where you stopped" instructions — are reusable across every structured agent you build. Keeping them in a prompt library like PromptABCD, tagged by the parse-error kind they cure, means the next time an agent returns something unparseable, your fix is already written and proven. Diagnose the kind once; reuse the cure everywhere.