Spot the Harness Problem
Reviewed by Human · Updated September 21, 2026
Live Demo — AI Terminal
Read this real-shaped failure transcript from an agent working on the knowledge-base app. As you read, decide: is this the model failing, or the harness failing?
Your prompt
Is It a Model Problem or a Harness Problem?
Ask one question: could a competent engineer have finished this task with the right docs, a way to run the code, and a checklist for done? If yes, the model was capable — the environment failed it. **Model problem**: the task genuinely exceeds what the model can reason about, even with perfect context. **Harness problem**: the model could do it, but nothing around it supplied context, scope, or verification. In practice, the vast majority of day-to-day agent failures are harness problems wearing a model-problem costume. The 'done, but broken' pattern is the giveaway — real reasoning limits look like confusion, not false confidence.
Same task, same model — no harness vs. a minimal harness
✗ No-harness run
One sentence, an empty repo, no definition of done. Nothing records what the agent tried and nothing checks the result. It writes plausible code, says 'done,' and moves on.
✓ Minimal-harness run
Same model, same task. Now it has instructions, a single scoped feature, and a verification gate. It can't declare victory without runnable proof.
Your turn
Re-read the transcript. Write the single line you would add to the agent's instructions that would have prevented the false 'done.' Then name which harness subsystem your line belongs to.
Reflect
If your line was 'run the app and paste the output before claiming done,' you just wrote a verification gate — the exact thing this transcript was missing.
Diagnose before you debug
When an agent fails, resist the urge to swap models or rewrite the prompt ten times. First diagnose: model problem or harness problem? Nine times out of ten the fix isn't a smarter model — it's a minimal harness that gives the agent something to be right against. Every failure you just diagnosed maps onto one of the five subsystems that make a harness work. Learn those, and 'the AI is unreliable' turns into 'the AI is missing verification' — a problem you can actually fix.
Go deeper
You've finished this module.
Mark it complete to earn your XP and keep your streak alive.
Progress saved locally · Sign up to earn XP