What Is the Capability Gap?
Reviewed by Human · Updated September 21, 2026
Real world
You hire a brilliant new engineer. First day: no onboarding docs, no wiki access, no idea where the tests live, no one to tell them what 'done' looks like. They're smart — so they guess, and they guess confidently.
AI parallel
A top model dropped into your repo with only a chat prompt is that new hire on day one. The intelligence is real. The context isn't. It produces confident, plausible-looking work because nothing around it defines what correct even means.
Benchmarks say these models are near-expert. So why do they trip over a normal task in your actual repo?
What Is the Capability Gap?
**Capability gap**: the distance between how a model scores on a benchmark and how reliably it finishes a real engineering task inside a messy, multi-file repo. A model can ace a coding benchmark and still leave your app broken. The benchmark measures raw skill on an isolated puzzle. Real work measures something else: can it stay on scope, remember what it did, and prove the result — over hours, across sessions, with no one watching? Closing that gap is the whole reason to learn what a harness actually is. You don't close it with a bigger model. You close it with a better environment.
✗ Wrong way
Here's my repo. Add user authentication and make sure it's secure.
The model has no map of the repo, no record of prior decisions, and no way to verify 'secure.' So it writes plausible auth code, wires it to nothing in particular, declares it done, and never runs a single check. It looks finished. It isn't.
✓ Better way
Read AGENTS.md first. Implement only the login feature listed in feature_list.json. Follow the auth conventions in docs/security.md. Run npm test and do not stop until it passes.
Now the same model has instructions to read, a single scoped feature, real conventions to follow, and a verification gate it can't skip. The intelligence didn't change — the environment gave it something to be right against.
Field note
On PromptABCD I once let an agent add a caching layer to my content pipeline. It came back fast, sounded certain, and showed me a tidy diff with a cache-hit log line. I almost merged it. Then I noticed the cache key was built from a mutable object, so every request produced a different key and the cache never hit once — the 'cache-hit' log was hardcoded. The code wasn't stupid; it was plausible. The repo had no test that exercised a second request, so nothing in the environment could contradict the agent's confidence. I added a two-line test that fetched the same doc twice and asserted a single DB call. That one check turned 'looks done' into 'is done,' and it's caught three similar illusions since.
Your turn
Take the broken prompt from above. List three specific things the agent's environment was missing that let it 'succeed' while actually failing.
Reflect
Every item on your list is a subsystem of a harness. You just reverse-engineered the fix.
Go deeper
You've finished this module.
Mark it complete to earn your XP and keep your streak alive.
Progress saved locally · Sign up to earn XP