The SWE-bench Harness Explained for Agent Builders

A team I know spent a week convinced their coding agent was broken. Its patches looked correct — clean diffs, sensible logic — but the swe-bench harness kept scoring them as failures. The bug wasn't in the agent. It was in their understanding of what the harness actually checks. Their patches were logically right and environmentally wrong: the tests failed on setup, not on logic, and the harness reports both the same way — as a fail. Understanding how the harness works would have saved them the week. This guide is that understanding, made practical.

Quick-Start: What SWE-bench Actually Tests

SWE-bench is an open-source benchmark from Princeton that measures whether a model or agent can resolve real GitHub issues. Each task hands the agent a real repository at the exact commit before a fix, plus the issue description, and asks it to produce a patch that resolves the issue. The swe-bench harness is the part that grades those patches — reproducibly, in isolation, the same way every time.

Here's the shape of a single task, conceptually:

instance_id:  scikit-learn__scikit-learn-25102
repo:         scikit-learn at base commit abc123
issue:        "Preserve DataFrame dtypes in transformer output"
your job:     produce a patch (a git diff) that fixes it
grading:      apply patch → run tests → check pass/fail invariants

What this does: it frames the task the harness evaluates — a repo snapshot, an issue, and a patch to produce. Crucially, the agent is not handed the failing test. It must read the issue, explore the code, and infer what a correct fix looks like, which is what makes the benchmark a real test of engineering rather than of test-passing. That single design choice — hiding the test — is what keeps the benchmark honest.

Understanding the Variables

Three concepts explain almost every SWE-bench result, and the week-long confusion above came from missing the third.

FAIL_TO_PASS tests. The tests that were failing before the fix and must pass after it. These prove the patch actually resolves the issue. If they don't all pass, the patch didn't do its job.

PASS_TO_PASS tests. The tests that were already passing and must still pass. These guard against a patch that fixes the issue while breaking something else — or that cheats by deleting the failing test. You can't score a pass by disabling tests, because the previously-passing ones would then break.

The environment. Each task runs in a Docker image pinned to the exact repository state, dependency versions, and toolchain of the original issue. This is the variable teams forget. A patch that's logically perfect still fails if it assumes a package version the pinned environment doesn't have, or touches a file the image lays out differently.

Success requires both test invariants at once: every FAIL_TO_PASS test passes and every PASS_TO_PASS test still passes. One without the other is a fail.

⚡ Pro tip: When a patch you're sure about scores as a failure, check the environment before you touch the logic. Run the harness's own Docker image and apply your patch by hand. Most "the harness is wrong" moments are actually a dependency mismatch or a path assumption — the patch is right for your machine and wrong for the pinned environment. The harness isn't lying; it's holding you to the exact world the bug lived in.

How the SWE-bench Harness Works, Step by Step

The grading sequence is deterministic, and knowing it tells you where things go wrong:

1. Build/pull the Docker image for the instance (pinned repo + deps)
2. Check out the base commit inside the container
3. Apply the agent's predicted patch as a git diff
4. Run the FAIL_TO_PASS and PASS_TO_PASS test sets
5. Resolved only if ALL F2P now pass AND ALL P2P still pass

What this does: it lays out the exact path from patch to verdict. Each step is a place a patch can die — the diff fails to apply cleanly (step 3), or the tests error on an environment issue (step 4), or a P2P regression sneaks in (step 5). Reading a failure means knowing which step it died at.

Running it yourself follows a standard shape. You produce a predictions file — one JSON line per instance with the

instance_id

model_patch

python -m swebench.harness.run_evaluation \
  --dataset_name princeton-nlp/SWE-bench_Verified \
  --predictions_path preds.jsonl \
  --run_id my-first-eval

What this does: it builds the Docker environments, applies each patch, runs the tests, and reports how many instances were resolved. Budget the disk — the images are large, and Docker needs roughly 120 GB free to build the full set comfortably. Tune

max_workers

⚡ Pro tip: Start on SWE-bench Verified, not the full set. Verified is a human-validated subset of 500 issues that real engineers confirmed are solvable with the given tests — so a failure there is almost always your patch, not an ambiguous or broken task. The original full set contains issues with under-specified requirements that make failures hard to interpret. Verified gives you a clean signal while you're learning.

Pro-Level Insights

Three things separate people who understand the harness from people who just run it:

Patch application is fragile. A diff generated against a slightly different file state won't apply. Generate patches against the exact base commit the harness uses, or step 3 fails before any test runs.

The hidden test is the point. Because the agent never sees the FAIL_TO_PASS tests, an agent that somehow reads them has broken the evaluation. There's a well-known failure where an agent on a public split treats a visible test as its stopping signal and tunes against it — which inflates the score meaninglessly. Held-out tests only measure real capability if they stay held out.

A passing patch isn't a good patch. The harness checks that tests pass, not that the code is secure, performant, or idiomatic. A patch can resolve the issue and still be something you'd reject in review. SWE-bench measures a specific, valuable thing — and only that thing.

One more insight that changes how you read scores: harness design itself moves the numbers. The same model can post noticeably different resolution rates depending on how the surrounding scaffold partitions the work — how it localizes the bug, how many attempts it allows, how it feeds test output back. This is why comparing two published scores only makes sense when the harness is held constant. A higher number can reflect a better scaffold rather than a better model, which is exactly the kind of thing a careful reader checks before drawing conclusions.

⚡ Pro tip: When you report your own SWE-bench numbers, state the harness and settings alongside the score — dataset split, number of attempts, model version. A bare "resolved 62%" is nearly meaningless without them, because the next person can't reproduce or compare it. The score is only as credible as the setup you disclose next to it.

Three teams using the harness well:

• A coding-agent startup runs Verified nightly to catch regressions, treating any drop in resolved count as a release blocker.
• A research group uses the per-instance Docker images as ready-made reproducible environments for studying how their agent explores a codebase, not just whether it succeeds.
• A developer-tools engineer builds a private, SWE-bench-style suite from their own repository's real bugs, using the harness's FAIL_TO_PASS/PASS_TO_PASS design as the template for tasks that reflect their actual codebase.

Troubleshooting Common Issues

⚠️ Common mistake: Blaming the harness when a logically correct patch fails, instead of checking the environment. The most common cause of a surprising failure is environmental — a dependency version, a file path, a patch generated against the wrong commit — not the harness misjudging your logic. The harness runs your patch in the exact pinned world the bug lived in, and code that's right for a different world fails there correctly. Reproduce inside the Docker image before concluding the grader is wrong.

Other issues you'll hit:

• Patches won't apply. You generated the diff against a different file state. Use the harness's base commit exactly.
• Out of disk. The images are large. Clear space or run a subset; the full build wants around 120 GB.
• Wildly variable scores. You're running once per instance. Coding agents are nondeterministic — report across multiple runs, as the reliable evaluations do.
• Your local score doesn't match a published one. Different harness versions, dataset splits, or attempt counts produce different numbers for the same agent. Before assuming your setup is wrong, confirm you're comparing against a result that used the same split and the same harness release — a Verified score and a full-set score aren't the same measurement, and mixing them is a common source of confusion.

Your Turn

Start by running the harness on Verified with a handful of instances and a simple patch generator, just to watch the build-apply-test cycle end to end. Once you've seen a real FAIL_TO_PASS flip from failing to passing inside the container, the whole benchmark stops being mysterious. Then reproduce a failure by hand inside its Docker image — that single exercise teaches you more about the harness than any writeup, including this one. You'll see for yourself how a patch that reads perfectly can die on a dependency mismatch, and that lesson sticks in a way no explanation can match.

The swe-bench harness is, at heart, a machine for reproducible test execution, and its FAIL_TO_PASS/PASS_TO_PASS design is a template worth borrowing for your own evals. The prompts you use to generate patches, and the harness-adapter configs that make runs reproducible, are worth keeping organized. Storing those in a library like PromptABCD — tagged by dataset and model — means your next benchmarking run starts from a working setup instead of a week of rediscovering why a right-looking patch scored as a failure.