Building a CLI Agent That Runs Tests Automatically

A developer asked their coding agent to fix a failing edge case, and it did — confidently. It edited the function, explained the fix, declared success, and moved on. What it never did was run the tests. Three other tests now failed because the "fix" changed behavior the rest of the code depended on, and nobody knew until CI went red an hour later. The agent had done the work and skipped the one step that proves work is done. Building a cli agent auto run tests loop closes that gap: after every change, the agent runs the suite, reads the results, and keeps going until things actually pass. This guide builds that loop.

An agent that edits code but doesn't run tests is guessing. An agent that runs tests after each change is verifying. The difference is the whole value.

Quick-Start (Copy This Right Now)

The core pattern is a verification step wired into the agent's loop: after the model makes a change, run the tests and feed the results back as an observation.

[object Object], subprocess

,[object Object], ,[object Object],(,[object Object],):
    r = subprocess.run(cmd.split(), capture_output=,[object Object],, text=,[object Object],, timeout=,[object Object],)
    passed = r.returncode == ,[object Object],
    ,[object Object],
    tail = ,[object Object],.join(r.stdout.splitlines()[-,[object Object],:])
    ,[object Object], passed, tail

,[object Object], ,[object Object],(,[object Object],):
    passed, output = run_tests()
    status = ,[object Object], ,[object Object], passed ,[object Object], ,[object Object],
    messages.append({,[object Object],: ,[object Object],,
                     ,[object Object],: ,[object Object],})
    ,[object Object], passed

What this does: Runs the test suite after a change, captures whether it passed, and feeds a compact summary back into the conversation as an observation. The agent sees the real result of its edit — pass or fail — and can act on it, instead of assuming success.

Wire verify_step into your loop after any tool that modifies code, and the agent stops flying blind.

Understanding the Variables

The test command is your contract with the agent about what "working" means. pytest -q, npm test, go test ./... — whatever your project uses becomes the ground truth the agent must satisfy. Making it configurable per project means the same agent verifies a Python service and a Node app without code changes.

The output tail matters more than it looks. A failing suite can produce hundreds of lines, and feeding all of it back floods the context window and buries the signal. The last 20–30 lines usually hold the actual failures and the summary, which is what the agent needs to diagnose the problem. Truncating to the tail keeps the observation focused and cheap.

The pass/fail boolean is the loop's real stop condition. This is the shift in mindset: the agent isn't done when the model says it's done — it's done when the tests are green. That boolean, not the model's confidence, is what should end the task.

⚡ Pro tip: Return the exit code's meaning, not just its number. Map a non-zero exit to a clear "TESTS FAILED" label and a zero to "TESTS PASSED" in the text you feed back, because the model reasons about labeled outcomes far more reliably than about raw return codes it has to interpret.

Why Should a cli agent auto run tests After Every Change?

Because the alternative is an agent that reports success it hasn't verified, which is worse than no agent at all — it's a confident source of broken code. When you make a cli agent auto run tests after each edit, you convert its claims from "I believe this works" into "the suite confirms this works," and only the second one is trustworthy.

The loop also lets the agent fix its own mistakes. When the tests fail, that failure goes back into the conversation, and the model reads the actual error — the assertion that broke, the exception that was raised — and tries again. Without the test result, it has no feedback signal and no way to know it went wrong. With it, the agent iterates toward green the way a developer does.

[object Object], ,[object Object],(,[object Object],):
    messages = [{,[object Object],: ,[object Object],, ,[object Object],: goal}]
    ,[object Object], attempt ,[object Object], ,[object Object],(max_attempts):
        agent_loop(messages)              ,[object Object],
        ,[object Object], verify_step(messages):         ,[object Object],
            ,[object Object], ,[object Object],
        messages.append({,[object Object],: ,[object Object],,
                         ,[object Object],: ,[object Object],})
    ,[object Object], ,[object Object],

What this does: Loops edit-then-verify until the tests pass or an attempt cap is hit. A failed run feeds the real error back and prompts another attempt, so the agent self-corrects. The cap ensures a genuinely stuck problem fails honestly instead of looping forever.

⚠️ Common mistake: Letting the agent declare a task complete without a passing test run. If "done" is defined by the model's say-so rather than a green suite, you'll ship changes that look right and break things elsewhere — exactly the silent-failure story that opened this guide. Make a passing test run a hard precondition for success, enforced in code.

Step-by-Step: Running Only the Affected Tests

Running the full suite after every change is correct but slow, and slow feedback makes the loop painful on a large project. The refinement is to run a targeted subset first for fast iteration, then the full suite once before declaring done.

[object Object], ,[object Object],(,[object Object],):
    ,[object Object],
    targets = map_files_to_tests(changed_files)   ,[object Object],
    ,[object Object], targets:
        ,[object Object], run_tests(,[object Object],)
    ,[object Object], run_tests()   ,[object Object],

What this does: Runs only the tests related to the files the agent changed, giving fast feedback during iteration. You still run the complete suite as a final gate, but the tight edit-fix cycle uses the quick subset so the agent isn't waiting five minutes per attempt.

The step-by-step: after each edit, run the affected tests for a fast signal; iterate until those pass; then run the full suite once to catch anything the change broke elsewhere; only then declare success. Fast where you need speed, thorough where you need certainty.

⚡ Pro tip: Guard against the agent "fixing" a test to make it pass. A model under pressure to reach green can decide the easiest path is to edit the test's assertion rather than the code. Detect changes to test files during a fix task and flag or block them — the agent's job is to make the code satisfy the tests, not to make the tests satisfy the code.

⚡ Pro tip: Cache the pre-change test state. Run the suite once before the agent touches anything, so you know which tests were already failing. Then a test that fails after the change is clearly the agent's doing, and one that was red before isn't wrongly blamed on it — the agent works against a known baseline instead of a mystery.

What If the Project Has No Test Suite?

Plenty of real codebases have thin test coverage or none, and an auto-run-tests loop seems to have nothing to run. The answer isn't to skip verification — it's to broaden what "verification" means. Even without unit tests, there are cheap checks that catch a large fraction of broken changes.

[object Object], ,[object Object],(,[object Object],):
    checks = {
        ,[object Object],: ,[object Object],,
        ,[object Object],:  ,[object Object],,
        ,[object Object],:  ,[object Object],,
        ,[object Object],:   ,[object Object],,
    }
    ,[object Object],
    ,[object Object], run_tests(checks[cmd_type])

What this does: Falls back to lighter-weight checks — does it parse, does it build, does it type-check, does it lint — when a full test suite isn't available. None of these prove correctness, but each catches a category of obvious breakage the model would otherwise declare fixed.

These proxies are a ladder. A change that doesn't parse is definitely broken; one that fails type-checking probably is; one that builds cleanly at least isn't catastrophically wrong. Running whatever rung the project supports gives the agent some ground-truth signal, which beats trusting its unverified word entirely.

⚡ Pro tip: When a project has no tests, have the agent offer to write one for the change it just made. A verification loop is far more valuable on the next edit if this edit leaves a test behind, and an agent that grows the suite as it works slowly turns an untested codebase into a verifiable one.

Pro-Level Variations

Teams shape the verify loop to their stack and risk.

A backend engineer wires the loop to run linting and type-checking alongside tests, so "green" means passing tests and clean types — catching a whole class of errors the tests alone would miss.

A frontend developer whose full suite is slow runs affected unit tests in the agent's inner loop and defers the browser-based end-to-end tests to a single final pass, balancing fast iteration against thorough coverage.

A data engineer whose "tests" are data-quality checks feeds the agent the row counts and validation failures after each pipeline edit, so the agent verifies the data looks right, not just that the code ran without error.

Each is the same edit-verify-iterate loop with a different definition of what passing means.

Troubleshooting Common Issues

If the agent loops without converging, your attempt cap is doing its job — but check whether the test output you're feeding back is actually useful; a truncation that cuts off the real error leaves the agent guessing. If tests are slow enough to make the loop unbearable, add affected-test selection before optimizing anything else. If the agent starts editing tests, add the test-file guard above.

And if the suite passes locally in the loop but fails in CI, your agent's environment differs from CI's — pin the same versions and flags, because an agent that verifies against a different setup than the one that gates your releases isn't really verifying.

Your Turn

Add a verify step to your agent: run the tests after each code change, feed a truncated result back, and make a passing run the hard requirement for declaring done. Add affected-test selection for speed and a test-file guard for integrity. You'll have an agent that proves its work instead of asserting it.

The test commands, the file-to-test mappings, and the prompts that tell the agent how to respond to failures are reusable configuration that encodes how your project defines "working." Keeping them in a library like PromptABCD means your next cli agent auto run tests loop starts from a verification setup you've already tuned, so every agent you build proves its changes green before it ever claims success.