Using a CLI Agent for Large Refactors

A team I heard about asked an agent to "migrate the whole codebase from callbacks to async/await." One prompt, one giant run — the big-bang approach to cli agent refactoring that feels efficient and quietly isn't. The agent churned for a while and produced a diff that touched two hundred files. Everything compiled. Most tests passed. They merged it. Two weeks later they found the agent had subtly changed error-handling semantics in about a dozen places — errors that used to propagate now got swallowed — and nobody caught it because the diff was too big to read and the tests didn't cover those paths. That's the classic cli agent refactoring disaster, and it comes from one bad instinct: treating a big refactor as a single agent task.

The fix isn't a better agent. It's a completely different way of structuring the work. Let's tear down the big-bang approach and build the version that's actually safe.

Before: The Big-Bang Refactor

The weak approach is the one that feels most natural — describe the end state and let the agent go.

claude
> Migrate the entire codebase from callbacks to async/await. Update all call sites.

What this does: Hands the agent a repo-wide transformation as a single unbounded task. It will attempt all of it in one sweep, producing a diff so large that meaningful review is impossible. This is the setup that hid the swallowed errors.

The appeal is obvious: one prompt, walk away, come back to a finished migration. The problem is equally obvious once it bites you: a two-hundred-file diff is not reviewable by any human, so you're not actually approving the change — you're approving the idea of the change and hoping. And "hope plus compiles plus most tests pass" is exactly how semantic bugs slip through.

⚠️ Common mistake: Judging a refactor by "it compiles and tests pass." A refactor is supposed to preserve behavior exactly. Compilation proves the types line up; passing tests prove the behaviors you happened to test are intact. Neither proves the behavior you didn't test — and legacy code is precisely the code with thin test coverage. That gap is where big-bang refactors bury their damage.

Why It Fails: Reviewability Collapses at Scale

Big-bang cli agent refactoring fails for a reason that has nothing to do with model capability. It fails because review does not scale to giant diffs. A reviewer can genuinely check a twenty-line change. At two hundred files, they scan, they trust, they rubber-stamp. The agent could be flawless and the process would still be unsafe, because the human checkpoint has been rendered meaningless by volume.

There's a compounding problem. In a giant refactor, the agent makes hundreds of small decisions, and a wrong one early (say, how to handle a callback that could fire twice) gets propagated consistently across every file. Now the bug isn't in one place you might spot — it's smeared uniformly across the whole diff, camouflaged by the very consistency that makes it look correct.

The only real defense is to keep each unit of change small enough that a human can actually read it. Which means the refactor can't be one task. It has to be many.

After: Characterize, Then Refactor in Small Commits

The strong approach has two parts the big-bang version skips entirely.

Part one: characterization tests first. Before changing anything, have the agent write tests that capture the current behavior — including the ugly parts. These aren't tests of what the code should do; they're tests of what it does do right now, so any behavior change during the refactor shows up as a failing test.

claude
> Before refactoring anything, write characterization tests ,[object Object], the current
> behavior of the payment callback module, including its error-handling paths.
> Cover what it does now, even ,[object Object], the current behavior looks wrong.

What this does: Builds a safety net that pins down existing behavior — especially error handling — as executable tests. Now if the refactor changes a behavior, a test goes red instead of shipping silently. This is the step that would have caught the swallowed errors.

Part two: refactor in small, reviewable commits, one module at a time, with the characterization tests as the guardrail.

aider --auto-test --test-cmd ,[object Object], src/payments/callbacks.py
> convert only this module,[object Object]

What this does: Scopes the migration to a single module, runs the characterization tests after each edit, and stops if behavior changes. Every commit is small enough to actually read, and the tests guarantee behavior preservation module by module instead of hoping it holds across two hundred files.

⚡ Pro tip: Make "behavior must stay identical" an explicit constraint in every refactor prompt. Agents love to "improve" code while they're in there — renaming things, tightening logic, fixing unrelated smells. In a refactor, that's not help; it's untracked behavior change hiding inside a change that's supposed to preserve behavior. The constraint keeps the agent honest.

How to Size Each Refactor Slice

"One module at a time" raises an obvious question: how big is a slice? Too big and you're back to unreviewable diffs; too small and a hundred-module migration takes forever. The sizing rule that works is behavioral, not lines-of-code: a slice is as big as the largest change a reviewer can hold in their head and verify against the characterization tests in one sitting. For most people that's one module or one clear seam, rarely more.

The pattern that makes this scale is the strangler fig — you grow the new implementation alongside the old one and cut consumers over gradually, instead of replacing everything at once. During cli agent refactoring of a large system, that means the old and new code paths coexist for a while, and you migrate callers in reviewable batches.

[object Object],
aider --auto-test --test-cmd ,[object Object], src/payments/callbacks.py
> add an async version alongside the callback version; don,[object Object]

What this does: Introduces the new async implementation without deleting the old one, so both are live and the characterization tests cover both. You migrate callers to the new path in small commits, and only delete the old path once nothing calls it — each step small and reversible.

The payoff is that a migration can pause safely at any point. If priorities shift mid-refactor, you're not stuck with a half-converted codebase that compiles but behaves inconsistently — you're at a clean boundary where old and new coexist and everything works.

⚡ Pro tip: When old and new paths run side by side, have the agent add a temporary assertion that both produce the same result, and run it in your test suite. That turns "I hope the new path matches the old one" into a checked fact for the whole migration window, and you delete the assertion when the old path goes.

⚡ Pro tip: Size the first slice deliberately small — smaller than feels necessary. The first module of any migration is where you discover the patterns the agent will repeat across all the others, so you want that first diff tiny enough to scrutinize line by line. Get the pattern right once and the rest of the refactor inherits it.

Breaking Down Each Element

Characterization tests are the foundation. They convert "the behavior should stay the same" from a hope into a checkable fact. Without them, you have no objective way to know a refactor preserved behavior; with them, every deviation is a red test.

Small commits are the review mechanism. One module per commit keeps each diff inside the range a human can actually verify. The refactor becomes a series of boring, readable steps instead of one unreadable leap.

The behavior-preservation constraint is the leash. It stops the agent from smuggling improvements into a change that's supposed to be purely mechanical.

The atomic Git trail is the undo. Because each module is its own commit, a step that goes wrong is a single

git revert

git ,[object Object], --oneline   ,[object Object],
git revert <bad-commit>   ,[object Object],

What this does: Shows the module-by-module commit trail and reverts a single bad step without touching the good ones. Granular history is what makes a large refactor recoverable instead of all-or-nothing.

Variations for Different Contexts

A legacy-systems engineer modernizing a decade-old monolith: characterization tests are non-negotiable here, because coverage is thin and behavior is load-bearing in ways nobody remembers. The agent writes the net first, then migrates in the smallest slices the module boundaries allow.

A staff engineer on a well-tested service: you can move in larger slices because your existing suite already characterizes a lot of behavior — but still commit per module and still forbid "improvements," because even good coverage has gaps.

A platform team running the refactor across many repos: script the per-module loop headless in CI, one PR per module per repo, so review stays human-sized even at fleet scale. The agent proposes; a person merges each small PR.

⚡ Pro tip: If a module is too tangled to write characterization tests for, that's a signal — not to skip the tests, but to refactor the testability first in a separate, tiny change before the real migration. Big cli agent refactoring goes wrong most often exactly at the modules too messy to test, because those are where behavior is least understood.

Save and Reuse This

Safe cli agent refactoring is never one task. Write characterization tests to pin current behavior, migrate one module per small reviewable commit, forbid unrelated "improvements," and keep the atomic Git trail so any step is revertable. The big-bang prompt feels faster and quietly ships the bugs you can't see; the module-by-module approach feels slower and actually preserves behavior.

The characterization-test prompts, the "behavior must stay identical" constraint, the per-module commit discipline — these are identical across every large refactor you'll ever run. Keep them in PromptABCD so your next migration starts from the safe structure instead of the two-hundred-file diff that hid swallowed errors for two weeks.