Scripting a CLI Agent in Your Build Pipeline

Picture this: you're a DevOps engineer at a mid-size SaaS company, and every Monday morning your inbox has three Dependabot PRs that broke the build over the weekend. Each one needs someone to bump a version, fix the two call sites the update changed, and re-run the tests. It's an hour of tedious work nobody wants, so it sits until Wednesday. You've got a terminal agent that could clearly do this — so you decide to wire a cli agent ci pipeline that handles the boring dependency fixes automatically, overnight, before anyone's awake.

Then you have to figure out how to do that safely, because an agent with write access running unattended in CI is exactly the kind of thing that goes wrong at 3 a.m. Here's how it actually played out.

The Problem the Team Faced

The team wanted an overnight job: for each failing dependency update, let the agent apply the version bump, fix the breakages, run the tests, and open a PR for a human to review in the morning. The value was obvious. The risk was equally obvious — nobody would be watching.

[object Object],
claude -p ,[object Object],

What this does: Runs the agent non-interactively and tells it to fix the update and commit to

main

Two things were wrong immediately. Committing to

main

The Wrong Approach

The wrong model treats a CI agent like an interactive one that just happens to run on a server. It isn't. In CI, nobody can answer a permission prompt, nobody can approve a diff, and nobody can re-authenticate a session. Every assumption that interactive use relies on — a human in the loop — is false.

[object Object],
claude   ,[object Object],

What this does: Launches an interactive session in an environment with no terminal to interact with. The job hangs until the runner times out. Even if auth succeeded, an unbounded interactive-style run has no turn cap and no permission scoping — it's the opposite of what CI needs.

⚠️ Common mistake: Using an interactive account login in CI. It needs a browser, and a job that waits for one hangs until the runner times out. CI needs a non-interactive credential — an API key stored as a repository secret — set as an environment variable the agent reads without prompting.

The Correct CLI Agent CI Pipeline Setup

Four changes turn the dangerous version into a safe one.

First, authenticate non-interactively with an API key from secrets, never an interactive login.

[object Object],
,[object Object], ANTHROPIC_API_KEY=,[object Object],

What this does: Provides a credential the agent uses without any browser or prompt. For predictable pay-as-you-go automation, an API key billing at standard rates is the right choice over a subscription login, which draws from personal usage limits and can't authenticate headlessly anyway.

Second, run bounded and non-interactive: print mode, a strict permission mode, an allowlist, and a turn cap.

claude -p ,[object Object], \
  --permission-mode dontAsk \
  --allowedTools ,[object Object], ,[object Object], ,[object Object], ,[object Object], \
  --max-turns 8

What this does: Runs one non-interactive pass that pre-approves only reading, editing, and the test/build commands — nothing else — and stops after eight turns.

dontAsk

Third, write to a branch and open a PR — never to a protected branch.

git checkout -b ,[object Object],
,[object Object],
gh ,[object Object], create --fill --label ,[object Object],

What this does: Isolates the agent's work on a throwaway branch and opens a labeled pull request. A human reviews and merges in the morning. The agent proposes; a person disposes.

main

Fourth, capture cost and outcome as structured output so the job is observable.

claude -p ,[object Object], --output-format json | jq ,[object Object],

What this does: Emits the run's cost and turn count as JSON so your pipeline can log it, alert on anomalies, and stop the job if a run costs more than a threshold. Unattended automation you can't observe is unattended automation you'll eventually regret.

⚡ Pro tip: Set a hard cost ceiling in the job, not just a turn cap. Parse

total_cost_usd

What Happens When the Agent Can't Fix It

The setup so far assumes the agent succeeds. Half the value of a well-built cli agent ci pipeline is what happens when it doesn't. An agent that can't fix a dependency break has three possible behaviors, and only one is acceptable. It can force a bad fix that makes tests pass dishonestly — unacceptable. It can fail silently and leave you thinking the job ran — worse. Or it can fail loudly, open a PR with its diagnosis and a clear "I couldn't complete this" note, and hand the problem to a human with a head start. That third behavior is the one to design for.

[object Object],
,[object Object], claude -p ,[object Object], --output-format json > run.json && \
   jq -e ,[object Object], run.json >/dev/null; ,[object Object],
  npm ,[object Object], && gh ,[object Object], create --fill --label ,[object Object],
,[object Object],
  gh issue create --title ,[object Object], \
    --body ,[object Object],
,[object Object],

What this does: Opens a PR only when the run genuinely succeeded and tests actually pass; otherwise it files an issue containing the agent's own account of why it got stuck. A human starts from a diagnosis instead of a mystery, and a failed fix never sneaks into a PR dressed as a successful one.

The instinct to avoid is letting the agent keep trying until it forces green. An agent under a "make the tests pass" mandate with enough turns will eventually do something ugly — skip the test, weaken the assertion, comment out the failing case. Bounding the run and treating failure as a legitimate, escalatable outcome is what keeps the automation honest.

⚡ Pro tip: Make "I couldn't do this safely" a first-class success path, not a crash. An agent job that cleanly escalates hard cases to humans is more valuable than one that attempts everything, because you can trust its PRs — a green PR from it means a real fix, not a forced one. Design the failure path as carefully as the happy path.

Results and What Changed

With the safe setup, the Monday-morning pile disappeared. The overnight job opened labeled PRs for the dependency fixes, each with the agent's changes already passing the test and build commands, and the team reviewed and merged them over coffee. The critical difference wasn't capability — the naive version was just as capable. It was that every safeguard assumed no human was present: non-interactive auth, bounded runs, branch-and-PR instead of direct commits, and observable cost.

The team also learned to keep the task narrow. The job only handled dependency bumps, not arbitrary fixes, because a tightly-scoped unattended agent is a predictable one. When they later wanted the agent to handle flaky-test quarantining too, they wrote a second narrow job rather than widening the first into a do-everything agent.

⚡ Pro tip: One CI job, one narrow task. A pipeline agent that does exactly one well-defined thing is auditable and debuggable; a pipeline agent asked to "fix whatever's broken" is neither. When you need a second automated behavior, add a second scoped job — resist the urge to grow one agent into a general-purpose fixer.

How to Apply This to Your Situation

A platform team maintaining many microservices: run one scoped agent job per repo that keeps dependencies current and opens PRs, with a shared allowlist and cost ceiling enforced across all of them.

A QA lead automating triage: a nightly job that reproduces newly-filed bugs as failing tests and attaches them to tickets — read-mostly, low-risk, high-value, and it never writes to

main

A solo maintainer of an open-source project: a scheduled job that drafts changelog updates and dependency PRs, gated so contributors review everything, so the automation saves work without ceding control of what merges.

⚡ Pro tip: Make every agent-opened PR visibly labeled and require human approval in branch protection. The label makes agent contributions easy to audit at a glance, and branch protection guarantees the "human reviews in the morning" step can't be skipped under deadline pressure. The safety comes from the process, not from trusting the agent.

One more governance note worth building in from the start: scope the API key the pipeline uses to the narrowest permissions your provider allows, and rotate it on a schedule. A CI credential that leaks is a credential someone else can now run an agent with, so treat it like any other production secret — least privilege, stored in the secrets manager, rotated regularly, and never echoed into a build log.

Next Steps

A safe cli agent ci pipeline flips every interactive assumption: non-interactive API-key auth, bounded and allowlisted runs, branch-and-PR instead of direct commits, and structured output for cost and observability. The naive version failed not because the agent was bad but because it was set up as if a human would be watching. In CI, nobody is.

The auth setup, the bounded-run flags, the branch-and-PR pattern, the cost-ceiling parsing — these are identical across every pipeline you'll automate. Save the whole template in PromptABCD so your next CI agent starts from a reviewed, safe scaffold instead of the direct-to-

main