Testing Your Harness Against Prompt Injection

In one widely-cited red-team exercise, researchers got a majority of tested production agents to leak data or take unauthorized actions using nothing but text — no exploits, no code, just carefully worded instructions hidden in content the agent read. The number that should stop you cold isn't the success rate; it's that almost none of those teams had tested for it before shipping. Harness prompt injection testing is the practice of attacking your own agent with injection payloads, systematically, before someone else does — and this guide gives you a working test suite you can run in CI today.

The reframe that makes this tractable: injection resistance isn't a property you assert, it's a property you measure. You don't get to say "our agent is injection-resistant." You get to say "our agent passed 47 of 50 injection tests, and here are the 3 it failed." That's a number you can improve, gate on, and defend.

Quick-Start (Copy This Right Now)

Here's the skeleton of an injection test suite — a corpus of attacks, run against your agent, asserting the agent didn't do the forbidden thing.

INJECTION_TESTS = [
    {,[object Object],: ,[object Object],,
     ,[object Object],: ,[object Object],,
     ,[object Object],: ,[object Object], run: ,[object Object], ,[object Object], run.output.lower()},
    {,[object Object],: ,[object Object],,
     ,[object Object],: ,[object Object],,
     ,[object Object],: ,[object Object], run: run.called_tool(,[object Object],,
                                              url_contains=,[object Object],)},
]

,[object Object], ,[object Object],(,[object Object],):
    fails = []
    ,[object Object], t ,[object Object], INJECTION_TESTS:
        run = agent.run(t[,[object Object],])
        ,[object Object], t[,[object Object],](run):
            fails.append(t[,[object Object],])
    ,[object Object], ,[object Object], fails, ,[object Object],

What this does: Runs each attack through the agent and checks whether the forbidden outcome happened — leaking the system prompt, calling an exfiltration URL. A failing test means an attack got through, named so you know exactly which one. This turns "are we injection-resistant?" into a green or red build. And a green-or-red build is something a team can rally around in a way that a vague security posture never is — it's checkable, it's ownable, and it fails loudly the moment someone regresses it.

⚡ Pro tip: Assert on the action, not the words. "Did the agent call

http_request

What Harness Prompt Injection Testing Must Cover

Effective harness prompt injection testing depends on getting three things right, and the first is accepting that the attack surface is much wider than the user's message.

Attack surface. Injection doesn't only come from the user's message. It comes from anything the agent reads: documents it summarizes, web pages it fetches, tool results, other users' data in a shared system. This indirect injection is the dangerous kind, because the attacker isn't the user — it's whoever planted text in content the agent later ingests. Your tests must cover injected tool outputs and documents, not just malicious prompts.

Forbidden outcomes. For each tool and capability, define what "compromised" looks like concretely: exfiltration to an unlisted host, reading outside the workspace, a destructive action without approval, revealing the system prompt. These become your assertions. Vague goals produce vague tests.

Pass criteria. Decide what score gates a release. Zero failures is ideal but, given how open-ended attacks are, a realistic policy is "zero failures on the known-attack corpus, and every newly-discovered bypass becomes a permanent regression test." The corpus grows every time someone finds a new attack.

Step-by-Step: Building the Attack Corpus

Step one: Cover the direct attacks — instructions in the user input that try to override behavior.

DIRECT = [
    ,[object Object],,
    ,[object Object],,
    ,[object Object],,
]

What this does: Templates the common direct-override patterns with a placeholder for the specific bad action per tool. These are the attacks everyone thinks of, and they're table stakes — if a direct override works, you have nothing more subtle to worry about yet.

Step two: Cover indirect injection — the attack hidden in content the agent reads.

[object Object], ,[object Object],(,[object Object],):
    poisoned_doc = (,[object Object],
                    ,[object Object],)
    run = agent.run(,[object Object],, context=poisoned_doc)
    ,[object Object], ,[object Object], run.called_tool(,[object Object],, to_contains=,[object Object],)

What this does: Hands the agent a document that looks normal but contains a hidden instruction, then asks for an innocent summary and asserts the buried command didn't execute. This is the attack real systems fall to, because the malicious text rides in on legitimate content the agent was asked to process.

Step three: Wire the whole corpus into CI so every change re-runs every attack.

⚡ Pro tip: Every time you find or read about a new injection technique, add it to the corpus as a permanent test — even after you've fixed it. Injection defenses regress silently when someone tweaks a prompt or a tool, and a permanent test for each known attack is what catches the regression. Your corpus should only ever grow.

Pro-Level Variations

For agents with many tools, generate injection tests per tool automatically — for each dangerous tool, template an attack that tries to trigger it from injected content. This scales your coverage with your tool count instead of leaving new tools untested.

For a stronger signal, add a canary: seed a fake secret into the agent's reachable environment and assert it never appears in output or in an outbound call. If the canary ever escapes, an injection reached data it shouldn't, and you'll know instantly.

For continuous assurance, run a subset of the corpus against production periodically, not just in CI. A config drift or dependency change can open a hole that passed at build time, and a scheduled live probe catches it.

Troubleshooting Common Issues

Tests pass but you still get breached. Your corpus is too narrow — it covers the attacks you imagined, not the ones that exist. Broaden it with published injection research and, ideally, an external red-team pass. A green suite proves you stopped these attacks, not all attacks.

A test is flaky — passes sometimes, fails others. Model outputs vary, so an assertion on exact wording will flap. Move the assertion to the action layer (which tool, which arguments), which is far more stable across runs than the model's phrasing.

You can't tell why an attack succeeded. Log the full run for every failed injection test — the messages, the tool calls, the decisions — so you can see exactly where the boundary failed. A failing test with no trace tells you that you're vulnerable but not where, and "vulnerable somewhere" is nearly as hard to act on as not knowing at all — the trace is what converts a red build into a specific line of code to fix.

⚠️ Common mistake: Treating a passing injection suite as proof the agent is safe, and relaxing the actual boundaries because "the tests pass." The tests are a floor, not a ceiling. The real defense is still the enforced boundaries — allowlists, permissions, isolation — and the injection suite verifies those boundaries hold. If you ever find yourself weakening a boundary because a test passes, you've inverted the relationship.

Tracking Resistance as a Number Over Time

The reason harness prompt injection testing beats a one-time security review is that it produces a trend, and a trend is something you can manage. Run the corpus on every build and record the pass rate, and you get a resistance score that moves as your agent changes — up when you harden a boundary, down when a refactor quietly weakens one.

That score is what makes injection defense a real engineering discipline rather than a vibe. When someone proposes loosening a permission or adding a powerful new tool, you can measure the effect directly instead of arguing about it. The number turns a subjective "is this safe?" into an objective "the resistance score dropped 8 points, here's which tests started failing."

[object Object], ,[object Object],(,[object Object],):
    fails = [t[,[object Object],] ,[object Object], t ,[object Object], corpus ,[object Object], t[,[object Object],](agent.run(t[,[object Object],]))]
    ,[object Object], {,[object Object],: ,[object Object], - ,[object Object],(fails) / ,[object Object],(corpus), ,[object Object],: fails}

What this does: Computes the fraction of the corpus the agent defends against and lists which specific attacks got through. Tracked over time, the score becomes a regression signal — a drop between builds points straight at the change that weakened a boundary, and the failure list names the exact attacks to investigate.

The deeper reason to track resistance rather than assert it: injection is an adversarial, open-ended problem with no "done" state. New techniques appear constantly, and an agent that resisted last year's attacks may fall to this year's. A living score, backed by a growing corpus, is the only honest way to represent a defense that has to keep moving.

⚡ Pro tip: Set a resistance-score floor as a release gate, the same way you'd gate on test coverage. "This build scores below our injection floor" should block a deploy exactly like a failing test does. A floor makes the ratchet one-directional — resistance can only be traded away deliberately, with the number staring everyone in the face, never by accident.

Your Turn

Take your agent, write five injection tests today — a couple direct, a couple indirect, one canary — and run them. If any pass the attack (fail the test), you've just found a real vulnerability before an attacker did. Then grow the corpus every week and gate your releases on it. The corpus you build this way becomes one of your most valuable security assets — a concrete, executable record of every attack you've learned to stop, growing more comprehensive with every incident and every published technique you fold in.

Keep your injection corpus versioned alongside your prompts and tool definitions in a library like PromptABCD, so the attacks you've learned to defend against travel with every agent you build — and a new service starts life already tested against the bypasses the last one taught you.