How to Handle Secrets and API Keys in a Harness

Picture this: you're debugging a flaky agent run at 11 p.m., you scroll back through the conversation log to see what the model did, and there — in plain text, in a tool result the model received and then echoed back in its reasoning — is your production Stripe key. It's been in every log line, every trace, every error report for weeks. Anyone with read access to your logging system has had your payment credentials the whole time. This is the failure mode that agent harness secrets management exists to prevent, and it's more common than anyone likes to admit, because the naive way to give an agent a secret is to just... hand it the secret.

The core problem is that a model's context is the least private place in your entire system. Everything in it gets logged, traced, and sometimes echoed back. A secret that enters the context has effectively been published. This teardown starts with the way most harnesses handle secrets, shows exactly why it leaks, and rebuilds it so the model never sees a credential at all.

Before: The Weak Prompt

Here's the common pattern — the harness passes secrets to tools by putting them where the tool can read them, which usually means the environment or, worse, the arguments.

[object Object], ,[object Object],(,[object Object],) -> ,[object Object],:
    key = os.environ[,[object Object],]
    resp = requests.post(,[object Object],,
                         auth=(key, ,[object Object],),
                         data={,[object Object],: amount, ,[object Object],: customer_id})
    ,[object Object], resp.text          ,[object Object],

What this does: Reads the Stripe key from the environment and charges a customer, then returns the raw API response straight back to the model. The key never appears in the arguments, which feels safe — but the response often contains sensitive data, and if the tool ever errors, the exception message can include the key or the full request. That return value goes directly into the model's context and from there into every log.

Why It Fails

There are three separate leaks hiding in that twelve-line function, and most harnesses have all three.

The first is the return value. Raw API responses contain more than you think — account identifiers, partial card numbers, internal reference tokens — and dumping the whole thing into the model's context means dumping it into your logs. The model rarely needs the full response; it needs a summary of what happened.

The second is error handling. When

requests.post

The third is the environment itself. If this tool runs inside a sandbox — and dangerous tools should — then

os.environ["STRIPE_KEY"]

⚠️ Common mistake: Assuming that keeping a secret out of the tool's arguments keeps it out of the model's context. The arguments are only one path. Return values, error messages, and inherited environments are three more, and secrets management has to close all four or it closes none.

After: The Improved Prompt

The rebuilt version follows one rule: secrets live only in a trusted broker, never in the model's context, never in the sandbox, never in a log. The model refers to credentials by name, and the broker substitutes the real value at the last possible moment, in trusted code, discarding it immediately after.

[object Object], ,[object Object],:
    ,[object Object], ,[object Object],(,[object Object],):
        ,[object Object],._store = store            ,[object Object],

    ,[object Object], ,[object Object],(,[object Object],) -> ,[object Object],:
        ,[object Object],
        key = ,[object Object],._store.get(spec[,[object Object],])      ,[object Object],
        resp = requests.request(spec[,[object Object],], spec[,[object Object],],
                                auth=(key, ,[object Object],), json=spec.get(,[object Object],),
                                timeout=,[object Object],)
        ,[object Object], key                                        ,[object Object],
        ,[object Object], summarize_response(resp)                ,[object Object],

What this does: The broker runs in the trusted harness layer, fetches the secret by reference right before the call, uses it, deletes it, and returns only a scrubbed summary — never the raw response. The model asked to call an API "using the credential named

stripe_prod

Breaking Down Each Element

Each piece closes one of the three leaks, so it's worth seeing how they map.

Reference, not value. The model and any sandboxed tool traffic in names —

secret_ref: "stripe_prod"

[object Object], ,[object Object],(,[object Object],) -> ,[object Object],:
    ,[object Object], {,[object Object],: resp.status_code,
            ,[object Object],: resp.ok,
            ,[object Object],: resp.json().get(,[object Object],) ,[object Object], resp.ok ,[object Object], ,[object Object],,
            ,[object Object],: scrub(resp.text[:,[object Object],]) ,[object Object], ,[object Object], resp.ok ,[object Object], ,[object Object],}

What this does: Returns only the fields the model actually needs — did it succeed, and the resulting charge ID — with any error text truncated and scrubbed. The full response, with all its sensitive extras, stays in the broker and dies there. The model gets enough to continue reasoning and nothing it could leak.

Just-in-time fetch. The broker pulls the secret from a vault at call time rather than holding it in a long-lived variable or config. Combined with short-lived credentials — tokens that expire in minutes — this means even a leaked value is worthless quickly.

Trusted-layer execution. Because the broker runs in the harness and not in the sandbox, the secret is never present in the environment the untrusted code can read. The sandbox gets the result of a privileged call, never the ability to make one itself.

⚡ Pro tip: Run a scrubber over everything on its way into a log or the model context, keyed to your secret patterns, as a backstop. Even with the broker pattern, a stray credential in a code path you forgot will show up somewhere eventually. A regex that redacts anything shaped like your keys is the seatbelt that catches the mistake the architecture missed.

Variations for Different Contexts

The broker pattern adapts to how much your tools need to touch secrets directly.

For tools that call a handful of known APIs, bake the calls into the broker as above — the model picks the operation, the broker owns the credentials. This is the tightest and the default choice.

For agents that need to run arbitrary user-provided integrations, issue the sandbox a short-lived, narrowly-scoped token instead of a root credential — a token that can only do the one thing this run needs, expiring in minutes. If it leaks, the blast radius is one operation and a few minutes wide.

For local development, use a secrets manager even then, and especially keep secrets out of the

.env

⚠️ Common mistake: Logging the request for debugging. It's tempting, when a tool call fails, to log the full outgoing request so you can see what went wrong — but that request often carries the auth header. Log the sanitized spec (method, URL, secret reference) instead of the realized request, and you get your debugging visibility without publishing your keys.

Rotating, Scoping, and Catching Leaks

Good agent harness secrets management doesn't stop at hiding the value — it assumes a leak will eventually happen anyway and limits what a leaked credential is worth. Two habits do most of that work.

Scope every credential to the narrowest thing it needs to do. An agent that reads from one table doesn't need a database admin key; it needs a read-only credential for that one table. When you can, issue credentials that are both scoped and short-lived, so a value that escapes is useless outside a tight window and a tiny slice of your system. The broker makes this natural, because it's the only thing minting credentials and can hand out a fresh, narrow one per call.

Then plan for rotation as a routine, not an emergency. Because the model only ever holds a reference like

stripe_prod

⚡ Pro tip: Rotate a credential on a schedule and then confirm the old value actually stops working. A rotation you never verify is a rotation that may have silently failed, and you find that out during the incident you were trying to prevent. Because references insulate the agent from the value, you can rotate aggressively without touching agent code — so there's no excuse not to.

⚡ Pro tip: Seed your logs and model context with a canary credential — a real-looking key that's actually a tripwire wired to alert if anyone ever uses it. If that canary is ever exercised, you know instantly that something is reading secrets out of a place it shouldn't, and you know it before a real credential gets abused the same way.

Save and Reuse This

The broker, the response summarizer, and the scrubber are exactly the kind of infrastructure you write once and must never get subtly wrong twice. A single new tool that returns a raw response undoes the whole pattern. Keep the broker and its conventions — reference-not-value, summarize-don't-return, scrub-everything — documented and versioned alongside your prompts and tool definitions in a library like PromptABCD, so every tool anyone adds inherits the same discipline, and the 11 p.m. discovery of a key in your logs becomes a mistake your architecture simply doesn't permit.