Multi-Tenant Agent Harness Design

A SaaS company shipped an agent to all its customers on shared infrastructure, and within a week two things went wrong. First, one enterprise customer kicked off a huge batch job that consumed every worker, and every other customer's agent ground to a halt — a classic noisy-neighbor stall. Then, worse, a bug in how runs were keyed meant one customer's agent occasionally loaded another customer's conversation history. Two tenants, one harness, no isolation. This is the failure that makes multi tenant agent harness design its own discipline, and this teardown rebuilds a shared-everything harness into one where tenants can't starve or see each other.

The trap is that a single-tenant harness looks like it "just works" for many tenants — you point all of them at the same code and it runs. It runs right up until one tenant's load or one tenant's data crosses into another's, and by then the isolation you skipped is a production incident instead of a design decision.

Before: The Weak Prompt

Here's the shared-everything version — one queue, one state store, one set of credentials, tenant identity treated as just another field.

[object Object], ,[object Object],(,[object Object],):
    state = STORE.load(prompt.run_id)          ,[object Object],
    result = agent.run(prompt, tools=TOOLS)    ,[object Object],
    STORE.save(prompt.run_id, state)
    ,[object Object], result

What this does: Loads state from a single global store keyed by run ID, runs the agent with one shared set of tools and credentials, and saves back to the same global store. The

tenant_id

Why It Fails

The data-isolation failure is the scary one. When state is keyed only by run ID in a global store, a run-ID collision, a caching bug, or a mistaken query returns tenant B's data to tenant A. There's no boundary preventing it — just the hope that every key is always right, forever, across every code path. Hope is not isolation, and in a multi-tenant system a single cross-tenant leak can be a breach-notification event.

The noisy-neighbor failure is the common one. A shared worker pool means work is first-come-first-served across all tenants, so one tenant submitting a thousand runs starves everyone else. Their heavy usage becomes your other customers' outage, and the affected customers did nothing wrong — they just shared a queue with a whale.

The credential failure is the quiet one. Shared credentials mean the agent acting for tenant A holds the same keys it uses for tenant B, so any confused-deputy bug or injection can reach across tenants. The blast radius of a compromise is every tenant at once, because there was never a per-tenant boundary to contain it.

⚠️ Common mistake: Treating tenant ID as application data rather than an isolation boundary. If tenant separation depends on every query remembering to filter by

tenant_id

After: The Improved Prompt

The rebuilt harness makes tenancy structural. State is namespaced per tenant, credentials are scoped per tenant, and the scheduler enforces per-tenant fairness — so isolation doesn't depend on anyone remembering to filter.

[object Object], ,[object Object],(,[object Object],):
    store = STORE.for_tenant(tenant.,[object Object],)        ,[object Object],
    creds = CREDS.scoped_to(tenant.,[object Object],)         ,[object Object],
    tools = build_tools(creds, tenant.limits)  ,[object Object],
    ,[object Object], ,[object Object], SCHEDULER.admit(tenant.,[object Object],):         ,[object Object],
        ,[object Object], RateLimited(tenant.,[object Object],)
    state = store.load(prompt.run_id)          ,[object Object],
    ,[object Object], agent.run(prompt, tools=tools)

What this does: Derives a tenant-scoped store whose keyspace physically cannot address another tenant's data, builds tools bound to credentials scoped to just this tenant, and checks a per-tenant admission gate before running. The tenant boundary is now enforced by construction at three layers — storage, credentials, scheduling — so no single forgotten filter can breach it.

Breaking Down Each Element

Each layer closes one of the three failures, and it's worth seeing how.

Namespaced storage closes the data leak.

STORE.for_tenant(tenant.id)

[object Object], ,[object Object],:
    ,[object Object], ,[object Object],(,[object Object],):
        ,[object Object],.base, ,[object Object],.prefix = base, ,[object Object],
    ,[object Object], ,[object Object],(,[object Object],):
        ,[object Object], ,[object Object],.base.get(,[object Object],.prefix + run_id)   ,[object Object],

What this does: Wraps the base store so every key is transparently prefixed with the tenant ID. Code that uses this handle can't accidentally read another tenant's data, because there's no way to express another tenant's key through it. The isolation moves from "remember to filter" to "can't not filter."

Scoped credentials close the cross-tenant reach. Each tenant's agent gets credentials that work only for that tenant's resources, so a confused-deputy bug in tenant A's run has nothing tenant B's it could touch. This is the multi tenant agent harness version of least privilege — least privilege, per tenant.

Fair scheduling closes the noisy-neighbor stall.

[object Object], ,[object Object],(,[object Object],):
    inflight = ,[object Object],.counts.get(tenant_id, ,[object Object],)
    ,[object Object], inflight < ,[object Object],.per_tenant_cap        ,[object Object],

What this does: Caps how many runs any single tenant can have in flight at once, so no tenant can consume the whole worker pool. A tenant that submits a thousand runs gets its fair slice and queues the rest; every other tenant keeps flowing. One tenant's spike stops being everyone's outage.

⚡ Pro tip: Enforce the tenant boundary at the lowest layer you can — ideally the storage handle and credential provider — not in business logic. Business-logic checks are the ones a future refactor quietly drops. A store handle that physically can't form another tenant's key survives refactors, because the isolation is a property of the object, not a line someone has to keep writing.

Variations for Different Contexts

For strict-isolation requirements — regulated data, security-sensitive tenants — go beyond namespacing to physical separation: a separate database, separate worker pool, sometimes separate infrastructure per tenant. More expensive, far stronger, and sometimes contractually required.

For many small tenants where per-tenant infrastructure would be wasteful, logical isolation (namespacing + scoping + fair scheduling, as above) is the pragmatic default. Most SaaS agents live here.

For a hybrid, tier it: shared logical isolation for your long tail of small customers, dedicated isolation for the enterprise tenants who need or pay for it. The harness supports both by treating the isolation level as a per-tenant setting rather than a global architecture choice.

⚠️ Common mistake: Adding per-tenant rate limits but forgetting per-tenant cost limits. A tenant can stay under your concurrency cap and still run up an enormous model bill with a few expensive long runs. Track and cap spend per tenant, not just concurrency, or one tenant's runaway costs land on your invoice with no ceiling.

Seeing Per-Tenant Behavior

Isolation stops tenants from harming each other; observability tells you which tenant is doing what, which is the other half of running a multi tenant agent harness well. The moment many tenants share infrastructure, "the system is slow" becomes a useless statement — you need "tenant 4471's runs are slow," and that requires tenant identity stamped on every metric, log, and trace.

Thread the tenant ID through as a first-class dimension on everything: run counts, latency, token spend, error rates, all sliced by tenant. This is what turns a vague performance complaint into a specific diagnosis, and it's what lets you spot a tenant whose usage pattern is about to become a problem before it actually does.

[object Object], ,[object Object],(,[object Object],):
    metrics.increment(,[object Object],, tags={,[object Object],: tenant_id})
    metrics.gauge(,[object Object],, step.tokens, tags={,[object Object],: tenant_id})
    metrics.timing(,[object Object],, step.latency, tags={,[object Object],: tenant_id})

What this does: Tags every metric with the tenant ID so you can break down load, cost, and latency per tenant. When one tenant's numbers spike, you see exactly whose — and you can act on that specific tenant instead of guessing across an anonymous aggregate.

Per-tenant visibility also feeds capacity decisions. Watching cost and load per tenant over time tells you which customers are heavy enough to warrant dedicated infrastructure and which are comfortable on shared — turning your isolation tiering from a guess into a data-driven choice.

⚡ Pro tip: Alert on per-tenant anomalies, not just global thresholds. A tenant whose token spend suddenly jumps 10x over their own baseline is worth a look even if your global spend is fine — it's often a runaway loop or a misconfigured integration on their side, and catching it early saves both their bill and your capacity.

Save and Reuse This

The tenant-scoped store, the scoped credential provider, and the fair scheduler are cross-cutting infrastructure that every multi-tenant agent must share — and if two services implement tenancy differently, one of them has the leak. The failure mode is specific and nasty: a new service copies 90% of the isolation pattern, misses the credential-scoping 10%, and now one endpoint acts across tenants while everything else is clean. Shared primitives are the only reliable defense, because they make the correct behavior the default and the leak something you'd have to deliberately opt out of.

⚡ Pro tip: Write a cross-tenant leak test into your suite — spin up two tenants, have one try every path to read the other's data, and assert every attempt fails. Run it on every change. Isolation is exactly the kind of property that a refactor silently weakens, and a standing test that actively tries to break tenancy is worth more than any amount of careful review, because it fails loudly the moment someone reintroduces the shared-everything bug.

Keep these isolation primitives and their conventions versioned alongside your prompts and tool definitions in a library like PromptABCD, so every agent you ship inherits the same structural boundaries, and "make it multi-tenant" is a property new services get for free instead of an isolation model each team reinvents — and occasionally gets wrong.