Rate Limiting Inside the Agent Harness

Picture this: you're an engineer who just launched an agent to a few hundred users, and your dashboard lights up red. Every request is failing with a 429. Your agent hit the provider's rate limit, and worse, your naive retry logic is now hammering the API with retries that also get rate-limited, so you're making the problem worse with every attempt. A little agent harness rate limiting — done proactively instead of reactively — would have prevented the whole cascade. Let's tear down the "just retry on 429" approach and build a limiter that stops the storm before it starts — one that shapes traffic on the way out instead of scrambling to recover after a rejection.

Before: Reacting to Rate Limits

Here's what most harnesses do about rate limits, which is to say, nothing until one hits:

[object Object], ,[object Object],(,[object Object],):
    ,[object Object], _ ,[object Object], ,[object Object],(tries):
        ,[object Object],:
            ,[object Object], model.complete(messages)
        ,[object Object], RateLimitError:
            time.sleep(,[object Object],)      ,[object Object],
    ,[object Object], RuntimeError(,[object Object],)

What this does: it fires requests with no restraint, and only when one comes back rate-limited does it wait a flat second and retry. It's purely reactive — the limiter does nothing until you're already over the limit. Under load, this is exactly the wrong behavior, because the retries pile onto the same overloaded endpoint, and a fixed one-second wait means a whole fleet of agents retries in lockstep.

Why That Approach Fails

Reactive rate limiting has three compounding failures that turn a manageable situation into an outage.

It creates retry storms. When many requests hit the limit at once, they all retry at once, producing a synchronized wave of traffic that hits the limit again. The retries don't relieve the pressure; they add to it. You've built a system that responds to overload by generating more load — the software equivalent of a crowd all pushing toward the same exit and jamming it.

It ignores the two different limits. Providers limit both requests per minute and tokens per minute, and for agents the token limit usually binds first, because agents send large, growing contexts. A limiter that only counts requests sails right past the token limit and gets rate-limited anyway, baffled, because by request count it looked fine.

And it wastes the failed calls. Every 429 is a request that consumed a round trip and returned nothing. At scale, a meaningful fraction of your latency and cost can be requests that were always going to be rejected. Reactive limiting pays full price for calls it should never have sent.

Effective agent harness rate limiting inverts all three of these. Instead of reacting after a rejection, it shapes traffic before sending, so requests leave your machine only when they'll be accepted. Instead of counting one dimension, it tracks both requests and tokens. And instead of wasting round trips on doomed calls, it holds them back until there's real budget. The shift from reactive to proactive is the whole fix, and everything below is how you build it.

⚠️ Common mistake: Rate-limiting on request count alone. For most agents, tokens per minute is the binding constraint, not requests per minute — a single agent step can carry tens of thousands of tokens of accumulated context. A request-only limiter will happily stay under the request cap while blowing through the token cap, then act surprised when the 429s arrive. Track both, and gate on whichever you'll hit first.

After: Proactive Token-Aware Rate Limiting

The fix is a client-side limiter that blocks before sending, tracking both requests and tokens, and respecting the provider's own retry hints. A token bucket does this cleanly:

[object Object], time, threading

,[object Object], ,[object Object],:
    ,[object Object], ,[object Object],(,[object Object],):
        ,[object Object],.rpm, ,[object Object],.tpm = rpm, tpm
        ,[object Object],.req_tokens = rpm          ,[object Object],
        ,[object Object],.tok_tokens = tpm          ,[object Object],
        ,[object Object],.updated = time.time()
        ,[object Object],.lock = threading.Lock()

    ,[object Object], ,[object Object],(,[object Object],):
        ,[object Object], ,[object Object],.lock:
            ,[object Object], ,[object Object],:
                now = time.time()
                elapsed = now - ,[object Object],.updated
                ,[object Object],.req_tokens = ,[object Object],(,[object Object],.rpm, ,[object Object],.req_tokens + elapsed * ,[object Object],.rpm / ,[object Object],)
                ,[object Object],.tok_tokens = ,[object Object],(,[object Object],.tpm, ,[object Object],.tok_tokens + elapsed * ,[object Object],.tpm / ,[object Object],)
                ,[object Object],.updated = now
                ,[object Object], ,[object Object],.req_tokens >= ,[object Object], ,[object Object], ,[object Object],.tok_tokens >= est_tokens:
                    ,[object Object],.req_tokens -= ,[object Object],
                    ,[object Object],.tok_tokens -= est_tokens
                    ,[object Object],
                time.sleep(,[object Object],)

What this does: it refills request and token allowances continuously over time, and before any call, it waits until both a request slot and enough token budget are available — then reserves them. Requests are throttled to fit inside both limits proactively, so you rarely hit a 429 at all. The storm never forms because the limiter smooths traffic before it leaves your machine.

The second half is honoring the provider's

Retry-After

[object Object], ,[object Object],(,[object Object],):
    limiter.acquire(estimate_tokens(messages))
    ,[object Object],:
        ,[object Object], model.complete(messages)
    ,[object Object], RateLimitError ,[object Object], e:
        wait = e.retry_after ,[object Object], ,[object Object],       ,[object Object],
        time.sleep(wait + random.random())
        ,[object Object], call_model(messages, model, limiter)

What this does: it reserves budget from the limiter before sending, and if a limit is somehow still hit, it waits exactly as long as the provider asks — plus jitter — instead of guessing. The provider knows when it'll accept traffic again; the

Retry-After

⚡ Pro tip: When a provider doesn't send a

Retry-After

Breaking Down the Limiter

Three design points make proactive rate limiting actually work.

Estimate tokens before sending. The limiter needs to know a request's token cost to reserve budget. A rough estimate from the message length is enough — you're gating, not billing. Over-estimate slightly to stay safe.

Sit below the provider's limit. Set your limiter a bit under the real cap — say 90% — to leave headroom for the occasional retry and for estimation error. Running right at the limit means any wobble tips you over.

Share the limiter across concurrent agents. If ten agents run in parallel against one provider account, they share one rate limit, so they must share one limiter. Ten independent limiters each think they have the full budget and collectively blow through it.

Prefer estimation over exactness. You'll never know a request's precise token count before sending it, and that's fine — the limiter's job is to keep you safely under a cap, not to bill accurately. A slightly conservative estimate that runs you at 88% of your real limit costs you a sliver of throughput and buys you near-zero rejections, which is a trade worth making every time. Chasing exact token accounting in the limiter is effort spent in the wrong place; a rough estimate with headroom does the job.

⚡ Pro tip: Make the limiter account-wide, not per-agent. The provider limits your whole account, so a single shared limiter across every agent and worker is the only thing that actually reflects reality. This is the mistake that catches teams scaling from one agent to a fleet — each new worker quietly assumes it owns the full quota.

Variations for Different Contexts

The right limiter setup shifts with scale, and matching it to your situation keeps you from either over-engineering or under-protecting:

• A solo developer running one agent needs only a simple limiter set just under their tier's limits, mostly to avoid the occasional burst rejection.
• A platform team running hundreds of concurrent agents needs a shared, distributed limiter — often backed by a central store — so every worker across every machine draws from one budget.
• A data team running a large batch job proactively paces the whole batch under the token limit, finishing reliably in a predictable time instead of racing to the limit and stalling on retries.

The batch case is worth dwelling on, because it flips the intuition. When you have a thousand tasks to process, the instinct is to fire them all as fast as possible. But against a rate limit, going flat-out means most requests get rejected and retried, and the batch actually finishes slower than if you'd paced it. A proactively paced batch that runs steadily at 90% of the token limit completes in a smooth, predictable window; an unpaced one thrashes against the limit, wastes round trips on rejections, and finishes late with a pile of 429s in the logs. Slower-but-steady beats fast-but-throttled almost every time at the limit.

⚡ Pro tip: Watch your 429 rate as a health metric. With good proactive limiting it should be near zero — an occasional one is fine, a steady stream means your limiter is set too high or isn't shared across workers. A rising 429 rate is an early warning that your traffic shaping has drifted out of step with your real limits.

Save and Reuse This

Good agent harness rate limiting is proactive, not reactive. Gate on both requests and tokens before you send, sit slightly under the provider's cap, honor

Retry-After

The limiter configuration and the token-estimation logic you settle on are reusable across every agent you run against a given provider. Keeping those configs and the prompt patterns that keep contexts lean — which directly lowers your token-per-minute pressure — organized in a library like PromptABCD means your next agent starts with rate limiting that reflects your real quotas, tuned once and reused everywhere. Shape the traffic before it leaves, and the red dashboard stays green.