Tool Call Batching Inside the Loop

Picture this: you're an engineer watching an agent gather data for a report, and it's painfully slow. It needs six independent lookups — six customers, six accounts — and it does them one per turn: call the model, get one tool request, run it, call the model again. Six full round-trips through the model for six lookups that don't depend on each other at all. Each round-trip is a second of latency and a full-context model call. Agent tool call batching fixes this by letting the model ask for all six at once, and the speedup is immediate.

Batching means the model emits multiple tool calls in a single turn, your loop runs them together, and it feeds all the results back at once. For independent operations, this collapses many round-trips into one. Let's tear apart the slow version and see exactly where the time goes.

Before: The Weak Prompt

Here's the one-at-a-time loop, the default shape most agents ship with:

[object Object], step ,[object Object], ,[object Object],(max_steps):
    reply = model.call(messages)              ,[object Object],
    ,[object Object], reply.tool_calls:
        call = reply.tool_calls[,[object Object],]            ,[object Object],
        result = tools[call.name](**call.args)
        messages.append(tool_result(call.,[object Object],, result))
    ,[object Object],:
        ,[object Object], reply.text

What this does: processes exactly one tool call per model round-trip, so six independent lookups cost six model calls and six sequential waits — even though nothing about them requires going in order.

For six independent lookups this is six model calls and six waits, strictly sequential. The model never gets the chance to say "run all of these" because the loop only ever reads the first call and ignores the rest.

Why It Fails

The cost is round-trips. Every model call carries the full context and adds real latency, so the expensive, slow part of an agent loop is usually the number of times you go back to the model — not the tools themselves. A loop that needs six sequential model calls to run six independent tools pays that round-trip tax six times over for work that could have been done in one.

There's a subtler waste too. Because each lookup happens on its own turn, the model re-reads and re-reasons about the growing transcript six times, spending tokens re-deciding what it already knew: that it needs these six lookups. It planned the whole batch on turn one and then was forced to dribble it out one call at a time.

This is what agent tool call batching actually recovers: not just the wall-clock time of sequential waits, but the wasted cognition of re-deriving the same plan six times. The model already decided, on turn one, that it needed all six lookups — the one-at-a-time loop just gave it no way to say so. Batching lets the model express the plan it already has in a single breath, instead of being interrogated for it one item per round-trip. The one-at-a-time loop doesn't just slow the agent down; it makes the agent repeat mental work it had no reason to repeat, turn after wasteful turn, all of it avoidable.

⚠️ Common mistake: Assuming tool latency is your bottleneck when it's actually round-trip count. Teams optimize the tools — caching, faster queries — while the real cost is the six sequential model calls wrapped around them. Profile where the wall-clock time goes; for many agents it's the model round-trips, and batching attacks exactly that.

⚡ Pro tip: Count model round-trips per task, not just tool calls. Two agents can make the same number of tool calls but a very different number of model calls — and the model calls dominate both latency and cost. Round-trips are the number to drive down.

After: The Improved Prompt

Two changes: let the model emit multiple calls, and run them together.

[object Object], step ,[object Object], ,[object Object],(max_steps):
    reply = model.call(messages)
    ,[object Object], reply.tool_calls:
        results = run_all(reply.tool_calls)       ,[object Object],
        ,[object Object], call, result ,[object Object], results:
            messages.append(tool_result(call.,[object Object],, result))
    ,[object Object],:
        ,[object Object], reply.text

,[object Object], ,[object Object],(,[object Object],):
    independent = [c ,[object Object], c ,[object Object], calls ,[object Object], ,[object Object], c.depends_on_others]
    ,[object Object], parallel_map(,[object Object], c: (c, tools[c.name](**c.args)), independent)

What this does: handles every tool call the model emits in a turn, running independent ones together and feeding all results back at once — so six independent lookups collapse into a single model round-trip plus one batch of tool execution.

You also nudge the model to batch, in the system prompt:

When you need multiple independent pieces of information, request
ALL of them in a single turn as separate tool calls. Only go one
at a time when a later call genuinely depends on an earlier result.

What this does: tells the model to emit independent tool calls together and reserve sequential calls for genuine dependencies — turning batching from something your loop merely permits into something the model actively does.

Breaking Down Each Element

Three pieces make batching pay off.

Handling all calls — looping over every tool call in the reply, not just the first — is the change that makes multi-call turns possible at all. Running independent calls together — the parallel map — is where the wall-clock speedup comes from, executing six lookups in the time of the slowest one. And the prompt nudge is what makes the model actually batch; without it, many models default to one call per turn out of habit even when they're allowed more.

The result on those six lookups: one model round-trip instead of six, and the six tools running concurrently instead of in series. The report that took twelve seconds takes three.

Worth being precise about where each saving comes from, because agent tool call batching bundles two distinct wins that people conflate. Collapsing six model round-trips into one saves the round-trip tax — context re-sends, model latency, re-reasoning — and that saving exists even if your tools run sequentially afterward. Running the tools concurrently saves the tool-wait time on top. You get the first win from batching alone; you get the second only if your executor actually runs the batch in parallel. Knowing which is which tells you where to look when the speedup is smaller than you hoped.

⚡ Pro tip: Only parallelize calls that are genuinely independent. If call B needs call A's result, batching them runs B on stale or missing input. Have the model mark dependencies, or default to running a batch concurrently only when you can prove the calls don't reference each other.

Variations for Different Contexts

Batching's payoff scales with how much independent work a task has.

A BI analyst's dashboard agent batches a dozen independent metric queries into one turn, turning a slow serial crawl into a single concurrent pull. A recruiter's sourcing agent batches profile lookups across many candidates at once rather than one per turn. A site-reliability engineer's diagnostic agent batches health checks across services simultaneously, so an incident triage that touched ten systems doesn't take ten round-trips to gather the picture.

Same batching mechanism, three tasks — each dominated by independent lookups that had no reason to be sequential.

The common thread is task shape: batching pays exactly in proportion to how much of a task is independent fan-out. A task that's a long chain of dependent steps — each needing the last one's result — gets almost nothing from batching, because there's nothing to batch. A task that's mostly "go gather these N unrelated things" gets a near-N-fold reduction in round-trips. Before you invest in batching, glance at your task's shape: lots of independent lookups means big wins, a strict dependency chain means look elsewhere for speed.

⚡ Pro tip: Log how many tool calls the model emits per turn before and after you add the batching nudge. If the number stays stubbornly at one, the model isn't taking the hint — strengthen the prompt with an explicit example of a good multi-call turn, since models batch far more readily when they've seen the shape you want.

⚡ Pro tip: Cap the batch size. A model that can request unlimited calls per turn will occasionally fire off thirty at once and overwhelm a downstream service or your rate limits. Set a sane ceiling and have the loop chunk anything larger, so batching speeds you up without turning into an accidental denial-of-service on your own tools.

Save and Reuse This

The batched loop — handle-all-calls plus run-independent-concurrently plus the batching nudge — is not task-specific. Any agent whose tasks involve multiple independent lookups benefits, and the machinery is identical; only the tools change. The batching nudge in the prompt is the reusable half people forget — the loop code is easy, but the exact wording that reliably gets a model to emit multi-call turns took iterating to land, and that's the part most worth keeping.

I keep the batched loop and the batching prompt language saved and versioned in PromptABCD, so every new agent starts able to gather independent data in one round-trip — instead of the one-at-a-time default that quietly turns a six-lookup task into six sequential model calls and a user watching a spinner.