Handling Streaming Tool Calls in the Harness

Here's a number that surprises people the first time they hit it: when a model streams a tool call, the arguments don't arrive all at once — they arrive as a sequence of partial fragments, sometimes dozens of them, and a naive harness that tries to act on the first fragment will dispatch a tool call with half its arguments missing. Harness streaming tool calls is the problem of correctly assembling those fragments into complete, validated calls before you execute anything — and getting it wrong produces some of the weirdest bugs in agent engineering, because the tool fires with arguments that were never fully formed.

Streaming is worth the trouble: it's what lets your UI show "calling search…" the instant the model commits to it, instead of waiting for the whole response. But the moment you stream, you inherit the assembly problem, and the assembly problem has sharp edges that a non-streaming harness never touches.

What Are Harness Streaming Tool Calls?

Harness streaming tool calls are tool invocations that the model emits incrementally over a stream rather than as a complete block. As the model generates a tool call, the harness receives a series of deltas: the tool name, then chunks of the arguments as a partial JSON string, growing piece by piece until the call is complete. Only when the stream signals that the tool call is finished do you have the full, valid arguments.

The mental model that helps: a streamed tool call is like watching someone type a command character by character. You can see it forming, but you must not hit enter until they're done — pressing enter on

delete --table orders --where "stat

[object Object],
,[object Object],
,[object Object],
,[object Object],
,[object Object],
,[object Object],

What this does: Illustrates the delta sequence for a single tool call — a start event with the name, several partial-argument chunks that concatenate into valid JSON only at the end, and a stop event. The arguments are a valid JSON object only after the last delta; at every intermediate point,

{"query": "quart

Why Handling Them Correctly Matters

The failure mode is acting too early. A harness that tries to parse arguments before the tool call completes gets invalid JSON, and depending on how it handles that, either crashes, retries needlessly, or — worst case — dispatches with partial or defaulted arguments. A search for

"quart"

"quarterly revenue"

The second reason is concurrency. Models often emit multiple tool calls in a single turn, and their deltas can interleave in the stream. If your assembly logic assumes one call at a time, two concurrent calls corrupt each other's arguments — call

t1

t2

⚠️ Common mistake: Trying to parse tool arguments on every delta to "act as soon as possible." Partial JSON isn't valid JSON, and parsing it either fails or, with a lenient parser, produces a half-object you might act on. Wait for the completion signal before parsing. The whole point of the stop event is to tell you when the arguments are finally safe to read.

Assembling Streamed Calls Safely

The correct pattern buffers argument fragments per tool-call ID and only parses and dispatches on the stop event.

[object Object], ,[object Object],:
    ,[object Object], ,[object Object],(,[object Object],):
        ,[object Object],.buffers = {}          ,[object Object],

    ,[object Object], ,[object Object],(,[object Object],):
        ,[object Object], event[,[object Object],] == ,[object Object],:
            ,[object Object],.buffers[event[,[object Object],]] = {,[object Object],: event[,[object Object],], ,[object Object],: ,[object Object],}
        ,[object Object], event[,[object Object],] == ,[object Object],:
            ,[object Object],.buffers[event[,[object Object],]][,[object Object],] += event[,[object Object],]
        ,[object Object], event[,[object Object],] == ,[object Object],:
            buf = ,[object Object],.buffers.pop(event[,[object Object],])
            args = json.loads(buf[,[object Object],])      ,[object Object],
            ,[object Object], {,[object Object],: buf[,[object Object],], ,[object Object],: args, ,[object Object],: event[,[object Object],]}

What this does: Keeps a separate buffer per tool-call ID, appends each argument fragment to the right buffer, and parses the accumulated string into arguments only when that call's stop event arrives. Concurrent calls stay isolated because each has its own buffer keyed by ID, and no call is ever parsed before it's complete. This handles both edges — partial arguments and interleaved concurrent calls — in one small class.

⚡ Pro tip: Surface the tool name to your UI on the start event, but hold the arguments until the stop event. The name is known immediately and safe to show ("🔍 Searching…"), giving the user instant feedback, while the arguments are still assembling. You get responsive UI without acting on incomplete data — the best of both, and it costs nothing beyond splitting when you read the name from when you read the arguments.

Dispatching Completed Calls, Sometimes in Parallel

Once a call completes, you dispatch it — and when the model emitted several, you often want them running concurrently rather than one after another.

[object Object], ,[object Object], ,[object Object],(,[object Object],):
    ,[object Object], ,[object Object], ,[object Object],(,[object Object],):
        args = validate(call[,[object Object],], call[,[object Object],])   ,[object Object],
        ,[object Object], ,[object Object], dispatch(call[,[object Object],], args)
    ,[object Object], ,[object Object], asyncio.gather(*(one(c) ,[object Object], c ,[object Object], calls))

What this does: Runs all completed tool calls from a turn concurrently with

asyncio.gather

⚡ Pro tip: Only parallelize tool calls that are genuinely independent. If two calls in the same turn have an ordering dependency — one reads what the other writes — running them concurrently is a race. When in doubt, run sequentially; the latency win from parallelism isn't worth a nondeterministic bug. Reserve concurrency for reads and clearly-independent actions.

Handling Errors and Cancellation Mid-Stream

Streaming introduces failure modes a batch response never has, because the response arrives over time and anything can go wrong partway through. The two that bite hardest are a stream that dies mid-call and a user who cancels while tool calls are assembling.

A dropped stream leaves you with tool-call buffers that started but never got their stop event — half-formed JSON that must never reach dispatch. The assembler needs a cleanup path that discards incomplete buffers and reports the interruption, rather than optimistically trying to parse whatever fragment it managed to collect.

[object Object], ,[object Object],(,[object Object],):
    incomplete = ,[object Object],(,[object Object],.buffers.keys())
    ,[object Object],.buffers.clear()
    ,[object Object], incomplete:
        ,[object Object], StreamInterrupted(,[object Object],)

What this does: On stream end, checks whether any tool-call buffers are still open — meaning their stop event never arrived — clears them, and raises rather than letting partial JSON leak forward. A tool call that was 80% streamed is thrown away cleanly instead of dispatched with truncated arguments, which is exactly the outcome you want: no action is far safer than a half-specified one.

Cancellation is the other case. When a user cancels mid-stream, you want to stop assembling and dispatch nothing that hasn't already fired — while not leaving a partially-executed turn in a weird state. The cleanest approach is to treat cancellation as "finish assembling what's in flight, execute nothing new," so the run stops at a clean boundary.

⚡ Pro tip: Buffer the whole turn's tool calls and dispatch them only after the turn's stream completes cleanly, rather than firing each call the instant its stop event arrives. Waiting for a clean end-of-turn means a stream that dies at call three of four doesn't leave calls one and two already executed against a turn that never finished — you either run the whole coherent turn or none of it, which is a much easier state to reason about on retry.

Common Mistakes

The mistake that causes the strangest bugs is a shared argument buffer across concurrent calls. When two tool calls stream at once and their fragments land in one buffer, you get JSON that's a mangled interleaving of both — and it sometimes even parses, into nonsense. Always key buffers by tool-call ID.

The second is forgetting that a stream can end mid-call. A dropped connection or a truncated response can leave a tool call started but never stopped, so its buffer holds incomplete JSON forever. Handle the incomplete-call case explicitly — discard it and surface an error — rather than letting a half-parsed fragment leak into dispatch.

The third is skipping validation on the streaming path. It's easy to validate arguments carefully in your non-streaming code and then bypass that validation when you rebuild the loop for streaming. The assembled arguments deserve exactly the same validation as any other tool call — streaming changes how the arguments arrive, not how much you should trust them.

Conclusion

Harness streaming tool calls give you responsive, real-time agent UIs at the cost of an assembly problem: fragments arrive incrementally and interleaved, and you must buffer per call ID, wait for the completion signal, parse only then, validate, and dispatch — in parallel when the calls are independent. Get that pipeline right and streaming is pure upside; get it wrong and you'll chase bugs where tools fire with arguments that never existed — the kind of bug that reproduces one time in twenty and looks like the model "misbehaving" when it's really your assembly logic acting on a fragment.

Because the assembly-and-dispatch pipeline is fiddly and identical across every streaming agent you build, keep it versioned alongside your prompts and tool schemas in a library like PromptABCD, so the buffering, validation, and parallelism logic you got right once is what every future agent inherits — instead of each one rediscovering the interleaved-buffer bug the hard way.