The Observe Step: Feeding Tool Results Back to the Model

Why does your agent keep making decisions that ignore what its tools just told it? You wired the tools correctly. They run. They return real data. And the agent acts as if it never saw the results. If that sounds familiar, the culprit is almost always the agent loop observation step — the moment where a tool's output is supposed to become something the model can actually read and use. Get that step wrong and the smartest model on earth behaves like it has amnesia.

The observation step is deceptively simple to describe: run the tool, take its result, and put it back into the conversation so the model sees it next turn. But how you put it back — the format, the placement, the framing — decides whether the model treats the result as a fact or as noise. This is the story of an agent that looked broken and was really just being fed badly.

The Problem the Team Faced

A fintech team built an agent to answer customer questions about their accounts. It had a

get_balance

get_transactions

They assumed the model was weak and lobbied to upgrade it. But the transcripts told a stranger story: the tools were returning correct data, and the model was ignoring it. The

get_balance

{'bal': 4021.55, 'ccy': 'USD', 'as_of': '...'}

A quick way to feel the problem: imagine reading that tool output yourself with no other context. If you — a human who knows the domain — can't instantly tell what the result says and which fields matter, the model has no chance.

{'bal': 4021.55, 'ccy': 'USD'}

bal

The Wrong Approach

Their first fix was to make the tools return "more." More fields, more detail, longer strings — on the theory that the model needed more context.

[object Object],
,[object Object], ,[object Object],(account.__dict__)   ,[object Object],

What this does: returns the entire account object as an unlabeled string dump — which makes the observation longer and harder to read, burying the three fields that mattered under thirty-seven that didn't.

It got worse. Now the model had to find the balance inside a wall of internal fields —

_cache_key

_partition

updated_by

What made this failure so hard to spot is that it looked like a model problem from every angle a dashboard shows. Success rate: low. Model: the obvious variable to blame. Tools: verified working in isolation. The one thing no dashboard surfaces is the actual bytes handed to the model each turn — and that was the entire bug. Metrics point at the model; only transcripts point at the observation step. If your monitoring never shows you the raw context, you'll spend weeks blaming the wrong component.

⚠️ Common mistake: Assuming an agent ignoring tool results needs more data. Usually it needs cleaner data. A model drowning in an unlabeled forty-field dump will ignore the result far more often than one handed three labeled fields. When an agent seems to ignore observations, shrink and label them before you add anything.

The Correct Prompt

The fix wasn't the model or the amount of data. It was formatting the observation as something a model reads well: labeled, scoped to what matters, and clearly marked as a tool result.

[object Object], ,[object Object],(,[object Object],):
    ,[object Object], (
        ,[object Object],
        ,[object Object],
        ,[object Object],
        ,[object Object],
    )

What this does: turns a raw dict into a clearly labeled, human-readable observation that names the tool, formats the number, states its freshness, and explicitly tells the model these figures are authoritative — removing every excuse to guess.

messages.append({
    ,[object Object],: ,[object Object],,
    ,[object Object],: call.,[object Object],,
    ,[object Object],: format_balance(raw_result),
})

What this does: appends the formatted observation using the proper tool-result role and matching call ID, so the model recognizes it as the answer to the exact call it made rather than as free-floating text.

Results and What Changed

The wrong-answer rate dropped by roughly three-quarters — same model, same tools, same questions. The only change was the agent loop observation step: from a raw dict dump to a labeled, scoped, framed result. The model stopped guessing because it could finally tell what it was looking at.

The "I don't have access" answers vanished entirely. Those had come from the model failing to recognize a tool result as data it did have — an identity problem, not a capability one. Label the observation clearly and the model knows exactly what it's holding.

The team learned something about where to spend effort, too. They'd been ready to pay for a more expensive model — real money, every call, forever — to fix what turned out to be a formatting bug fixed once for free. Observation formatting is one of the highest-return, lowest-cost improvements in agent building precisely because it's invisible: nobody profiles it, so it stays broken while people reach for expensive fixes to a cheap problem.

⚡ Pro tip: When an agent underperforms, print the exact context the model sees for one failing turn before you consider a bigger model. Half the time the fix is right there in the printout — a mangled observation, an unlabeled result, a truncated field — and it costs nothing to fix.

⚡ Pro tip: Prefix every observation with an explicit tag naming the tool and result —

[TOOL RESULT: get_balance]

How to Apply This to Your Situation

Audit your own observations by printing exactly what the model receives after each tool call. Not what the tool returns — what lands in the conversation. You'll frequently find raw dicts, truncated JSON, stack traces dressed as data, or results with no label at all. Each is a reason for the model to ignore or misread the result.

Then format for a reader, not a machine. Label the source. Surface the two or three fields that matter and drop the rest. State freshness or authority if it's relevant. And convert errors into readable observations too —

[TOOL ERROR: get_balance] rate limited, retry shortly

Apply one convention across every tool, not per-tool improvisation. If

get_balance

get_transactions

⚡ Pro tip: Put units, currency, and timestamps inside the observation text, not just in the raw data. A number without its unit is an invitation to guess; "$4,021.55 USD, as of 2 minutes ago" leaves nothing to infer. The model answers from what it can read, so place the disambiguating context where it actually reads.

⚡ Pro tip: Keep observations short and structured. If a tool returns fifty fields, format the three the agent needs and stash the rest behind a follow-up tool the model can call if it wants them. A lean observation step keeps the context window healthy and keeps the model focused on what matters.

Next Steps

Take one agent and rewrite a single tool's observation formatting: add a label, format the key fields, drop the noise, frame authority. Run the same ten questions before and after. The improvement is usually large enough to see immediately, and it costs nothing in model spend. If you only fix one thing in an underperforming agent this week, make it the observation step — it's the highest ratio of impact to effort in the whole loop.

The observation-formatting helpers become reusable across every tool and every agent. I keep a small library of them saved in PromptABCD — the label convention, the field-scoping pattern, the error format — so a new agent's observation step starts legible by default, instead of dumping raw dicts and looking broken until someone reads the transcript.