AI Agent Observability: What to Log and Why

How do you debug an agent that failed twenty minutes ago, for one specific user, in a way you can't reproduce no matter how many times you retry it? For most teams the honest answer is: you don't. You apologize, you guess, and you hope it doesn't happen again. That's not a debugging strategy — it's a prayer, and it's what life looks like without AI agent observability.

Observability is the practice of logging enough about what an agent did that you can reconstruct any run after the fact. Here's how one platform team went from flying blind to seeing every decision their agent made.

The Problem the Platform Engineer Faced

A platform engineer at a mid-size SaaS company owned an agent that answered account questions by calling internal APIs. It worked most of the time. But "most of the time" left a steady trickle of angry tickets: the agent gave a wrong answer, or timed out, or contradicted itself — and every single time, the engineer couldn't reproduce it.

The logs were the problem. They captured the final response and nothing else. When a customer said "your bot told me my plan was cancelled," the engineer had the bot's reply and no idea how it got there. Which API did it call? Did the call fail? Did it retry? Did it misread the result? The log said "response sent" and stopped there.

Every incident became a multi-hour archaeology dig with no artifacts. The engineer was debugging a black box by shaking it and listening.

The Wrong Approach to AI Agent Observability

The original logging looked responsible from a distance. It logged requests and responses, like any web service.

[object Object], ,[object Object],(,[object Object],):
    logger.info(,[object Object],)
    response = agent.run(request.text)
    logger.info(,[object Object],)
    ,[object Object], response

What this does: it records the user's input and the agent's final output — which is enough to observe a stateless web endpoint but blind to everything an agent does in between.

The gap is everything that matters. An agent's failures live in the middle — the tool call that returned stale data, the reasoning step that misread it, the retry that silently doubled a charge. Logging only the endpoints of a multi-step process is like a flight recorder that captures takeoff and the crash but nothing in between. The interesting part is exactly what's missing.

⚠️ Common mistake: Logging an agent like a request-response service. An agent is a process, not a function call. If your logs can't tell you the sequence of steps it took, they can't tell you why it failed, and you're back to guessing.

The Correct Approach: Trace Every Step

The fix was to treat each run as a trace — a structured record of every step, tied to one run ID, that you can replay end to end.

[object Object], ,[object Object],(,[object Object],):
    trace = Trace(run_id=uuid4(), user=request.user)
    ,[object Object], step ,[object Object], agent.run_steps(request.text):
        trace.log(
            step_type=step.,[object Object],,        ,[object Object],
            ,[object Object],=step.,[object Object],,
            output=step.output,
            tool=step.tool_name,
            latency_ms=step.latency,
            tokens=step.tokens,
            error=step.error,
        )
    trace.finalize(response=agent.answer)
    ,[object Object], agent.answer

What this does: it records every model call, tool call, and decision as a linked, timestamped step under one run ID — so any run can be pulled up and read like a story instead of guessed at from its ending.

Now when a ticket came in, the engineer pulled the run ID and read the whole sequence: the agent called the billing API, got a timeout, retried, got stale cached data showing a cancelled plan, and reported it as fact. The bug was visible in seconds. It had always been there — it just hadn't been recorded.

Results and What Changed

The change paid off immediately and then kept paying off. Mean time to diagnose an incident dropped from hours to minutes, because every incident now came with a complete record instead of a shrug. The stale-cache bug that had generated tickets for weeks was found and fixed in one afternoon once it was finally visible.

The traces also surfaced problems nobody had reported. Reading through them, the team found the agent was making a redundant API call on 40% of runs, quietly inflating both cost and latency. No customer had complained about it — it just made everything a little slower and more expensive, invisibly, until observability made it visible.

And the traces became the team's test set. Every real failure, now fully recorded, turned into a permanent regression case. Observability didn't just help them fix bugs faster; it fed directly into stopping those bugs from coming back.

⚡ Pro tip: Log a stable run ID and surface it to users in error messages. When a customer can paste "run 8f3a-..." into a ticket, you jump straight to the exact failed trajectory instead of hunting through logs by timestamp and hoping. It turns a vague complaint into a precise lookup.

How to Apply This AI Agent Observability Setup to Your Situation

You can start today without any special platform.

Log at the step level, not the request level. Every model call, tool call, and branch decision gets a record, all linked by one run ID. This single change is what separates a debuggable agent from a black box, and it costs you a logging call per step.

Capture the boring fields that turn out to matter — latency and token count per step, the tool name, and any error. When you're chasing a slow or expensive run later, these are exactly the fields you'll wish you'd recorded, and they're nearly free to log now.

Make traces easy to read, not just easy to store. A wall of JSON nobody opens is not observability. A data scientist debugging a research agent and an SRE debugging a payments agent both need to skim a run in seconds, so invest a little in a readable view of the trajectory.

Link traces to the things around them, not just to themselves. A trace tied to a user session, a request ID, and the deployed version tells you not only what the agent did but the context it did it in — which release introduced the regression, whether one user hit it repeatedly, whether it clusters around a particular input shape. An isolated trace answers "what happened in this run." A connected trace answers "is this a pattern," and patterns are what you actually fix. A platform team chasing an intermittent failure found their answer not in any single trace but in noticing that every failing run shared one deployed prompt version, a correlation invisible until the traces were joined to release metadata.

⚡ Pro tip: Sample full verbose traces rather than storing everything at maximum detail forever. Keep lightweight traces for every run and rich, full-payload traces for a sample plus every error. You get the debuggability where it counts without a storage bill that grows faster than your traffic.

⚡ Pro tip: Alert on trajectory shape, not just errors. A run that suddenly takes twelve steps when it normally takes four hasn't thrown an error, but something is wrong. Watching step count, tool-call count, and latency distributions catches degradation long before it becomes an outage anyone files a ticket about.

There's a discipline question hiding inside observability that's easy to get wrong: what not to log. Traces are gold for debugging and a liability if they capture raw sensitive data. An agent handling medical or financial questions will, by default, write patient details and account numbers straight into its trace, which turns your debugging system into a second copy of the data you're most obligated to protect. The fix is to redact at the logging boundary — hash or mask identifiers before they hit the trace, keeping the shape of the run without the sensitive payload. A fintech team learned this the expensive way when a routine log export turned out to contain full transaction records nobody had meant to persist.

The other half of the discipline is retention. Full traces are worth keeping for days, not years. Set a retention window that matches how far back you actually debug, and let older verbose traces expire. Observability that grows without bound stops being a tool and becomes a cost center, and the ninety-day-old trace you never opened isn't earning its storage.

⚡ Pro tip: Redact sensitive fields before they're written, not after. Post-hoc scrubbing always misses something, and "we'll clean the logs later" is how sensitive data ends up in six downstream systems. Mask at the source, where there's exactly one code path to get right.

Next Steps

Add step-level tracing to one agent this week and pull up a trace the next time something breaks. The first time you diagnose a "can't reproduce" bug in two minutes, you'll never ship an agent without observability again. The instinct that once said "add logging later" flips permanently to "no agent goes to production blind," because you've felt the difference between reading a failure and guessing at one.

As your trace schema and the queries you run against it mature, keep them shared. Teams that store their observability conventions and analysis prompts in a common library like PromptABCD roll consistent logging onto every new agent instead of reinventing it each time. Observability you design once and reuse everywhere is how a fleet of agents stays debuggable as it grows, instead of getting harder to understand with every new agent you add to it.