Instrumenting a Harness With OpenTelemetry

How do you debug an agent run that touched four services, made twelve model calls, and failed somewhere in the middle — when all you have is a pile of disconnected log lines? That was the exact question a platform team faced after their agent moved from prototype to production, and their answer was to instrument the whole thing with agent harness opentelemetry. This is the story of what they built, why tracing fit agent runs so naturally, and the specific decisions that turned an undebuggable black box into a run they could inspect span by span.

The insight that unlocked it: an agent run is already shaped like a trace. A run contains steps; each step contains a model call and some tool calls; tool calls sometimes call other services. That's a tree of nested operations with timing — which is precisely what distributed tracing was built to represent. They weren't forcing a fit; they were naming a structure that was already there.

The Problem This Team Faced

Their agent worked, mostly. But when a run went wrong, debugging was archaeology. Logs from the harness, the tool services, and the downstream APIs landed in different places with no shared thread connecting them. Reconstructing "what did run X actually do, in what order, and where did it slow down or fail" took an engineer an hour of grep and guesswork per incident.

The specific pain was correlation and timing. They could see that a run was slow but not which step. They could see a tool failed but not what the agent did before that led there. Every question required manually stitching timestamps across systems, and the stitching was error-prone.

⚠️ Common mistake: Debugging agents with unstructured, uncorrelated logs and assuming more logging is the fix. Adding log lines to a system with no correlation just gives you more disconnected lines to stitch. The problem isn't log volume; it's the missing structure that says "these events all belong to this run, in this order, nested this way."

The Wrong Approach

Their first attempt was to add a shared request ID to every log line and grep by it. Better than nothing, and still painful.

log.info(,[object Object],, extra={,[object Object],: run_id, ,[object Object],: step})
log.info(,[object Object],, extra={,[object Object],: run_id, ,[object Object],: name})

What this does: Stamps a run ID on log lines so you can filter to one run. It gives you correlation but not structure — you get a flat, time-ordered list of events with no sense of which tool call belonged to which step, how long each took, or how they nested. Reconstructing the tree from flat logs is still manual, and timing analysis means subtracting timestamps by hand.

The flat-log approach also couldn't follow the run across services. When the agent called a tool that called another service, the run ID didn't propagate, so the downstream service's logs were an island. The very cases that were hardest to debug — failures deep in a call chain — were exactly the ones the run ID couldn't reach.

The Correct Approach

They modeled the run as a trace: one root span for the whole run, child spans for each step, and grandchild spans for each model call and tool call. OpenTelemetry propagated the trace context across services automatically, so a tool that called another service extended the same trace.

[object Object], opentelemetry ,[object Object], trace
tracer = trace.get_tracer(,[object Object],)

,[object Object], ,[object Object],(,[object Object],):
    ,[object Object], tracer.start_as_current_span(,[object Object],) ,[object Object], run_span:
        run_span.set_attribute(,[object Object],, run_id)
        ,[object Object], ,[object Object], done:
            ,[object Object], tracer.start_as_current_span(,[object Object],) ,[object Object], step_span:
                step_span.set_attribute(,[object Object],, step)
                ,[object Object], tracer.start_as_current_span(,[object Object],) ,[object Object], llm_span:
                    resp = model.call(messages)
                    llm_span.set_attribute(,[object Object],, resp.usage.total)
                ,[object Object], call ,[object Object], resp.tool_calls:
                    ,[object Object], tracer.start_as_current_span(,[object Object],) ,[object Object], t_span:
                        t_span.set_attribute(,[object Object],, call.name)
                        dispatch(call)

What this does: Wraps the run, each step, each model call, and each tool call in nested spans, tagging them with attributes like token count and tool name. The result is a trace tree that shows exactly what happened, in order, with timing at every level. "Which step was slow?" becomes obvious from the span durations; "what did the agent do before the failure?" is the sequence of sibling spans right there in the tree.

Because they used OpenTelemetry's context propagation, a tool call that hit another service carried the trace context along, and that service's spans joined the same trace.

[object Object], ,[object Object],(,[object Object],):
    ,[object Object], tracer.start_as_current_span(,[object Object],) ,[object Object], span:
        headers = {}
        inject(headers)                 ,[object Object],
        span.set_attribute(,[object Object],, url)
        ,[object Object], requests.get(url, headers=headers)

What this does: Injects the current trace context into outgoing request headers so the downstream service continues the same trace instead of starting its own. Now a run that spans four services shows up as one connected tree, and the failure deep in the call chain is visible in context — the island problem is gone.

Results and What Changed

Debugging time per incident dropped from an hour to minutes. An engineer opens the trace for the failed run and sees it: the step that was slow is the long span, the tool that failed is the red span, the sequence that led there is the spans before it. No stitching, no grep — the structure is the answer.

The unexpected win was performance visibility. With agent harness opentelemetry in place, they could see aggregate timing across all runs: model calls were 70% of latency, one tool was a consistent outlier, a particular step type always ran long. These were invisible in flat logs and obvious in trace analytics, and they drove the next round of optimization.

⚡ Pro tip: Follow the emerging semantic conventions for LLM spans — standard attribute names for model, token counts, tool names — rather than inventing your own. When your attributes match the conventions, observability tools render agent traces with purpose-built views (token cost per span, model breakdowns) for free, instead of showing generic spans you have to interpret.

How to Apply This to Your Situation

Start with three span levels: the run, the step, and the individual model and tool calls. That structure alone transforms debuggability, and it maps directly onto the loop you already have — you're wrapping code you've already written, not restructuring it.

Tag spans with the attributes you'll actually query: run ID, step number, model, token counts, tool names, and a status on failures. Resist tagging everything; a few well-chosen attributes you filter and aggregate on beat a hundred you never look at.

Propagate context across every service boundary so multi-service runs stay one trace — this is the piece that pays off most on the hardest incidents. And link your traces to your audit log by carrying the trace ID as an audit field, so a compliance question and a performance question can both start from the same run and cross-reference each other.

⚡ Pro tip: Set span status to error and record the exception on any failed step or tool call, so failed runs are visually distinct and searchable. "Show me all runs with an error span in the last hour" becomes a one-click query, and each result opens straight to the span that failed — turning your traces into an incident dashboard, not just a debugging tool.

Keeping Tracing Affordable at Scale

The one worry teams raise about agent harness opentelemetry is cost — a busy agent produces a lot of spans, and storing every span from every run gets expensive fast. The answer isn't to trace less; it's to sample intelligently, keeping the traces that matter and dropping the ones that don't.

Tail-based sampling is the right tool here. Instead of deciding whether to keep a trace when it starts (when you don't yet know if it's interesting), you decide when it finishes, once you know the outcome. Keep 100% of traces that errored or ran slow, and a small percentage of the fast, successful ones. You pay for the traces you'd actually open and almost nothing for the boring majority.

[object Object], ,[object Object],(,[object Object],):
    ,[object Object], trace.has_error ,[object Object], trace.duration_s > SLOW_THRESHOLD:
        ,[object Object], ,[object Object],                     ,[object Object],
    ,[object Object], random.random() < ,[object Object],       ,[object Object],

What this does: Retains every failed or slow trace — the ones you'll investigate — while keeping only a sample of the healthy majority. Your storage bill tracks the number of interesting runs rather than total runs, which is what you want, because a thousand identical successful traces teach you nothing that the first few didn't.

The other cost lever is attribute discipline. Every attribute on every span is stored, so a span tagged with the full model response is far heavier than one tagged with a token count and a content hash. Put the heavy content in your audit log (which you're keeping deliberately) and keep spans lean with the metadata you filter and aggregate on. Traces are for shape and timing; the audit log is for content.

⚠️ Common mistake: Putting full prompts and model responses as span attributes "so it's all in one place." That bloats every trace, can leak sensitive data into your observability vendor, and duplicates what your audit log already holds. Reference the audit entry from the span instead — a run ID or trace ID link — and keep the bulky text in the store built for it.

Next Steps

Tracing tells you what happened and when; the natural next step is deterministic replay, which lets you re-run exactly what happened to debug why. Traces point you at the failing run; replay lets you step through it. Together they cover observation and reproduction — the trace shows you the failure exists and where it lives, and replay lets you reproduce it on demand until you understand it.

Keep your span conventions and instrumentation helpers versioned alongside your prompts in a library like PromptABCD, so every agent you build emits the same trace shape and your observability tooling works across all of them without per-service reinvention.