How to Evaluate an AI Agent

Most AI agent evaluation guides are wrong about what to measure first. They tell you to grade the final answer — thumbs up or down, right or wrong — and move on. That works for a chatbot. For an agent that takes ten steps to get somewhere, final-answer scoring hides exactly the failures you most need to see. An agent can reach the right answer through a broken, expensive, un-reproducible path, and answer-only scoring calls that a win.

Here's how one team learned that the hard way, and what their AI agent evaluation looked like once they fixed it.

The Problem the Ops Team Faced

A mid-size logistics company built an agent to answer questions about shipments — where's order 4021, why is it delayed, what's the ETA. It pulled from three internal APIs and wrote a summary. In testing, it scored 92% on a set of question-answer pairs. Leadership greenlit it.

Two weeks into production, support tickets climbed instead of falling. The agent's answers were usually right, so what was going wrong? Customers were getting correct ETAs — eventually — after the agent made four redundant API calls, occasionally timed out and retried, and sometimes contradicted itself mid-answer before landing on the truth. The final answers looked fine in the eval. The path to them was a mess, and the path was what customers actually experienced.

Their 92% was real and useless. It measured the destination and ignored the journey.

The Wrong Approach to AI Agent Evaluation

Their original eval was a single comparison: does the agent's final answer match the expected answer? Something like this.

[object Object], ,[object Object],(,[object Object],):
    correct = ,[object Object],
    ,[object Object], c ,[object Object], cases:
        answer = agent.run(c[,[object Object],])
        ,[object Object], c[,[object Object],].lower() ,[object Object], answer.lower():
            correct += ,[object Object],
    ,[object Object], correct / ,[object Object],(cases)

What this does: it runs each test question, checks whether the expected string appears in the final answer, and reports a pass rate — measuring only the endpoint, never how the agent got there.

This missed three whole categories of failure. It couldn't see that the agent called the same API four times, because cost and tool calls weren't measured. It couldn't see that the agent nearly failed and recovered by luck, because only the final string was checked. And it couldn't catch a right answer reached by wrong reasoning — a coincidence that would break the moment the data shifted.

⚠️ Common mistake: Treating a high final-answer score as proof an agent is production-ready. Final-answer accuracy tells you the agent can sometimes get there. It says nothing about whether it gets there reliably, cheaply, or the same way twice.

The Correct Approach: Evaluate the Trajectory

The fix was to evaluate the full trajectory — every step the agent took — not just the last line. They started recording each run as a structured trace and scoring it on several axes.

[object Object], ,[object Object],(,[object Object],):
    results = []
    ,[object Object], c ,[object Object], cases:
        trace = agent.run_traced(c[,[object Object],])   ,[object Object],
        results.append({
            ,[object Object],: c[,[object Object],].lower() ,[object Object], trace.answer.lower(),
            ,[object Object],: ,[object Object],(trace.tool_calls),
            ,[object Object],: count_duplicate_calls(trace.tool_calls),
            ,[object Object],: trace.had_error ,[object Object], trace.correct,
            ,[object Object],: ,[object Object],(trace.steps),
            ,[object Object],: trace.total_tokens,
        })
    ,[object Object], summarize(results)

What this does: it records the whole trajectory and scores accuracy alongside tool-call count, redundant calls, error recovery, step count, and cost — so a run that lands the right answer inefficiently no longer looks identical to a clean one.

Now the 92% split apart. Yes, 92% correct — but the average run made 3.1 API calls when 1.5 would do, 18% of runs recovered from an error the eval had been silently forgiving, and the worst 10% of runs cost four times the median. The single number had been hiding four separate problems.

Results and What Changed

With trajectory-level evaluation, the team could finally target the real issues. They added a simple cache to kill redundant calls, which dropped average tool calls from 3.1 to 1.6 and cut cost by roughly 40%. They turned the "recovered from error" cases into their next test set, because a lucky recovery today is an outage tomorrow. And they set a per-run cost ceiling as a hard eval gate, so a change that made answers slightly better but doubled cost would fail review.

The final-answer accuracy barely moved — it went from 92% to 93%. But the metric that mattered to customers, consistent and fast answers, improved dramatically. Median response time fell by half. The lesson stuck: the number that looks like quality and the number customers feel are often not the same number.

The team also found something they hadn't thought to measure at all — variance. Two identical questions could take three steps one time and seven the next, depending on which order the model happened to call things. Average metrics hid this completely; the mean looked stable while individual runs swung wildly. Once they started tracking the spread instead of only the average, they saw that the worst 10% of runs were driving nearly all the support tickets. Fixing the tail turned out to matter far more than nudging the average, which is the opposite of what a single accuracy number would have told them to do.

⚡ Pro tip: Build your eval set from production failures, not imagined ones. Every real incident becomes a permanent test case. Over a few months this turns your eval from a guess about what might break into a precise map of what actually has.

How to Apply This AI Agent Evaluation to Your Situation

You don't need a platform to start. You need three things.

First, trace everything. Before you can evaluate a trajectory, you have to record it — every model call, tool call, input, and output, tied to a run ID. If you're only logging final answers today, fix that first; it's the foundation everything else stands on. A trace doesn't need to be fancy — a JSON blob per run with the ordered steps, their inputs, their outputs, and a timestamp is enough to unlock every trajectory metric in this post. The teams that skip this step spend months arguing about quality with no shared evidence to point at.

Second, define the axes that matter for your task. A research agent cares about source quality and coverage. A transaction agent cares about correctness and never double-acting. A support agent cares about tone and resolution. Pick three or four axes, not fifteen, and make each one measurable.

Third, set gates, not just scores. A gate is a threshold a change must clear to ship: accuracy stays above X, cost stays below Y, no run exceeds Z steps. Gates turn evaluation from a report you glance at into a guardrail that stops regressions automatically.

Fourth, separate offline evaluation from online monitoring, because they answer different questions. Offline evals on a fixed test set tell you whether a change is safe to ship. Online monitoring on live traffic tells you whether reality still matches your test set — and it drifts constantly as users ask things you never imagined. A retail-analytics team learned that their offline scores held steady for months while live quality quietly slipped, because their test set had frozen in time while customer questions kept evolving. You need both: the test set to gate changes, the live stream to keep the test set honest.

It helps to picture how this lands across roles. A data scientist tuning a research agent watches source-quality and coverage trends per release. A platform engineer running a payments agent watches the double-action rate like a hawk, because one duplicate charge is worse than ten slightly-worse answers. A product manager on a support agent watches resolution rate and tone flags together, since a technically correct but cold reply still loses the customer. Same discipline, different axes — the trick is choosing the two or three that map to what actually hurts when they slip.

⚡ Pro tip: When a metric and your gut disagree, trust the trace before either one. Pull three or four real runs and read them end to end. Aggregate numbers compress away the exact moment things go wrong; a raw trajectory shows you the specific step where the agent took a wrong turn, which is where every real fix actually starts.

⚡ Pro tip: Score a slice of runs with a second model acting as judge, but always calibrate it against human labels first. An LLM judge that hasn't been checked against real human judgment on your task drifts confidently in the wrong direction and gives you false comfort.

⚡ Pro tip: Track your evaluation results over time, not just per release. A single snapshot tells you where you are. A trend line tells you whether last month's "small" prompt change quietly cost you three points of accuracy that nobody noticed at the time.

Next Steps

Start by adding tracing to one agent and re-scoring your existing eval set on trajectory, not just answers. You'll almost always find a gap between your headline number and reality — that gap is your roadmap.

Don't wait for a platform or a perfect harness. A spreadsheet of traced runs scored by hand on three axes beats a polished dashboard measuring the wrong thing. Start crude, start this week, and let the questions your evaluation can't yet answer tell you what to build next.

As your eval cases and judge prompts pile up, keep them somewhere your whole team can reach. Groups that store their evaluation prompts and criteria in a shared library like PromptABCD build on each other's work instead of everyone inventing their own quiet, slightly-wrong version of "good." A good evaluation is a shared asset — the better it gets, the more every agent you ship benefits from it.