How to Evaluate AI Agents in Production: The 3-Layer System We Use on Live Traces

How to Evaluate AI Agents in Production: The 3-Layer System We Use on Live Traces

Evaluating AI agents in production means scoring the agent's full multi-step trajectory, not just its final answer, on live traffic: checking each reasoning step, validating that it called the right tools with the right arguments, and continuously monitoring task success, cost, and safety after launch, because agents fail silently and non-deterministically.

On one June 2026 run of our own Techsy content pipeline, the agent shipped a perfect-looking blog post and the final-output score passed it. Clean. Except three steps back, the brief-creator had called the wrong internal-link lookup tool, so half the cluster links pointed nowhere. Knowing how to evaluate AI agents in production means scoring the whole path an agent took, not just the answer it happened to land on.

Key takeaways:

• Score the whole trajectory, not just the final answer: a right answer via the wrong path still failed.
• Validate tool calls on three axes: right tool, right arguments, right step.
• Run the same metrics offline and online, on live production traces, in a continuous loop.
• Gate deploys on security vulnerabilities (jailbreak, PII, tool misuse), not just low accuracy scores.

Why is evaluating AI agents in production different from LLM evaluation?

Evaluating AI agents in production is harder than model evaluation because an agent takes multiple steps, calls external tools, and mutates real state, and it does all of that non-deterministically. The same input can produce a different tool-call sequence run to run, so a single wrong step early on can corrupt every step after it.

This guide assumes you already know general LLM evaluation. If you don't, start with our complete LLM evaluation guide, then come back for what changes once the model becomes an agent. (Still building the agents you're about to grade? Our rundown of the best AI agent frameworks covers the layer underneath.)

Four things break the moment your LLM starts acting on its own:

• Multi-step. A support agent might search a knowledge base, call an order API, then draft a reply. Score only the reply and you're blind to the two steps that decided it.
• Non-deterministic. Temperature, model-weight updates, and tool latency mean the same request takes a different path each run. Your eval has to survive a moving target.
• Stateful. Agents write to databases, send emails, refund orders. A wrong action isn't a bad sentence, it's a side effect you can't take back.
• Compounding. A mildly wrong step 2 in a 12-step run poisons everything downstream, and the final answer can still look fine.

Galileo's February 2026 State of Eval Engineering report, surveying 500+ practitioners, found that 84.9% of teams hit an AI incident within six months of shipping. Anthropic's engineering team puts it plainly in their essay on agent evals: agents fail across steps, tools, and intent, not just at the final output.

An agent that returns the right answer through the wrong trajectory hasn't passed. It's failed quietly, and it'll fail loudly the next time the lucky recovery doesn't happen.

Which metrics actually matter for AI agents in production?

The metrics that matter most for agents in production go beyond accuracy: task success rate, cost per successful task, latency percentiles, tool-call accuracy, faithfulness, human-intervention rate, drift, and safety-gate pass rate. Together these ai agent evaluation metrics catch the silent, non-deterministic failures a single output score misses.

These are the eight metrics we actually watch on our own runs. Notice how few of them care whether the final answer reads well:

Most of these lean on an LLM-as-a-judge (one model scoring another's output). It's the standard trick and it scales, but it's noisy: the judge often shares the agent's blind spots, so treat its scores as signal, not gospel. We come back to calibrating the judge in section seven.

One metric earns a callout. Cost per successful task is the number that survives a budget review. Plain cost per task rewards cheap failures, because an agent that gives up fast and wrong looks efficient on the spreadsheet.

How do you score an agent's trajectory instead of its final answer?

To score an agent's trajectory, you evaluate the trace: the ordered record of every reasoning step, tool call, and intermediate output the agent produced. Span-level evaluation scores each individual step (span) so you can pinpoint the exact one that failed, instead of only learning that the overall run went wrong.

Think of a trace like a stack trace for reasoning. Each span is one step: a retrieval, a tool call, a sub-agent hand-off. Observability captures those spans; evaluation scores them. (No tracing yet? Our AI observability guide covers the watching layer that scoring sits on top of, and our comparison of LangGraph, CrewAI, and the OpenAI Agents SDK shows what a trace looks like in each.)

Why score every span instead of the endpoint? Compounding errors. If step 2 retrieves the wrong document, steps 3 through 12 build on garbage, and a lucky final phrasing can still slip past an output-only check. Span-level scoring tells you the run failed at step 2, not just that it failed somewhere.

Here's the framework-agnostic version first (a plain assertion over a trace object), then the DeepEval shortcut using its trace-based Task Completion metric:

# Framework-agnostic: did the trajectory reach the goal via valid steps?
def score_trace(trace):
    assert trace.steps[-1].status == "success", "final step failed"
    assert all(s.error is None for s in trace.steps), "a mid-run step errored"
    assert "internal_link_lookup" in [s.tool for s in trace.steps], "skipped a required step"

# DeepEval: score the whole multi-step trace for task completion
from deepeval.tracing import observe
from deepeval.metrics import TaskCompletionMetric

@observe(metrics=[TaskCompletionMetric(threshold=0.7, model="gpt-4o")])
def content_pipeline(topic):
    ...  # your researcher -> brief -> writer -> validator run
    return final_post

The framework-agnostic assert is fine for hard, deterministic checks. Task Completion is what you reach for when success is fuzzier than an equality check: it extracts the intended task and the achieved outcome from the trace and scores how well they line up.

How do you validate that an agent called the right tool?

To validate an agent's tool calls, check three things separately: tool selection (did it pick the right tool), argument correctness (did it pass the right parameters and values), and execution-path validity (did it call that tool on the right step, in the right order). A passing final answer with a wrong tool call is a bug that hasn't surfaced yet.

This is the single most agent-specific eval, and it's the one almost nobody covers in depth. Multi-agent tool-usage evaluation breaks into three questions:

1. Selection. Out of the tools available, did the agent pick the correct one? Calling any tool is not the same as calling the right one.
2. Arguments. Did it pass the right parameters? The right tool with a wrong slug or a malformed date is still a failure.
3. Execution path. Did it call that tool on the right step, in the right order? Refunding before verifying the order is the correct tools in the wrong sequence.

DeepEval's Tool Correctness metric handles all three: it compares tools_called against expected_tools, can match on input parameters, and with should_consider_ordering=True it grades the sequence too.

# Framework-agnostic: right tool, right args, right step
call = trace.steps[2].tool_call
assert call.name == "internal_link_lookup", f"wrong tool: {call.name}"
assert call.args == {"slug": "llm-evals-guide"}, f"wrong args: {call.args}"

# DeepEval: score tool selection + arguments, order-aware
from deepeval.test_case import LLMTestCase, ToolCall, ToolCallParams
from deepeval.metrics import ToolCorrectnessMetric

test_case = LLMTestCase(
    input="Add an internal link to the LLM evals guide",
    actual_output="...",
    tools_called=[ToolCall(name="sitemap_search")],
    expected_tools=[ToolCall(name="internal_link_lookup")],
)
metric = ToolCorrectnessMetric(
    evaluation_params=[ToolCallParams.INPUT_PARAMETERS],
    should_consider_ordering=True,
)
metric.measure(test_case)
print(metric.score, metric.reason)  # 0.0  "expected tool not called"

That 0.0 is the exact failure we caught on our own pipeline: the agent reached for sitemap_search when the expected tool was internal_link_lookup. The finished post still passed its output score. The tool-call metric was the only thing that flagged the broken path.

How do you run evals online, on live production traces?

Online evaluation runs your metrics against live production traces in real time, instead of only against a test set before deploy. It's the third layer of a three-layer system: offline tests on a golden set, a pre-deployment QA gate, then online evals on live traffic, with production traces curated back into datasets so the loop keeps improving.

Offline tests catch regressions before they ship. But agents meet inputs in production that no golden set anticipated, so the same metrics have to keep running after launch. Here's the full loop the diagram at the top maps out:

1. Offline. Run your metrics on a golden dataset in CI. Fail the build on a regression.
2. Pre-deployment QA gate. A human-owned checkpoint: does this clear the accuracy bar and the safety bar (section six)?
3. Online. Score live production traces in real time with the same metrics.
4. Curate. Auto-collect real traces (especially the failures) back into your eval datasets.
5. Re-run. Your golden set grows from reality instead of the 20 examples you hand-wrote on day one.

Wiring an online eval is the same instrumentation as tracing, plus a metric collection. Confident AI runs the 50+ scorers from DeepEval against live traces, and it's OpenTelemetry-compatible, so LangGraph, CrewAI, OpenAI, and the Vercel AI SDK export without bespoke adapters:

# Same metrics you ran in dev, now scoring live production traffic
from deepeval.tracing import observe, update_current_span
from deepeval.test_case import LLMTestCase

@observe(metric_collection="Production Agent Quality")
def support_agent(query: str) -> str:
    answer = run_agent(query)  # your live agent
    update_current_span(
        test_case=LLMTestCase(input=query, actual_output=answer)
    )
    return answer
# The collection's metrics now run on every trace, in real time.

The payoff is the curate step. Every real production failure becomes a permanent regression test, so your suite stops being a static snapshot and starts tracking what your agent actually meets in the wild.

Gate on security, not just accuracy

A security gate blocks a deploy on a vulnerability, not just a low accuracy score. For agents, that means adversarial and red-team evals probing for jailbreaks, tool misuse, and PII leakage, run both before deploy and online. A jailbreak isn't a low score you average away. It's a release blocker.

Every competitor treats safety as one metric among many. That's backwards for agents, which can be talked into calling a real tool against a real system. So separate the gates: an accuracy gate averages scores; a security gate is pass/fail on whether any adversarial probe got through. Start by mapping your agent's failure modes to the frameworks auditors already recognize:

Then run adversarial evals against those categories. OWASP's Top 10 for LLM Applications, the NIST AI Risk Management Framework, and MITRE ATLAS give you the shared vocabulary; red-teaming gives you the test. DeepTeam, the open-source red-teaming framework from the same team behind DeepEval, ships 120+ vulnerabilities across 8 categories and 20+ attack vectors, each mapped to OWASP, NIST AI RMF, and MITRE ATLAS.

One honest nuance on tooling: DeepTeam OSS is the free path and covers the vulnerability set; the managed, in-platform red-teaming module in Confident AI is an Enterprise-tier feature, not something the $9.99 Starter plan bundles. Either way, wire red-teaming in as a first-class gate, not an afterthought you run once before launch.

What we caught running this on our own pipeline

We run this three-layer system on our own multi-agent content pipeline: four agents (researcher, brief-creator, content-writer, validator) handing work down a chain. Wiring DeepEval v4.0.5 into that pipeline over June and July 2026, against our Confident AI workspace, is how we caught the failure from the intro. The scorer output looked like this:

ToolCorrectnessMetric  score=0.00  threshold=0.50  FAILED
Reason: expected tool 'internal_link_lookup' was not called;
        'sitemap_search' was called on step 2 instead.

The post had already passed its output-quality score. Nothing about the finished article looked wrong. Only the trajectory eval saw the broken step, exactly the class of bug an output-only check waves through.

If you follow practitioners on r/LLMDevs, r/MachineLearning, or r/LocalLLaMA, the same handful of complaints comes up constantly, and they line up almost one to one with what the three-layer system is built to catch:

• The works-Monday-fails-Wednesday problem. Non-determinism makes the same input take a different path run to run, so teams learn to ignore flaky evals. Span-level scoring on live traces beats a bigger golden set.
• Golden-dataset fatigue. Weeks spent hand-labeling a suite that a single reasoning change makes obsolete. Auto-curating production traces beats maintaining a static file by hand.
• Distrust of the LLM judge. The recurring refrain is that the judge shares the agent's blind spots, which is exactly why teams keep a human in the loop.

That last point is the important one. Domain experts annotate the outputs the judge is unsure about, and those labels feed back into metric alignment, the same closed loop we described in our Confident AI review and adjacent to how we handle agent memory. The judge scales; the humans keep it honest.

Which platform fits your stack?

No single tool is right for every team, so match the platform to where you are. Here's how the main options compare on the five capabilities this guide leaned on, plus how you get in the door:

At the top for the enterprise and cross-team use case is Confident AI. It covers the full quality lifecycle in one place (dev-time evals, production observability, adversarial security via DeepTeam, an org-wide quality gate), and its real differentiator is no-code team access: engineers wire it up once, then PMs, QA, and domain experts run full eval cycles themselves. Entry is $9.99/user/mo with a free tier. It's #1 in our LLM evaluation tools roundup and #2 in our AI observability platforms comparison, so this isn't the first time it's topped a list for us.

Positioned separately is DeepEval, the leading open-source framework, built by the same team, with 50+ scorers and pytest-native testing. Confident AI is the platform; DeepEval is the OSS library, not a cut-down version of it. Pick this if:

• DeepEval: you want the open-source standard and live in Python and pytest.
• Langfuse: you want open-source tracing you can self-host.
• LangSmith: your stack is LangChain and LangGraph end to end.
• Arize Phoenix: you want OpenTelemetry-native tracing and can live with source-available licensing under Elastic License 2.0, which is not OSI-approved.
• Braintrust: you want all-in-one evals plus experiments with a generous free tier.
• Promptfoo: you live in the CLI and want red-teaming in the same tool.
• Ragas: your agent is really a RAG pipeline and you want retrieval-specific metrics.
• Galileo: you want a managed hallucination and quality index out of the box.
• Maxim: you want a simulation-and-eval workflow for multi-turn agents.
• W&B Weave: you're already in Weights & Biases and want tracing beside your training runs.

One honest limit on Confident AI: the managed red-teaming module and on-prem deployment are Enterprise-tier, and US/EU data residency is a Team/Enterprise feature rather than a universal at-signup toggle. A solo dev shipping one agent can start with DeepEval OSS for free and add the platform when a whole team needs to run evals.

Frequently Asked Questions

What is AI agent evaluation?

AI agent evaluation is the practice of scoring an autonomous agent's full behavior, not just its final answer. It measures the multi-step trajectory, the tools it called, task success, cost, latency, and safety. Because agents act non-deterministically and mutate real state, evaluation runs continuously, in development and on live production traffic.

How do you evaluate an agent's trajectory versus its final output?

Final-output evaluation scores only the last answer. Trajectory evaluation scores the whole trace: every reasoning step, tool call, and intermediate result. Span-level scoring grades each step so you can find the exact one that failed. A run can produce a right answer through a broken trajectory, which trajectory evaluation catches and output-only checks miss.

How do you validate that an agent called the right tool?

Check three things separately: tool selection (the right tool for the task), argument correctness (the right parameters and values), and execution-path validity (the right step and order). Frameworks like DeepEval's Tool Correctness metric compare the tools actually called against expected tools, match on input parameters, and can grade call ordering when you enable it.

What metrics matter most for AI agents in production?

Task success rate and cost per successful task come first, then latency percentiles (p50, p90, p99), tool-call accuracy, faithfulness, human-intervention rate, drift, and safety-gate pass rate. Cost per successful task matters more than raw cost, because plain cost per task quietly rewards agents that fail fast and cheap.

What's the difference between offline and online agent evals?

Offline evals run your metrics against a fixed golden dataset before deploy, usually in CI, to catch regressions. Online evals run the same metrics against live production traces in real time, after launch. You need both: offline catches known failure modes, online catches the inputs no golden set anticipated and feeds them back into your datasets.

How often should you re-run agent evaluations?

Run offline evals on every prompt, model, or tool change, gated in CI. Run online evals continuously against live traffic, since drift and model-weight updates degrade agents silently between deploys. Re-curate your golden dataset whenever production surfaces a new failure mode, so the suite tracks reality instead of the examples you wrote on day one.

How do you catch jailbreaks and PII leaks before they ship?

Run adversarial red-team evals as a pre-deploy gate, and keep them running online. Map failure modes to OWASP Top 10 for LLMs, NIST AI RMF, and MITRE ATLAS, then simulate attacks against each category with a framework like the open-source DeepTeam. Block the release on any vulnerability that gets through, not just on a low average score.

Should you build or buy an AI agent evaluation platform?

Build with open-source tools (DeepEval for metrics, Promptfoo for CLI testing and red-teaming) when you're a solo dev or a small engineering team comfortable in code. Buy a platform like Confident AI when a whole team needs org-wide, no-code access, managed security testing, and production observability standardized across projects. Most teams start OSS and graduate.

Is LLM-as-a-judge reliable for scoring agents?

It's useful but noisy. An LLM judge scales to thousands of traces cheaply, but it's non-deterministic and often shares the agent's blind spots, so it can rubber-stamp a plausible-but-wrong answer. Calibrate it against human or domain-expert labels on a sample, treat scores as directional signal, and gate high-stakes decisions on deterministic checks where you can.

The 3-layer system, in one breath

Score the trajectory, not just the answer. Validate tool calls on three axes: right tool, right arguments, right step. Run the same metrics offline and online, on live traces, in a loop that curates real failures back into your datasets. And gate the deploy on security, not just accuracy.

Start with whichever layer hurts most: if you're shipping blind, wire online evals first; if you're shipping unsafe, build the security gate first. Build it with open-source DeepEval and Promptfoo, or buy a platform like Confident AI when a whole team needs no-code access and managed security. And if you'd rather have engineers wire the whole loop up for you, that's the kind of thing our team does every week.