How to evaluate AI agents: the four-layer eval stack, with code and a ship/no-ship checklist

Evaluating an AI agent means measuring three things an LLM eval never had to: the trajectory (which tools it called, in what order, with what arguments), the world state it left behind (did the refund exist, did the database change, did the file compile), and its reliability across repeated attempts at the same task. A chatbot eval grades a piece of text. An agent eval grades a distributed system with a language model in the control loop, where the same input can produce different actions and every attempt ships to a real user. The stack that handles this has four layers: graders (cheapest one that still discriminates), trajectory and state checks, reliability measured with pass^k rather than pass@1, and a qualification step that turns evidence into permission.

One number frames the whole discipline. An agent scores 94 percent. May it draft a reply? Read a customer record? Issue a refund? The score cannot say, because each action carries a different consequence and the missing six percent may contain every case that matters. A score does not grant authority; qualification does. This article walks the four layers with the numbers behind them, gives you a runnable eval harness in about sixty lines of Python, and ends with the ship/no-ship checklist we run before an agent is allowed to act.

In August, an internal audit of one of our own research drafts found the kind of result that should worry anyone who trusts an evaluation dashboard. The experiment had run 2,160 trials across several models. Its table said one model misbehaved on 8.8 percent of cases, which made it the second-safest model tested. But 421 trials, 19.5 percent of the run, had returned empty responses, all on reasoning models whose reasoning tokens had exhausted the output budget. The evaluator had silently counted every empty response as honest. On the valid trials, the real rate was 84 percent. The missing-data policy had inverted the ranking.

No smarter model would have caught that, and no new benchmark. The evaluator needed one rule: an absent or unparseable observation can never be evidence of safe behaviour. That is the central problem of agent evaluation. The system producing the score has its own state, software, blind spots and incentives. It can reject a correct agent, certify a dangerous one, reward a shortcut, or hide attrition in the denominator. Evaluation is not a stage after engineering. It is a second harness around the first one.

Why LLM evals do not transfer to agents

A production agent is a probabilistic distributed system with a language model at the decision point. Each word matters. Probabilistic: the same input yields different actions, so reliability is a distribution, not a property. Distributed: it calls tools, retrievers, services and other agents, with latency, partial failure and retries, so half of "agent quality" is plain reliability engineering. Model in the control loop: the non-determinism is not at the edges, it is in the choice of what to do next, which is what makes agents useful and what makes them hard to grade.

Experts do not begin with metrics. They begin with a model of how the thing fails. Before you write a single eval, list the ways this agent can go wrong and mark each one "annoying" or "unacceptable". That sort drives everything downstream.

Two consequences follow. There is never one accuracy number; there is a portfolio, reported per task class, because a refund and an FAQ answer are not the same product. And targets are set per class: an internal document QA agent might need 90 percent grounded with zero unsupported high-confidence claims, a CRM updater 99 percent correct field mapping with zero unauthorised writes, and a payment action 100 percent human approval before execution. One global number is how a team hides its real failures from itself.

Layer 1: graders, and the cost ladder

Anthropic's guidance names three grader types: code-based (fast, objective, brittle), model-based (flexible, nuanced, non-deterministic) and human (the gold standard, expensive). OpenAI's evals platform turns the same idea into a ladder, and the rule is to reach for the lowest rung that can still tell right from wrong.

The mechanical detail most teams miss: a tool-call accuracy check is a code grader over a structured trace, not a special agent-eval product. Did send_email get called with a recipient outside the thread? Did the amount argument equal the legitimate payee's? Those are assertions, and they are exact. You need far fewer model judges than you think. Most of what teams reach for a judge to do is a deterministic state check in disguise. Where you do use a judge, give it an "unknown" escape hatch so it cannot hallucinate a grade, and use one judge per rubric dimension rather than one judge scoring everything.

Layer 2: grade the world before the words

The most important verifier question is not "which judge model?" It is "where does truth leave a machine-checkable trace?" For an agent that acts, truth surfaces are everywhere if you look: the database state after the run, an immutable tool receipt, the authorisation decision at the capability boundary, executable tests over the produced artifact, a reconciliation read from the external system, a cited source and the exact claim it supports, and the delayed real outcome (a reopened ticket, a chargeback, an override). Use the strongest surface available for each claim. Generation may need broad, probabilistic intelligence; verification can usually be narrowed into a test, a state comparison or an independent lookup. Search for that asymmetry before you buy a bigger judge.

Anthropic is right to warn against requiring one exact tool sequence: agents find valid paths the test author never imagined, and a grader that insists on search-then-read-then-calculate rejects a correct read-then-calculate. But "grade outcomes, not paths" becomes dangerous as an absolute. A refund existing at the end is not success if identity verification was skipped. Correct code is not acceptable if the agent copied a hidden answer key. A harmless final answer does not erase an attempted exfiltration that the network happened to block. The distinction that holds: do not grade preferred implementation details when several paths are valid; do grade trajectory invariants when the path changes authority, safety, cost, privacy or recoverability. Final state grades completion. The trace grades required and forbidden transitions. And do not use the model's hidden reasoning as ground truth; tool calls, admitted observations, policy decisions and effects are the durable trajectory.

Layer 3: reliability, with pass^k and honest denominators

One successful run proves possibility. For a task attempted k times, two metrics answer opposite questions:

pass@k : at least one of k attempts succeeds
pass^k : all k attempts succeed

pass@k rewards optimism and is right when a human filters the candidates, as in code generation. It is wrong for an autonomous agent, where every attempt ships and nobody picks the winner. So Sierra's τ-bench inverted it. Their headline was not a score. It was that GPT-4o succeeded on under half the tasks and, run eight times on the same retail task, got it right all eight times less than a quarter of the time. If a task's per-attempt success rate is p, then pass^k is p to the power k, and the arithmetic is unforgiving:

Why this beats any capability number: averaging hides catastrophic variance. Two agents both report 70 percent. One fails the same 30 percent of tasks every time, which is predictable, so you route those to a human. The other fails a random 30 percent each run, a slot machine where every user rolls the dice. pass^k separates them. pass@1 cannot. Pick k from your real interaction volume and report pass^k, not pass@1, for anything that runs unattended.

Three habits keep the number honest. First, cluster by task: ten samples on each of thirty tasks are not three hundred independent trials, and Anthropic's own warning is that clustered standard errors can be more than three times the naive ones, so casual A/B comparisons declare false winners constantly. Second, respect what zero failures means. With zero failures in 100 independent trials, the one-sided 95 percent upper bound on the true failure rate is still about 3 percent; you need roughly 3,000 clean trials to push that bound under 0.1 percent, and agent trials are rarely independent. Statistics cannot justify a "never" claim at realistic sample sizes, so for catastrophic actions you pair testing with structural prevention: the agent has no permission, the policy engine rejects the call, a human must approve. Use tests to show the control fires; use the control to bound the consequence. Third, denominators are architecture. Every run ends in exactly one of: valid pass, valid fail, system error, grader error, timeout, missing output, invalid case, censored outcome. Never coerce the last six into pass or fail silently. That is the August audit in one rule.

A framing worth stealing from METR: tag every task with the time a competent human takes, plot success against task length, and read off your reliability horizon at the threshold your risk tolerance demands. "This agent handles tasks a human does in twenty minutes, at 95 percent" is a deployable sentence. "It scores 72" is not.

Layer 4: qualification, where evidence becomes permission

Four words that should not be synonyms:

A benchmark contributes measurements; it cannot qualify a deployment. A model judge contributes judgments; it cannot own the release decision. A human sign-off without declared evidence is accountability without instrumentation. The qualification layer adds what a generic eval API cannot know: authority, risk tolerance, sufficiency, ownership and consequence. In practice it is a promotion gate: this model-and-harness version may draft replies in shadow mode; it may read records for these two task classes; it may not touch refunds until pass^8 on the refund population clears the bar and the injection suite shows zero successful exfiltrations. Treat evaluation as a safety-critical decision system, not a number-producing utility. The full method, with the graders, the promotion records and the statistics, is what we teach in the Trust Engineering course; the short version is free in The Agent Eval Playbook.

The golden set: one validity condition

You are estimating one quantity: the expected correctness of the agent on the real input distribution, as judged by an oracle that decides truth. That estimate means something only if both the inputs and the oracle are produced independently of the agent under test. Break the first and you measure the wrong distribution. Break the second and you measure self-consistency instead of correctness. When the agent, or a sibling sharing its prompt, writes its own test cases or its own answer key, the number trends toward 100 percent and means nothing. That is circularity, and no better grader fixes it, because it is a structural defect in the experiment.

The practical version is small and real. Start with twenty to fifty tasks drawn from real failures; early agents fail with large effect sizes, so small samples already discriminate. Grow from four sources: hand-authored golden cases, anonymised production logs, adversarial cases from the red-team track, and synthetic variants of known cases. Give every case a datasheet: goal, initial state, available tools, expected and forbidden behaviour, ground truth, grader, severity if failed, owner, last reviewed. And keep one debugging heuristic within reach: if a task scores zero at pass@100 on a frontier model, the task is probably broken, not the agent.

If you use a model judge, validate it first

The most dangerous failure in this field is a confident, biased, unvalidated judge that everyone trusts because it has decimals. Hand-label a gold set, with one domain expert owning the labels. Run the judge. Measure agreement per class and report Cohen's kappa, not raw agreement. Iterate the rubric on the disagreements until agreement clears 90 percent, then re-validate on a schedule because your product drifts and the judge decays with it. Design against the measured biases: judges favour the answer shown first (GPT-4 stayed consistent only about 65 percent of the time when the order was swapped in the original study), they favour longer answers (a padding attack fooled some judges 91 percent of the time), and they favour their own family's outputs by ten to twenty-five points. So: binary pass/fail rather than a seven-point scale, one criterion per call, a critique before the verdict, a reference where one exists, and for high-stakes calls a panel across model families rather than one big judge.

The harness, in code

Everything above fits in a small test harness. This one runs each case n times, records a typed terminal state per trial, computes pass@1 and pass^k per task with the unbiased estimator, and refuses to let a timeout or a grader crash count as anything but what it is.

import math, statistics

TERMINAL = {"VALID_PASS", "VALID_FAIL", "SYSTEM_ERROR",
            "GRADER_ERROR", "TIMEOUT", "MISSING_OUTPUT"}

def run_case(case, agent, grader, n):
    """n independent trials of one case; every trial ends typed."""
    states = []
    for seed in range(n):
        try:
            world = agent.run(case.initial_state, case.goal, seed=seed)
        except TimeoutError:
            states.append("TIMEOUT"); continue
        except Exception:
            states.append("SYSTEM_ERROR"); continue
        if world is None:
            states.append("MISSING_OUTPUT"); continue
        try:
            ok = grader(world, case)          # grades the world state
        except Exception:
            states.append("GRADER_ERROR"); continue
        states.append("VALID_PASS" if ok else "VALID_FAIL")
    assert all(s in TERMINAL for s in states)
    return states

def pass_hat_k(states, k):
    """Unbiased pass^k for one task from n trials with c successes."""
    n, c = len(states), states.count("VALID_PASS")
    if k > n: raise ValueError("k larger than trials")
    return math.comb(c, k) / math.comb(n, k)

def evaluate(suite, agent, grader, n=8, k=5):
    per_task, attrition = {}, {}
    for case in suite:
        states = run_case(case, agent, grader, n)
        per_task[case.id] = {
            "pass@1": states.count("VALID_PASS") / n,
            f"pass^{k}": pass_hat_k(states, k),
        }
        for s in states:
            if s not in ("VALID_PASS", "VALID_FAIL"):
                attrition[s] = attrition.get(s, 0) + 1
    p1 = statistics.mean(v["pass@1"] for v in per_task.values())
    pk = statistics.mean(v[f"pass^{k}"] for v in per_task.values())
    worst = min(per_task.items(), key=lambda kv: kv[1][f"pass^{k}"])
    return {"pass@1": p1, f"pass^{k}": pk,
            "worst_task": worst, "attrition": attrition,
            "tasks": len(per_task), "trials": len(per_task) * n}

# report: pass^k per task class, the worst task, and attrition
# by state. An attrition entry is a finding, never a pass.

Three things to notice. Reliability is computed per task and then averaged, so the clustering is built in. The worst task is reported next to the mean, because the mean is where the demo lives and the worst task is where the incident lives. And attrition comes back as a first-class result. If a candidate times out more often, it does not get to look safer by disappearing from the denominator.

The ship/no-ship checklist

Fifteen checks. An agent that acts on anything real passes all of them; an internal read-only assistant can skip the last three and say so in writing.

1. Failure-mode inventory done; unacceptable failures named.
2. Risk classification and tool-permission review: least privilege, read-only by default.
3. Tracing live, with prompt, model and tool versions on every span.
4. Offline eval set of at least twenty to fifty real cases, graded cheapest-first, with datasheets.
5. Every model judge validated against a human expert: agreement above 90 percent, kappa reported.
6. Trajectory and state-diff graders on every tool-using path.
7. Adversarial suite reporting benign utility, utility under attack and attack success rate.
8. Approval gates and dry-run on irreversible or high-impact actions.
9. Per-task-class targets, not one global number; cost per successful task tracked.
10. Evals in CI as regression gates; hard fail on any safety or privacy regression regardless of average.
11. Error bars on every "v2 beats v1" claim; no shipping on a delta inside the interval.
12. Production sampling with a flywheel: new failures promoted into the eval set.
13. Rollback plan for model, prompt and tool changes.
14. Incident path and a named risk owner.
15. pass^k reported, not only pass@1, for anything that acts unattended.

A cold shower to calibrate against: a 2025 study of agents actually in production found 68 percent execute at most ten steps before a human is involved, 74 percent rely primarily on human evaluation, and of the half that use a model judge, none in the deep case studies used it without human review. Benchmarks reward long autonomy and automated grading. Production runs short, keeps a person in the loop, and treats the judge as that person's assistant. Build your eval for the world you ship into.

Tool-level tutorials for the layers above: RAGAS and Giskard for RAG evaluation, and LightEval for model benchmarks. This article is the method they plug into.

What I'd do

Write the failure inventory this week, twenty lines, with "unacceptable" marked, before anyone builds a grader. Turn on tracing and read fifty real transcripts yourself; expect three failure clusters to cover most of it, and build the first evals for those, not for the failures you imagined. Grade world state with code wherever truth leaves a trace, and reserve the judge for the genuinely subjective residue, validated against your own labels with kappa in the report. Run every case eight times and put pass^5, the worst task and the attrition table on the same page as the mean, then decide what the agent has earned: shadow, read, or act, per task class. Wire the suite into CI with a hard fail on safety, and revisit the promotion decision every time the model, prompt or tools change. Keep the whole thing as a living document the team runs on, the operating layer for the agent, and tell the truth in it; trusting the output is the habit this discipline exists to replace. If you would rather build it with us, that is Trust Engineering.

FAQ

How do you evaluate an AI agent?

In four layers: graders chosen cheapest-first (string checks and code over the trace before any model judge), trajectory and world-state checks (final state for completion, the trace for required and forbidden transitions), reliability across repeated trials measured with pass^k and honest denominators, and a qualification step that decides which permissions the evidence supports. Start from a failure-mode inventory and twenty to fifty real cases.

What is pass^k?

The probability that all k independent attempts at a task succeed, introduced with Sierra's τ-bench. If per-attempt success is p, pass^k equals p to the power k, so a 70 percent agent passes five in a row 17 percent of the time. It exposes inconsistency that pass@1, the average, hides, and it is the right metric for anything that runs unattended.

Agent evals vs LLM evals: what is the difference?

An LLM eval grades text. An agent eval grades a trajectory, the world state it produced, and its reliability across attempts, because an agent acts and can act differently on the same input. That adds tool-call grading over traces, state-diff checks, typed terminal states for every trial, and reliability metrics, on top of the text quality checks.

What is a golden dataset for agents?

A set of test cases whose inputs and ground truth were produced independently of the agent under test, each with initial state, goal, available tools, expected and forbidden behaviour, a grader, a severity and an owner. Start with twenty to fifty real failures and grow from golden, historical, adversarial and synthetic sources. If the agent writes its own cases or answer key, the number measures self-consistency, not correctness.

How many test cases do you need?

Twenty to fifty real cases discriminate well early, because immature agents fail with large effect sizes. Grow the set as improvements get subtle. For reliability, run each case several times and cluster by task; for "never" claims, no realistic sample size suffices, so pair testing with structural controls that prevent the action outright.

When is an agent ready to ship?

When the fifteen-point checklist passes for its task class: failure inventory, least privilege, tracing, a validated eval set and judge, trajectory and state graders, an adversarial suite, approval gates on irreversible actions, per-class targets, CI gates with hard fails on safety, error bars, a production flywheel, rollback, a named owner, and pass^k reported. Readiness is a permission decision per task class, not a threshold on one score.

The August table is still in our records, with the empty responses now shown as their own row. It ranks the models differently, and it is less flattering, and it is the version we trust. The fix was not a better model or a bigger benchmark. It was refusing to let silence count as success.

A score is a measurement. What the agent may do with it is a decision, and someone has to own the decision.

Charafeddine MouzouniSeptember 13, 2026