How to know when you can trust an agent.
A production AI agent is not "a prompt plus tools." It is a probabilistic distributed system with a language model in the control loop. The same input can produce different actions, so reliability is a distribution, not a property. It calls tools, services, and other agents, so half of "agent quality" is plain reliability engineering. And the non-determinism sits at the decision point, which is exactly what makes these systems powerful and hard to evaluate.
Most writing on agent evaluation is either academic (benchmarks that don't resemble your product) or promotional (a vendor's dashboard, retold). This is neither. It is a working manual, assembled from what the teams shipping the most-used agents actually do, and from original research where the public literature falls short.
The spine of all of it is one sentence:
No autonomy without evaluation. No evaluation without traces. No traces without ownership. No deployment without rollback.
And underneath it, the validity condition the whole discipline turns on: an evaluation is only as trustworthy as the independence between its data and the system under test. The moment the thing being measured is also the thing doing the measuring, you no longer have an evaluation. You have an expensive way of agreeing with yourself.
The six non-negotiables
Strip away the branding and every serious lab converges on the same six moves. Do only these and you are ahead of most teams shipping agents today.
- Grade outcomes, not trajectories. Check the end state of the world, not the path the agent took to get there.
- Look at your own data first. Read real traces and build a failure taxonomy before you choose a single metric.
- The LLM judge is a model you must evaluate. Validate it against a human expert, or every number downstream is fiction.
- Report reliability, not just capability.
pass^k, not pass@1, for anything that acts.
- Security is a separate axis. Capability is not safety. Least privilege and human gates, not "ignore injections."
- Data independence is the validity condition. The agent never writes its own test cases or its own answer key.
What's inside
Seven parts, twenty chapters, three appendices. It moves from the mental model to the lab canon to the machinery to production, adversarial testing, real case studies, and governance.
- Part I — Foundations: the mental model of what you're actually evaluating, and how to generate a golden dataset without fooling yourself.
- Part II — The Lab Canon: how Anthropic, OpenAI, and Google / DeepMind actually evaluate agents.
- Part III — The Core Machinery: LLM-as-a-judge (with a build-and-validate runbook), agent benchmarks worth stealing, and calibration, uncertainty, and abstention as a runtime trust layer.
- Part IV — Production: observability and tooling, and reliability engineering for agents (SLOs, bounded autonomy, graceful failure).
- Part V — Adversarial: security and red-team evaluation, a red-team starter kit for tool-using agents, and a measured exploitation surface.
- Part VI — Practice: production case studies and the applied playbook: how to actually evaluate and ship reliable agents.
- Part VII — Governance & Assurance: risk registers, assurance cases, audit trails, and change management for production agents.
- Appendices: a glossary, ready-to-use templates and artifacts, and an annotated reading list.
Who should read this
- AI / Platform Leads: the full field guide. The machinery and production chapters are the core.
- ML / Software Engineers: the judge runbook, the benchmarks, and the reliability-engineering chapter.
- Security / Compliance: the adversarial part and governance and assurance.
- Engineering leadership: the mental model, the six non-negotiables, and the applied playbook.
Newcomers should read Part I, then the applied playbook, then return for depth. Practitioners can enter at the chapter matching the gap in front of them; the cross-references pull in the rest.
