The review loop is what makes agent work shippable

An agent that opens a pull request, drafts a claim decision or reconciles a batch of invoices produces a candidate. Nobody in a large company can ship a candidate. What they can ship is output that has been checked against what was asked and against the rules and budget that apply to it.

The part of the system that does that checking is where we spend most of our design time. This is a deliberate choice. Models improve every few months, and we expect the agents doing the work to be swapped many times over the life of a deployment. The review loop, the criteria it applies and the record it leaves stay in place. They are also what we expect a CTO or an internal auditor to ask about first.

The agent alone is a demo

Gartner predicted in June 2025 that over 40% of agentic AI projects will be cancelled by the end of 2027, and named escalating costs, unclear business value and inadequate risk controls as the reasons. We read that as a description of agent work that nobody verifies. If you do not check output, you cannot say what fraction of it was right. Then you cannot say what a useful result costs, and you cannot show anyone that risk was under control.

Demos hide this because in a demo a person is the review step. They glance at the output, rerun when it looks off, and present the good run. In production that person is either absent or buried in volume.

What a review step checks

We split review into checks with different owners and different failure modes.

Check	Question	How it is decided
Spec adherence	Does the output do what the task asked, and nothing beyond it?	A reviewer agent compares output with the task spec and its acceptance criteria
Tests	Does it pass everything we can check mechanically?	Test suites, schema validation, linters, reconciliation totals
Policy	Did the agent stay inside what it is allowed to touch?	Allowlists for tools, data and scope, enforced outside the model
Cost	Did it stay within the budget for this task type?	Token, tool call and wall-clock counts from the trace

Tests run first wherever they exist. A deterministic check is cheaper than a model's opinion and easier to trust. The reviewer agent handles what tests cannot express: whether a refactor actually addresses the ticket, or whether a contract summary left out the one clause that changes the outcome.

Policy checks must not depend on the agent behaving well. OWASP lists prompt injection as the first risk for LLM applications, and its guidance is to handle privileged functions in code rather than hand them to the model, with a person approving privileged operations. So our policy check reads the trace: which tools were called, with what arguments, and what data crossed which boundary. It compares that against an allowlist. An agent that was talked into reading a file outside its scope fails review, however good the output looks.

Cost belongs in review because a correct answer that took fifteen attempts is a signal. Either the spec is unclear or this is the wrong agent for the task. We would rather learn that at review time than from the monthly invoice.

The reviewer is a different agent

The reviewer never shares a context with the author. It gets its own instructions and its own view of the task, and where it makes sense, a different model.

Part of the reason is empirical. Panickssery, Bowman and Feng (2024) found that LLM evaluators can recognise their own generations, and that this ability correlates with a bias in favour of their own output. An author grading itself inherits its blind spots, and its own reasoning sitting in the context window reads as convincing.

The other part is structural. When author and reviewer are one process, nothing in the record shows that a check happened. With separate agents, the trace shows a hand-off: what the author produced, what the reviewer looked at, what it flagged, and the verdict. That record is what an auditor will ask for.

Anthropic's engineering team describes this shape as an evaluator-optimizer workflow: one call generates, another evaluates and returns feedback, in a loop. They note it works best when evaluation criteria are clear. A large share of our work on any task type is making them clear, which means turning a loose request into acceptance criteria a reviewer can apply. We also cap the loop. If author and reviewer have not converged after a few rounds, the task escalates to a person instead of burning budget.

Where a person signs off

Many tasks need no human. A report reformatted to a new template, passing schema validation and review, can go straight through. We put a person in the loop where being wrong is expensive or hard to undo: payments, changes to customer records, production deploys, anything with regulatory weight. Tasks where the reviewer was unsure, or where no tests exist, go to a person as well.

Two rules keep sign-off from becoming a rubber stamp. The person sees the reviewer's findings before the raw output, so their attention goes where the uncertainty is. And the approval lands in the same trace as the agent work, with who approved what and when. We also watch the volume. If every task needs a human, the factory has become a queue for people, and the review criteria or the task split need another look.

Accepted tasks as the unit

We count work in accepted tasks: tasks that passed review, got sign-off where it was required, and were not rolled back afterwards. We do not count runs, tokens, or tasks the agent reports as done.

This keeps the denominator fair. A rejected run still costs money and produces nothing, and per-run metrics quietly forget that. It also gives the client something to verify in their own systems: a merged change or a posted ledger entry.

The unit has a weakness and we want to be plain about it. Acceptance is only as good as the review behind it. A lax reviewer inflates the count. So we sample accepted tasks for human audit and track how often an accepted task later turns out to be wrong. When that rate goes up, the review step is broken, and we treat it as a production incident.

The agents will keep changing. The review loop, its criteria and its traces belong to the client and stay with them, and they are what we would ask any vendor, ourselves included, to show before trusting agents with live work.