Measuring cost per accepted task

Ask what an agent costs and the answer usually comes as a price per million tokens or a cost per run. Both are easy to read off an invoice or a dashboard. Neither tells you what one finished unit of work costs, and that is the number a head of operations compares with the process they run today.

We measure cost per accepted task instead. Below is what goes into it and how far it can drift from the cheaper-looking numbers.

Why per-token and per-run numbers mislead

Token prices describe inputs. A cheaper model that needs several attempts and still fails review often costs more per useful result than an expensive model that gets it right on the first try. Kapoor and colleagues made a similar argument about public benchmarks in "AI Agents That Matter" (2024): evaluations that look only at accuracy reward agents that are needlessly complex and costly, so cost should be reported and optimised alongside accuracy. Inside a company the same logic holds, with one adjustment. The accuracy that counts is whether your own review accepts the work.

Cost per run has a different blind spot. A run that ends in a rejected output costs as much as one that ends in an accepted output, and a per-run average treats them as equal. Retries disappear into the mean.

Both metrics also leave out the two lines that tend to dominate: the review itself (reviewer agents, test runs, CI minutes) and people. Human time scales with how often agents are wrong or unsure, which is exactly what token and run counts do not show. When Gartner listed escalating costs among the reasons it expects many agentic AI projects to be cancelled by the end of 2027, we suspect a good part of that cost was never on the model invoice.

The formula

cost per accepted task =
    ( author agent spend, all attempts
    + reviewer agent and test spend
    + human minutes x loaded rate per minute
    + allocated platform cost )
  / number of accepted tasks

An accepted task is one that passed review, got human sign-off where the task type requires it, and was not rolled back within an agreed window (we propose two weeks as a starting point, then adjust per task type). Rejected and abandoned tasks sit in the numerator and not in the denominator. That asymmetry is the point of the metric.

Tasks escalated to a person and finished by hand are human work. Their cost counts, because the factory caused the escalation, but they do not count as accepted agent tasks.

In practice the inputs are:

• Author spend across every attempt, including retries after a rejection.
• Reviewer spend plus deterministic checks: test suites, CI time, validation jobs.
• Human minutes for sign-off, escalations, audit samples and fixing anything rolled back.
• A share of fixed cost per task type: model capacity commitments, trace storage, the observability stack.

We leave out the one-off cost of building the factory and amortise it separately, and we say so in every report. Mixing build cost into the running metric makes the first months look terrible and the later months look better than they are.

A worked example

The numbers below are invented for illustration. They do not come from a deployment and they are not a benchmark of any specific model.

Take a task type where agents classify incoming supplier invoices, route them, and write a short note on any exception. Volume is 1,000 tasks a month. We compare two configurations. Each review costs €0.02 in reviewer spend, tests and CI cost €10 a month, an escalation takes a person 12 minutes, and we audit a fixed sample of 40 accepted tasks a month at 3 minutes each. Human time is costed at €60 an hour, so €1 a minute.

	Option A	Option B
Author cost per attempt	€0.04	€0.15
Average attempts per task	1.8	1.2
Author spend	€72	€180
Reviewer spend	€36	€24
Tests and CI	€10	€10
Accepted tasks	780	930
Escalated to a person	220	70
Human minutes (escalations plus audit)	2,760	960
Human cost	€2,760	€960
Total monthly cost	€2,878	€1,174
Cost per accepted task	€3.69	€1.26

On a per-run dashboard, Option A costs €0.06 per attempt including review and Option B costs €0.17, so A looks almost three times cheaper. Per accepted task, B costs about a third of A. The model bill barely matters here. Human minutes on escalations account for most of the total in both columns, and the configuration that escalates less wins even though its model is almost four times more expensive per call.

The example also shows which inputs deserve the most care. If the true time per escalation were 4 minutes instead of 12, the gap between A and B would narrow a lot. Acceptance rate and minutes per escalation are the two numbers we measure first on any new task type, and we take minutes from timestamps in the review tool, not from estimates.

What changes when you measure this way

Model choice becomes an experiment with a clear outcome. We can run two configurations on the same task type for a few weeks and compare cost per accepted task, instead of comparing price sheets.

Spending more on review stops looking like overhead. A stricter reviewer that costs a few cents more per task is worth it if it lowers escalations or catches rework that would otherwise reach a person later.

Budgets per task become enforceable. Because every attempt is in the trace, we can cap spend per task type and escalate when a task exceeds it, rather than finding out at the end of the month.

The metric can be gamed, and the easiest way is a lenient reviewer. So we never report cost per accepted task alone. It sits next to the acceptance rate and the rate at which accepted tasks are later found wrong, and if that last number moves, the cost figure is not trusted until we know why.

What it takes

All of this depends on one piece of plumbing: every attempt, review, test run and human action is linked to a task ID in the trace. Without that link the formula cannot be computed, only estimated. We design the trace schema before the first agent goes live, it runs inside the client's infrastructure, and the client owns it. The same traces then serve debugging and audit, which is part of why we think the effort pays for itself.

One open question we have not settled: the loaded rate for human time. Finance teams calculate it differently, and the choice moves the result. We agree the rate with the client up front and keep it fixed for comparisons, so that a change in the number reflects a change in the work.