Measuring return on an AI system without fooling yourself

Why the time saved number is usually made up

Time saved is a gap between a number you saw and a number you imagined. You know how long the task took with the tool. The other half, how long it would have taken with no tool, never ran. People fill that half in after the fact, and they fill it in with a story about the work they just did well.

There is a measured version of this. In a trial run by METR in early 2025, 16 skilled open source developers worked 246 real tasks from their own repos, with AI tools allowed on a random half of them. The tasks where AI was allowed took about 19 percent longer. The same people, right after the work, said AI had made them about 20 percent faster. Their read of what would have happened was off by some 40 points, in the way that felt good.

That is one study on one group, and it does not mean your tool is slowing your team down. It means self report is not proof. If your case is a survey that asked people how many hours a week the tool saves them, times a loaded hourly rate, you do not have a number. You have a mood, scaled by payroll.

The second break in that sum is the multiplier. Saved hours turn into money only if one of two things happens: you pay for fewer hours, or the free hours make something you can sell. Neither of them happens on its own. An hour saved in twelve places, in ten minute slices, is an hour nobody can spend. Say which of the two you expect before you start, because that choice sets what you have to count.

• Never build a case on how much time do you think this saves you.
• Write down which path you claim: fewer hours paid, or more sold.
• If you can name neither, this is a skill test, not an ROI case. Say so.

Set a baseline before the tool turns up

The baseline is the whole ball game, and it is the step every team skips, because it is dull and it holds up the demo. Give it four weeks if you can, two at the least, and run it on the way you work now with nothing new in it. You are not grading your people. You are writing down what a normal month looks like, the bad weeks too, so that a later number has something to sit next to.

Keep the spread, not just the mean. One mean handle time says almost nothing, since the tail is where the cost sits. Keep the median and the 90th, both. If the after picture moves the median and leaves the tail alone, the tool is helping with easy work, which may be just what you wanted, and is a very different claim from the team got faster.

If the tool is already in, you can still get a baseline back, as long as you take it from systems that logged the work and not from people who recall it. Ticket times, commit and merge times, publish dates, CRM stage moves, call lengths, invoice dates. Pull the same window from last year as well as the months before the tool went in, since that gives you a shot at telling a real change from a seasonal one.

Freeze the words on day one. What counts as one unit of work, what counts as done, which items are out, and who says so. Every rule you leave loose gets widened later, always the way that flatters the project. That is not spin. It is what people do when a number they care about sits near the line.

• Four weeks of baseline, two at the least, on the process as it is.
• Keep the median and the 90th, not the mean on its own.
• Build it back from logs, never from memory, if you start late.
• Fix the words for done and for out of scope before the tool lands.

Pick one unit of work and define done

A pilot you can measure has one unit. A ticket. A blog post. A pull request. A lead worked. A claim paid. Teams who try to measure a whole team end up with a board of numbers that all move for their own reasons, and no way to say which move came from the tool.

Set done at the far end, past the point where the AI stops. A draft is not a post. A drafted reply is not a solved case. A patch is not merged code. Wherever the tool hands off, push your line one step past it, since that is where the work it made for the next person shows up.

Write the out of scope rules now. Which items you skip, what you do with an item that starts before the tool and ends after, what you do with one that comes back, and whether a duplicate counts once or twice. Then hold to them. The way an honest team ends up with a false number is by dropping the awkward cases one at a time, each for a reason that seemed fair on the day.

• One unit only: a ticket, a post, a pull request, a lead, a claim.
• Done means shipped, sent, merged, paid, or closed and not back.
• Set the line one step past the point where the tool stops.
• Count what you drop, and keep that count next to the result.

What to log, and where the data already sits

You want one record per unit of work, written as it happens, not pieced together at the end of the quarter. Most of the fields are already in the tools you run. The new ones are the arm, how much a human had to edit, and the rework flag. Here is a record that has held up in real pilots, one JSON line per unit:

{ id, unit: ticket, arm: tool | control, opened_at, first_touch_at, done_at, handle_seconds, assisted: true, model_calls: 3, tokens_in: 8400, tokens_out: 1100, human_edit_ratio: 0.31, escalated: false, reopened_7d: false, reviewer, approved_at, outcome: resolved }

Three of those do the heavy lifting. The arm lets you match like with like later on. The edit ratio, read as changed chars over final chars, is the cheapest proxy you have for how much of the job the tool really did. And the reopen flag is the one that kills a false win, since the fastest way to cut handle time is to close things that are not done.

Log the bad path as hard as the good one. Count drafts thrown out, hand offs, retries, and items a person took over in full. A pilot that reports only the assisted items that shipped is reporting a filtered sample, and that filter is the very thing you set out to measure. If the tool wrote ten drafts and two were used, the cost per used draft is ten drafts, not two.

• One event per unit, written as it happens, stored where you can query it.
• Log the arm on every item, the control arm too.
• Log calls and tokens, so cost per unit is a sum and not a guess.
• Log the rework: cases that come back, hand offs, drafts you threw out.

Output metrics and outcome metrics are not the same

An output metric counts what came out of the machine. An outcome metric counts what the business got. Drafts written is output; posts that go live and get read is outcome. Replies drawn is output; cases closed that do not come back is outcome. Lines of code is output; a change that ships and stays up is outcome. Output moves first, and it moves more, since making the thing is the cheap half of the job.

Pair every speed measure with a quality measure and read the two as one, or you will buy speed with defects and book it as a win. The DORA four keys are the clean worked case of that pairing: two flow measures, how often you deploy and how long a change takes, held next to two stability ones, how often a change fails and how fast you recover. The point is not the four names. The point is that a flow number read on its own can always be made to look better by lowering the bar.

Pick your quality half from what already hurts when it goes wrong. Support: cases that come back inside seven days, and hand off rate. Content: how many drafts go live, and edits made after they go live. Sales: meetings held out of meetings booked, not mails sent. Code: how often a change fails. Each of those is already in your stack, which is why they outlive a pilot better than a survey nobody will run twice.

This rule costs you time and you should know it. Outcome numbers are slow. A case comes back in a week, a post earns its traffic over a month, a merged change takes a quarter to prove it made no mess. Your pilot will end before those land. Report the output number with the date the outcome number is due, and hold the call open until then when the size of the spend is worth the wait.

• Output is what came out: drafts, replies, calls logged, lines of code.
• Outcome: posts read, cases shut for good, code that stays up.
• Read a speed number next to a quality number, or do not read it.
• Name the date the slow half of the number is due to land.

Count the whole cost, not the licence

The licence is the easy line and often the small one. Add the model spend, which on an agent that reads long files does not track seat count at all. Add the review time, which is new work you made, at the rate of the person who does it, and the reviewer is usually a senior one. Add the rework on bad output. Add the set up: the prompts, the wiring, the test suite, and the person who keeps them alive when a model shifts under them.

Then add the cost of the counting, since it is not free and hiding it is how counting gets dropped in week three. Wiring up events, running a control arm, and writing the read out is real time from a real person. Put a number on it and ask if the spend earns it. A tool at a few hundred a month does not earn a four week trial with a control arm. A system that touches every customer does.

Report net, per unit, with the sum on show. Cost per unit is total spend over the window, divided by units done in the window, where spend is licence plus tokens plus review time plus rework time. Anyone who wants to fight your call can then fight a line, which is the point. A single ROI percent with no division on show cannot be argued with, and numbers you cannot argue with do not last five minutes in a finance room.

• Licence and seats, plus model spend, read per unit where you can.
• Review time at the rate of the person who does the review.
• Rework: hand offs, cases that come back, drafts you threw out.
• Upkeep: prompts, wiring, tests, and the person who owns them.
• The counting itself, in hours, named as a cost line.

How to run a fair before and after when volume changes

Volume will move during your pilot. A release lands, a campaign runs, a rival goes down, a holiday empties the queue. Totals go dead the moment that happens, since total hours rise with volume whatever the tool does. Switch to rates: minutes per unit, cost per unit, cases back per hundred, units per person per week. A rate lives through a volume change. A total does not.

Rates on their own still leave you open, since the mix can move too. Easy tickets arriving in bulk will cut your handle time with no help from any tool. So hold something back. In order of strength: split by coin flip at the item level, which is what METR did and which beats all the rest; split by team or by queue; stage the roll out so week one is one group and week three is the next; or keep one queue on the old way for the whole run. A hold out of 20 to 30 percent is enough to catch a big effect and cheap enough to live through the meeting.

With a hold out, weigh the change in the treated group against the change in the control group over the same weeks, rather than before against after in the treated group alone. That subtraction is what takes out the release, the season, and the new hire, since those hit both groups. In public policy this is called difference in differences, and the Magenta Book sets out when it holds, which is worth a read before you claim it.

Two guard rails. Fix the window and the sums before you look at the data, since a window picked after the fact is a result picked after the fact. And keep away from the fan arm: if the pilot group put their hands up and the control group did not, you are measuring keenness. Assign the arms. Do not let them form.

• Use a rate per unit, not a total, the moment volume moves.
• Hold out 20 to 30 percent of items or people for the whole run.
• Weigh the change in one arm against the change in the other.
• Fix the window and the sums before you look at the numbers.

A worked example, with numbers that are made up

Every number below is ILLUSTRATIVE. They are invented to show the sums and the traps. They are not results we measured, not a customer case, and not a claim about what any tool does. Copy the method, not the figures.

Set up for the example. A support team of four. The unit is one ticket. Done means solved and not back inside seven days. The baseline is four weeks with no tool. Then four weeks with a drafting aid given to three of the four, with the fourth held out on the same kinds of queue.

Baseline, illustrative: 1,200 tickets, median handle time 14.0 minutes, 11 percent of them come back, and one that comes back costs about 9 more minutes. So the real minutes per ticket is 14.0 plus 0.11 times 9, which is 14.99.

Pilot, illustrative: volume rose to 1,500 tickets because a release shipped. Total team hours went up, so the naive read is that the tool made things worse. Per ticket, the treated group shows a median handle time of 11.2 minutes and 13 percent coming back. Real minutes is 11.2 plus 0.13 times 9, which is 12.37. That is 2.62 minutes better per ticket, or 17.5 percent, and you can only see it because you moved to a rate.

Now the hold out, which is the part that decides whether you learned a thing. Say the held out person, illustrative again, also got better, from 14.99 to 13.60 real minutes, because the release made the tickets easier. The change in the treated group is minus 2.62. The change in the control group is minus 1.39. The gap is minus 1.23 minutes per ticket, and that gap is your estimate of what the tool did. With no hold out you would have claimed 2.62 and been wrong by more than double.

Money, illustrative. 1,500 tickets times 1.23 minutes is 1,845 minutes, near enough 30.8 hours over four weeks. Against that: $260 of licence and tokens, plus 6 hours of a lead keeping prompts and wiring alive, plus review time that already sits inside the handle time. So the honest claim is that the team took on 300 more tickets with no fifth hire, and gave back about 25 hours net. The dishonest read of the same data is a 17.5 percent lift and 65 hours saved, and it comes out of the same sheet.

What the example would not prove, even if the numbers were real. One team, one month, one queue. The share of tickets coming back went the wrong way by two points, which is a quality signal you would want to watch for another month before you scale. And one person as a control arm is a tiny sample, so the honest write up says the effect looks positive, looks small, and here is the date we check it again.

How AI ROI numbers go wrong

The table below lists the ways a pilot ends up with a confident wrong number. Most of them are not fraud. They are the ordinary result of counting something you want to be true, on a deadline, with data that was never collected for the job.

Two of them carry extra weight because they are near universal. Counting output and calling it outcome, which lifts every early pilot. And picking the window after you see the data, which can make any result you like out of an honest set of numbers. Guard against both in writing, before the tool is switched on.

Write the decision rule before the pilot starts

Decide up front what result makes you keep this, what result makes you kill it, and what result means you do not know. One page, signed by whoever owns the budget, before anyone touches the tool. It takes twenty minutes and it is the only real guard against the ending where the pilot runs for ever because every number can be read two ways.

Make the rule a rate and a threshold, not a direction. Not handle time gets better, which noise alone will satisfy. More like: real minutes per ticket, read as the gap between the arms, improves by at least 8 percent, with the share of tickets coming back no more than one point above baseline, over four weeks. Add the not sure branch: if the gap is positive but under the bar, we run four more weeks once, then stop.

Then say who reads the result and when. A number with no meeting on it is a number that gets lost, and the tool stays either way, which is the ending the whole job was meant to head off. Put the review date in the calendar on the day the baseline starts.

• Name the metric, the bar, and the window, up front.
• Include a stopping rule for the case where you cannot tell.
• Name the person who decides, and the date they decide on.

What Agentik can and cannot tell you here

We sell AI operating systems that install on the MCP host you already pay for, so two of these lines work out a bit differently for us, and you should know which. The model runs on your host and your host pays the tokens, so the model spend in your cost per unit is a bill you already get and can read, not a line we invoice. Work that publishes, sends, spends, or resets waits for a human approval, so the approval is a real event with a real time stamp, and you can read review time off it instead of guessing at it.

What we cannot do is tell you whether the work got better. We do not sit in your CRM, your CMS, or your support desk, so the outcome half of every metric on this page comes out of your systems, not ours. Any vendor who hands you an ROI dashboard for their own product is showing you their output counts with a currency sign in front. The baseline, the hold out, and the outcome number are yours to build, and they are the one part of this that no vendor can hand you.

What this method does not measure, and when it is too much

Small teams cannot get the numbers to speak, and no amount of care fixes that. With four people and 200 items you will not reliably see a 10 percent effect, and a clean looking result at that size is as likely to be noise as signal. The right move is not to fake a decimal place. It is to say what size of effect you could have seen at all, report the direction, and treat the call as a judgement helped by numbers rather than one settled by them.

The method also misses all that is not the unit you picked. The learning curve, which means an early read sells short a tool people are still getting used to, and a late read may oversell it once the easy wins are taken. Morale, which is real and moves both ways. Option value, which is the work that becomes possible rather than faster, and which no before and after can see, since the before holds none of it. If the case for a tool is that it lets you do a thing you were not doing at all, count take up and quality of the new thing, and drop the ROI frame.

And the method costs more than some calls are worth. A four week baseline, a hold out arm, events logged per task, and a write up is perhaps a week of one person, spread over two months. Do not spend that to judge a tool that costs less than the counting. For small buys, run it a month, ask the people using it if they would fight to keep it, and move on. Save all this for the systems that touch every customer, every invoice, or all that you publish.

• Under a few hundred items, give a direction and a doubt, not a percent.
• Read it twice: once at week four, once at week twelve. The curve moves.
• If the tool adds new work rather than faster work, count take up instead.
• Never spend more on the counting than the call is worth.