Chapter 16 / 20
16. Measure the Result, Not the Usage
Provisional draft. These short chapters are not approved for publication. Original practitioner interviews, deeper reporting and author review remain outstanding.
Imagine an illustrative finance meeting that begins with a number: engineers say they save six hours a week. Somebody multiplies those hours by a salary and proposes a return. Yet payroll is unchanged, reviewers may be working longer, and no one has identified the customer result that filled the new capacity. The arithmetic is tidy because the hardest questions were left out.
Build a ledger with different columns. Actual savings are expenses removed. Avoided costs depend on a credible counterfactual. Created capacity is useful only if it is applied to valuable work; it is not automatically cash. Enabled revenue needs an observed path from the change to customer use and contribution margin. Reduced risk needs a baseline, severity, and assumptions about losses that might have occurred. Count tool fees, compute, integration, training, review, rework, governance, and ongoing support in the same observation window. Do not put the same change in two benefit columns.
IBM's published method and RevTech account is unusually explicit about categories and costs. It reports an approximately tenfold modeled return for its own tool using matched pre/post windows and business-loss assumptions. That is informative, not independently verified cash savings or a randomized effect. Likewise, the Microsoft-led field experiments measured a pooled increase in completed developer tasks among people given a coding assistant. They did not measure a matching gain in company profit.
For your own pilot, compare similar work over the same period; randomize eligible work or phase rollout where feasible. Log staffing, task mix, platform changes, and changed approval rules. A before/after improvement during several concurrent reforms is an observation, not a single-cause estimate. Make uncertain assumptions visible and show a range rather than a precise-looking point. Short pilots may be able to test speed and review effort while remaining too small to establish a change in rare incidents.
Use delivery and developer-experience metrics as diagnostics. Shorter lead time with worse change failure or overloaded reviewers is a warning. More PRs without customer use are not a benefit line. Satisfaction is worth listening to because people matter, but it cannot stand in for a customer or financial claim.
Audit your claim: Take your current AI business case and label every line "measured," "self-reported," "modeled," or "unknown." Find the most influential assumption and the next observation that could falsify it. If the conclusion survives that challenge, you have a stronger decision than an adoption chart could offer.