Browse the book & resources

Supporting material

Desk-researched interview 7: Proving a pilot mattered

Desk research, not original interviews. These question sets synthesize published accounts; no practitioner was interviewed for this book.

Question: What would count as evidence that a pilot helped customers, improved financial performance, or reduced operational risk—not merely that employees liked a tool?

Method: Desk research from published engineering pilot accounts and a non-AI customer-outcome comparison. This is a proposed evaluation interview, not a firsthand report of an AI pilot's causal customer impact. Research snapshot: November 2026.

What the published accounts establish

  • HubSpot's initial AI coding-assistant pilot included whole teams for more than two months. Its account says it saw positive feedback and modest measured engineering productivity gains, and later examined review burden, cycle time and production incident rates. A chart with sanitized team-level data is described as showing no correlation between AI adoption and incident rates. That is not proof of reduced operational risk, nor is a $19-per-user monthly price a calculated net return once integration, verification and training are included. The article does not report pilot-attributable customer or financial outcomes. HubSpot engineering, 2025.
  • Salesforce HRP reports a 22-to-11-day cycle-time change, more throughput and steady code maintainability during an AI-adoption program. Its account also notes process changes to approval gates and proposes expanding measures of change failure, time to resolution and security-bug lead time. These before/after delivery metrics are useful but do not establish a customer, net-financial or causal risk improvement; concurrent changes complicate attribution. Salesforce Engineering, 2026.
  • As a non-AI contrast, Thoughtworks' Etsy case links faster customer validation (a pilot team's “time to learning,” 50 to five days) with a separate inventory-quality initiative reporting $11 million incremental value. It shows the kind of outcome an organization might try to trace, but it is not evidence of an AI pilot and does not isolate every concurrent cause. Thoughtworks/Etsy case study.

Synthesis, not testimony

Evidence that would count: before rollout, name the pilot workflow, intended customer population, baseline, decision threshold and comparison (randomized eligible teams or a credible matched cohort where feasible). Then trace the chain from tool use through released change to customer adoption, task success or retention (effective), net contribution margin or cost per validated outcome after licenses, review, training and operations (financial), or severity-weighted incidents, change failure, recovery time and security exposure (reliable). Track idea-to-value lead time (efficient), decisions to keep/kill/scale (innovative) and reviewer load and sustainable learning (engaging) as counter-metrics. Specify the observation window and risks of different task mixes or simultaneous process changes.

An employee saying “I like the tool,” high seat adoption, more PRs, shorter local coding time or a stable incident count alone would not settle any of the three business claims. A null or adverse result is a valuable pilot finding if it changes the decision.

Questions to ask a practitioner

  1. Which customer, margin or risk measure is the primary decision metric, and what change would justify scaling?
  2. What comparison separates the pilot from release-process redesign, seasonal demand and selection of easier tasks?
  3. Who counts verification cost and downstream workload, and what would make you stop the pilot?

Follow-up desk-researched responses

1. Primary measure and scaling threshold? For a customer-feedback triage pilot, a candidate primary metric is report-to-validated-action time, paired with customer resolution time and wrong-classification rate. Salesforce's BugWiser account says a historical classification workload took over 300 person-days of manual effort, versus less than a week with the system, and about 90% of classification predictions were accepted unchanged in early deployments. Person-days of effort and elapsed week are different units; the article does not provide the customer-resolution lift or a predeclared scale threshold. Those must be agreed locally before scaling. Salesforce BugWiser account.

2. How to separate the pilot from other changes? None of the published HubSpot/Salesforce accounts above supplies a controlled customer-outcome comparison. A proposed design is to assign similar eligible teams, incidents or work items to pilot and comparison groups before adoption, measure the same outcome window and risk mix, and record concurrent process changes. Salesforce HRP explicitly mentions approval optimization during its before/after period; don't attribute its 22-to-11-day cycle-time change solely to AI. Salesforce HRP account.

3. Who counts costs; when would you stop? Finance and engineering should jointly reconcile tool spend, build/hosting, verification hours and downstream reviewer/on-call work with the same workflow and period. A different Salesforce engineering account reports over $500,000 in annual token-cost savings from an internal mock LLM benchmarking service; it does not disclose full implementation/operating cost or prove that an AI coding-assistant pilot earned that amount. This is a concrete cost line, not net ROI. A proposed stop rule: pause if the predefined customer/risk metric worsens materially, a critical safety gate fails, or full incremental cost exceeds demonstrated benefit after a sufficient observation window. The threshold, owner and stop authority are not specified in these accounts. Salesforce mock-service account; Google Cloud/DORA ROI discussion.

Evidence gap: The cited AI accounts do not provide a controlled, end-to-end estimate of AI's effect on customers, net profit or operational risk.