Chapter 14 / 20
14. Choose a Bet Worth Testing
Provisional draft. These short chapters are not approved for publication. Original practitioner interviews, deeper reporting and author review remain outstanding.
Suppose customer bug reports sit in a triage queue for days because each report needs a component owner and a reproducible case. An AI classifier could suggest an owner faster. That is a plausible bet, not yet a customer benefit. Misrouting one severe report could make the customer wait longer than before.
Write the bet as a chain: suggested classification → verified routing → faster appropriate fix → shorter customer resolution time. Name the customer group, the receiving team, and the decision owner. Compare the AI classifier with a non-AI alternative: simpler categories, clearer ownership rules, or better intake. A fast classifier only matters if downstream capacity can act on its suggestions.
Salesforce's BugWiser account reports a dramatic reduction in a historical classification workload and roughly 90% of classification predictions accepted unchanged in early deployment. Its comparison contrasts person-days of manual effort with elapsed time using a system; those are different units. It does not establish the customer-resolution improvement or net return from the same pilot. Use the account as evidence that this task might be worth testing, not as a forecast to insert into your budget.
Choose one primary claim. If the customer is the intended beneficiary, make verified resolution time primary. Check routing accuracy, severe-case misses, review labor, tool and operating cost, and the triage team's workload. Specify the observation window and the threshold that would trigger a pause before people become invested in keeping the tool. A small, representative pilot beats a large launch whose outcome cannot be interpreted.
Do not solve an absence of evidence by multiplying indicators. Ten green metrics can coexist with an unchanged customer result. Give each metric a role: outcome, mechanism, cost, or safety check. If no reliable baseline exists, start with instrumentation and a smaller question. Waiting for a defensible test is different from waiting for perfect certainty.
The pilot card: Complete this sentence with named owners: "We will test [intervention] against [alternative] for [population], expecting [primary outcome] by [time], while pausing if [guardrail] worsens." If the last clause cannot be written, the bet is not ready.