Chapter 09 / 20
9. Product & Customer Focus: Did Anyone Benefit?
Provisional draft. These short chapters are not approved for publication. Original practitioner interviews, deeper reporting and author review remain outstanding.
An experiment can run faster without finding an answer sooner. A team may build more variants, launch them quickly, and never decide which customer behavior would count as a success. At that point experimentation is a production line for uncertainty, not learning.
The Thoughtworks/Etsy case describes pilot teams reducing time to customer validation from about 50 days to five after changes to prototyping and product delivery. It is a non-AI account, and its authors caution against isolating one process tweak as the cause. The useful lesson is to measure time to a validated decision, not the number of mockups. A fast prototype that reveals a bad idea can be valuable if it prevents a costly launch.
AI may help cluster feedback, propose interview questions, and make a prototype cheap enough to test. Each step needs its own check. Feedback summaries can bury rare but serious harms; synthetic personas cannot consent to a feature or show how it works in real life; a mockup praised in a meeting is not evidence of retained customer use. Invite people from the affected segment to try the thing, observe the task, and record where it fails.
Business metrics belong in the picture, but their order matters. A customer may accomplish a task more easily while the company spends more to serve it; that is a trade-off to assess, not a reason to ignore the customer. A feature can also raise short-term revenue while making support or trust worse. Separate adoption, task completion, customer effort, complaints, retention, and margin. State which is the primary outcome for this decision and which are checks against making another perspective worse.
The customer test: Write a customer job in one sentence and a disconfirming test for the proposed feature. Ask who might be excluded from the test, what you'd learn if adoption is high but task success is low, and whether the same learning is possible with a less costly non-AI experiment.