Browse the book & resources

Chapter 10 / 20

10. Innovation & Learning: What Changed Because We Learned?

Provisional draft. These short chapters are not approved for publication. Original practitioner interviews, deeper reporting and author review remain outstanding.

Picture an illustrative demo day: six teams build prototypes, and the closing presentation is full of energy. A month later, nobody can say which idea was tested with a customer or which should be stopped. The event produced artifacts and enthusiasm. It has not yet produced a decision.

Innovation means taking a worthwhile uncertainty seriously enough to expose it to evidence. Start with the smallest question that could change a choice: do customers understand the workflow, can a team maintain it, will a new service reduce recovery time, or is a suspected use case too risky? A useful experiment can end with "do not proceed." That answer protects time and trust that would otherwise be spent defending an attractive idea.

AI can reduce the cost of a prototype, generate candidate test cases, or help a team explore an unfamiliar design space. It can also produce enough plausible alternatives to defer judgment indefinitely. Non-AI discovery—watching someone attempt the task, sampling incident records, talking to the reviewer of a similar change—may reach the real uncertainty faster. Compare the two routes by time to credible decision, not number of ideas generated.

The Superhuman/AWS account reports more experiments per engineer in a self-serve growth team; it credits cross-functional work as well as AI. It does not publish an AI-attributable customer-value increase from those experiments. That distinction does not make the experimentation meaningless. It tells us which later question remains: which experiments changed a product decision, and what happened to customers afterward?

Learning also has to travel. If a pilot team discovers a safe evaluation method, make it easy for others to reuse. If it discovers a failure, publish the boundary as clearly as a success story. Protect time to reflect; otherwise every available hour goes to the next demo and no one updates the route.

At the junction: For the last three experiments, find the written hypothesis, the observation that could have changed minds, and the resulting keep/change/stop decision. If none exists, choose one unresolved question and design a smaller test before starting another showcase.

Sources cited in this chapter

  1. https://aws.amazon.com/blogs/migration-and-modernization/superhumans-ai-advantage-measuring-the-business-value-of-ai-adoption/

Browse all references →