8 Steps to a Strategic Plan for Experiments
The following excerpt is from The Age of Customer Equity: Data-Driven Strategies to Build a Sustainable Company. Available at Amazon, Barnes & Noble, Porchlight, and your local bookstore.
Every company has limited resources, and this is especially true for experiments. A strategic plan for experiments tracks all the ideas, prioritizes them, and records the result. Here is how it works:
Assume you have limited resources. This will require you to prioritize your experiments.
Gather hypotheses. Customer-centric experiments tend to be more powerful when they cross departments. If a cross-department experiment goes forward, then use the Jeff Bezos “2-pizza team rule” to remain nimble. Include data that illustrates the problem if possible. This forms your baseline measurement for success. Clearly state what success looks like and what will be done if the experiment succeeds.
Value each hypothesis. There are two ways to do this. Companies new to the Learning Zone may want to start with conversion use cases and then back into the many elements touched on the path to conversion. For example, customers that abandon the cart did not click to calculate their shipping costs. Therefore, we could calculate the potential value of this experiment by adding up all the customers who did calculate their shipping costs and assess a one-percent lift, an increase in potential sales. This is just a logical guess, but the valuation can illustrate the financial impact of an experiment. A second way to do this for more advanced Learners is to include customer lifetime value (CLV). For example, we know more medium-value customers abandon the cart than high-value customers. If we moved two percent of medium-value customers to high value by unblocking this point of friction, it would be worth $X to us. The difference in the first example is we are using historic numbers. In the second example, we are using CLV which could produce larger numbers because it accounts for the potential of all future purchases (subject to the time frame of the calculation).
Evaluate promising hypotheses including the level of effort. Do we have enough traffic/calls/interactions to get statistical significance in a reasonable time? Is there a seasonal effect that might affect the experiment? Can we do it technically? Do we have resources to create assets for the experiment? Who will you target and how? Does this experiment fit with other initiatives? This last question is to make sure the purpose for the experiment does not disappear while it runs. For example, there is a new payment type on your site that you want to improve, but another team has already decided to remove that payment type soon. This would be a fruitless experiment.
Schedule the experiment for development. In addition to including a clear control, check to be sure it is doing what you think it is doing before the experiment goes live. Then check again to ensure the experiment is not creating a negative customer experience.
Monitor the experiment. This includes multiple channels such as call center, online, service agents. Are people having weird experiences?
Analyze the results. Significance is reached and the experiment has a winner—or more likely, it is inconclusive. Fight the urge to simply run it again. Let the analysts dig in to understand confounding results and extract learnings. Then, either release the winner or adjust and go back to step one.
Socialize the knowledge. This is the hook that supports the evolution of customer-centric business culture. To move through the Learning Zone, a company must be able to absorb and eventually build upon each customer insight. Socializing what you have learned already can also prevent repetitive experiments. Knowledge libraries are helpful here, but the best socialization is an executive shout-out.
When Learners begin this process, they may want to operate linear experiments, each one running after the other. That works out to approximately eight experiments a year. As it becomes clear that results are not always earth-shaking, the desire to run more experiments faster will increase. At this point, it is wise to add a project manager to coordinate the many aspects of experimentation.