"Hi, we're Growth, LET'S EXPERIMENT ON EVERYTHING! Sound good?"
3 quick thoughts on how to introduce and scale experimentation
Growth teams are positioned inside SaaS companies to optimize the existing business and validate areas of uncertainty. They are added to growing companies that have product-market fit and established ways of working.
The main ‘thing’ growth teams do is run experiments. We look to significantly improve the conversion rates of key input metrics that drive revenue. Generally, these experiments are directly tied to one of the “pirate metrics”
It’s likely the growth team will be running more experiments than product or marketing teams. More language around ‘significance’ or ‘validation’ will start to pop-up over and over you might even get a fun eye-roll from your colleagues when dropping these words after a long zoom call.
Introducing experimentation at scale requires diligence. Just because you run experiments all the time doesn’t mean other teams should follow suit.
“To be clear, not all innovation decisions can be tested, and not all test results should be followed blindly. Ethical, legal, or strategic considerations may favor a different course of action, and in such cases, good experiments can add clarity to why decisions are made.” - Stefan Thomke
Here are 3 things to think about when introducing & scaling experiments at your organization
#1 Don’t Push Experiments on Everyone
I like to think we can always run an experiment at Jobber to validate something. Yet, we need to understand the costs behind that experiment.
Without a doubt, the biggest pain an experiment will take from a business is time. This is especially true in B2B as sample sizes decrease further and further down the funnel.
Is it worth it? When experimenting in an area of focus you may have an experiment that will inform the next experiment, and that experiment will inform the one after that, and we go on and on and on.
Sequential experimentation relies on learnings from previous experiments and will further the cost of your experiments:
“When the identification of a solution involves more than a single experiment, the learning gained from previous iterations may serve as an important input to the design of the next one. When that happens, experimentation occurs sequentially. Because it took two months to build and test a prototype, there simply wasn’t enough time for a purely sequentially learning strategy.” - Stefan Thomke
With these considerations (and many more), not all decision-making should be solved by running experiments. Unfortunately, growth teams create anxiety for teams who want to make a decision without running an experiment.
Growth needs to ensure other teams have confidence in their decision making regardless if they ran an experiment or not.
If they start questioning “just do’s” vs “running an experiment” you need to help them consider the paths of both options and ensure them that either decision is fine as long as they acknowledge how that decision was made.
#2 Tool Teams with Experimentation Processes
Typically growth teams are leaders in experimentation. They start out as the centralized source of how to communicate, document, and run experiments.
As a result, scaling growth will also scale experimentation, but for me, it is a mistake for growth organizations to put their blinders on and be the only source for company experiments.
Stefan Thomke suggests an approach called the “center-of-excellence model”. Here, the growth organization is still looked at as the experts within experimentation but they put in the considerable effort with key members outside of growth to properly train and coach them on how to run experiments.
This allows experimentation to properly spread across the business to increase the rate of learnings the company is generating.
If you don’t enable other teams to run experiments you will become a bottleneck for the organization, as Thomke describes:
“..the number of tests was now limited only by the company’s ability to “feed” hypotheses. When companies reach this inflection point—experimentation growth is constrained by organizational issues—management needs to focus on issues such as culture, integration of tests into decision making, and even governance.”
#3 Balance Incremental Improvements with Big Swings
As experimentation gains traction within companies it becomes important to not get caught up in incremental changes. You will quickly become a well-oiled machine of endless experiments tied to optimization.
Over time a team will hit diminishing returns within that machine and leadership will push experiments into areas of uncertainty that require a different mindset.
“When you have a strong culture of experimentation that makes incremental improvements to an existing product, there comes a point when the people who built the original product are gone and new products are not in your DNA any longer. You have become a lean and mean machine for customer conversion, for micro-optimizations driven by experimentation. But when you want to branch out into new areas, you no longer have people who think big, who knows how to do this.”
You may have noticed I quoted Stefan Thomke a lot in this post. He wrote a book called “Experimentation Works: The Surprising Power of Business Experiments”.
I highly recommend this book to anyone in growth. Each member of our team has a copy and we read a chapter every week and met for 1 hour sharing our thoughts and insights.