Our real-world testing was highlighted in the Harvard Business Review
Why this year? We think 2021 is going to be a year of great change.
If you are not experimenting online, this is your year.
Why this year? We think 2021 is going to be a year of great change. Finding opportunity in the intersections of big shifts will require creativity and speed. Experimentation identifies beacons of opportunity by validating promising ideas—quickly.
What is online experimentation? Everything from A|B tests for incremental improvement to the strategic, game-changing tests of product concepts and brand positions that we do at Spark. The common thread: feedback based on behavior in real-life settings.
Here are a few reasons to consider running online experiments:
Most tools for strategy validation are more, um, time-honored than effective. Surveys, for example, are widely used but not predictive of behavior. “People SAY that data privacy is important to them and they will exercise their right to be forgotten,” writes Maria Twigge, “but what they often DO is blindly agree yes, yes, yes to sharing their data to get at products they want.”
Derivative approaches to strategy—applying so-called “best practices” to your organization--don’t work either. Practices rarely translate directly.
What works to validate strategy? Behavior. Presenting potential customers with an authentic experience allows you to measure their interest. Clicks on content or ads that present new potential products or new brand positions measure relative interest. Additional data further down the funnel, like an email sign-up on a “coming soon” landing page that provides details on the new offering, further validates interest.
Strategy is about making choices. Leaders make better choices about strategic options when they supplement gut feel with evidence.
Is online experimentation a substitute for measuring customer reaction to a fully functional new product or a comprehensive brand repositioning? Of course not. But it’s a great way to eliminate concepts and positions that have no customer interest.
Experimenting helps avoid expensive commitments, like building offerings that have no demand. It also helps to fine-tune offerings that are of interest to potential customers: testing three variations of a new product concept in parallel, for instance, helps identify which set of features generates the most engagement.
If you want to manage risk, running iterative online experiments early in an initiative development process can zero in on promising concepts and focus organizational priorities—and avoid disasters.
People are pretty terrible at predicting how new ideas are going to perform. Even those whose predictions average above the mean are rarely able to quantify the value of an innovation with any accuracy.
Online experiments provide data. Knowing what proportion of a target audience will click on an ad for a new offering, how many will sign up to learn more, what proportion make a purchase—all these bits of data add up to real insight in an emerging sales funnel. Data creates confidence to move forward—or tells the team to go back to the drawing board. Data is the feedback loop that helps a team learn.
While we respect the Lean Startup approach of build-launch-learn-rebuild, it still has an element of commitment to a particular path. Most teams can only build one product at a time, so they are often stuck iterating on the first build. By the time you know it doesn’t work, you’ve wasted time on the wrong thing.
How can you learn faster? Test product concepts, not finished products, in parallel and use market feedback to drive the direction of what you build.
New products are important: at least a quarter of revenue and profits comes from products launched within the previous three years. But companies are often successful because they become really good at one thing, which can make trying something new a cultural challenge.
Online experiments can open the floodgates for new ideas from all corners of an organization. Booking.com, for example, has as many as 1000 experiments running at any one time, and all employees can suggest experiments. Wow. And it works: annual growth was 19% between 2012 and 2019.
