Is behavioral testing more powerful than conventional market research?
Conventional tools like focus groups and surveys are part of the fabric of customer research, but they haven’t changed much since their debut in the first half of the twentieth century.
Behavioral testing isn’t new, either. For decades, marketers have performed ethnographic research like shopalongs, and direct marketers have deployed ‘wet tests’ to see which printed media strategies delivered the most sales.
What’s different is the impact of digital on both types of research.
The shift to a digital world has reduced trust in conventional research: professional survey-takers, for example, muddy results. But the shift to digital has enabled enormous innovation in behavioral testing. In fact, companies like Capital One used digital behavioral testing to build entire lines of business.
The rise of massive ad platforms has enabled market researchers to design powerful experiments that deliver results with empirical certainty. Here are 12 reasons why behavioral beats conventional research today:
- Eliminates the "say vs. do" gap. People don't behave the way they say they will. Surveys and focus groups capture “stated preference”; behavioral testing captures actual decisions as people are going about their day.
- Evidence, not opinion. Conventional research produces findings that are easy to argue with, and they usually require more research to understand. Real-world behavioral research produces *facts* like click-through or conversion rates. There’s no debate about what works.
- Tests strategy, not just creative. Other behavioral research like A|B testing focuses on tactics—copy, messaging, imagery, etc.. Multivariate behavioral experimentation tests strategic elements: product concept, brand identity, audience, positioning, value proposition. And unlike surveys, it shows what performs in the real world.
- Eliminates confirmation bias. A lot of conventional research validates what teams already believe. Behavioral experimentation reveals which hypotheses don’t work—no matter how strong the gut feel that generated them.
- Delivers faster. Most conventional research (focus groups, surveys, discrete choice analysis, shopalongs) with real humans takes months to create, field, and analyze. An online behavioral test with real humans takes days or weeks..
- Shows what actually drives choice. Surveys ask people why they behave a certain way. Most people are terrible at that sort of self-analysis. Multivariate behavioral data shows how each variable affects behavior. Stimuli→result. It’s science!
- Isolates variables. Multivariate/factorial design (testing every combination of variables) lets you isolate the effect of audience targeting from the effect of messaging from the effect of imagery and so on and so on. And you can do it all at once, without running lots of separate studies.
- Reduces risk at the best possible scale. Validating before you launch a new brand strategy or a new product means you make mistakes on a small scale, which saves both investment and embarrassment.
- Produces defensible, replicable findings. Behavioral experimentation uses the scientific method and metrics to ensure statistical validity. That means you can explain what hypotheses you tested and what the result was. There’s no "themes emerged from our focus groups." It’s hard data, not interpretation.
- Adapts to fast-moving situations. The world is changing fast these days. Multivariate behavioral testing creates a real-world baseline. Because it’s fast, subsequent tests can track changes to behavior by audience and identify the variables that are most affected so that you can test new values to get ahead of market shifts.
- Dodges the Hawthorne Effect. Most other behavioral research (ethnographic and observational research) takes place in controlled environments, which creates biased results since subjects know they are being observed. Online behavioral experimentation captures data in a 100% natural environment.
- Aligns incentives. Conventional research firms get paid to deliver a report. Behavioral testing is about testing multiple strategies in parallel so that you find one that works.
--
The Fastest Little Pig: Why Strategy Needs a Race, Not a Decree
Bake-offs, brackets, cage-fights, and contests. Duels, debates, pageants, and pig races.
Competition is so deeply embedded in our culture that just about everything is organized like a NASCAR race. We're a country of leaderboards: followers on social media, billionaires jockeying for a top-five slot, NFL draft picks, the fastest fast-food drive-throughs (Taco Bell is tops, if you're wondering).
And yet most strategic decisions within companies are almost entirely devoid of competition. Instead of running the pigs, someone simply anoints one—the pig with the most potential, the best pedigree, the most persuasive sponsor—and declares it the winner before the race has started.
--
The slide deck as a substitute for evidence
Walk into any strategy presentation and you'll find the same underlying logic driving the slide deck. It goes something like this: We know the industry (Premise A). We surveyed 1,500 people (Premise B). Therefore, this $50 million investment will work (Conclusion).
This is deductive reasoning, and it's the dominant operating system of corporate strategy. It has antecedents in the consulting industry, where Barbara Minto pioneered The Pyramid Principle at McKinsey before she was laid off in 1973. Develop a hypothesis, gather evidence that supports it, build a convincing, if rhetorical, argument. It feels rigorous and, when delivered well, it blows everyone away.
The problem is that arguments like these are only as good as their premises—and in innovation, marketing, and brand strategy, the premises are almost always weaker than they appear. "We know the industry" often means we know how the industry worked last year, or five years ago, before the cultural and competitive shifts that are now rewriting the rules. "We surveyed 1,500 people" means we collected ‘stated preferences’—i.e., what people say they'll do, which is a notoriously unreliable predictor of what people actually do when faced with a real choice in the real world.
Put two shaky premises together and you get a conclusion that feels defensible but is, at its core, theoretical. It ignores a fundamental truth about markets: they are complex adaptive systems, constantly changing, full of interactions and feedback loops that no survey or framework can fully anticipate. The fastest pig is not always the one that looked fastest in the paddock.
--
What competition actually looks like in strategy
If we believe in competition as a mechanism for finding the best answer in almost every other domain, why don't we apply it to strategy itself?
Instead of building the case for a single strategic direction and defending it through layers of approval, what if you ‘ran the pigs’? Developed three or four genuinely distinct strategic hypotheses and expressed them through rigorously applied variables like positioning, creative style, messaging. And then tested them against each other in the real world, with real audiences making genuine behavioral choices?
That’s factorial testing. Rather than the sequential logic of A/B testing—one variable at a time, one question at a time—factorial design lets you put multiple strategies into competition simultaneously and to understand how applying different variables (creative style, etc.) make them perform better or worse. A statistically valid sample of every audience sees every concept. The interactions between strategy and audience become visible, and you learn how different variables (emotion vs utility, for example) drive success metrics.
To strain the metaphor: you don't just learn which pig won; you learn which pig runs fastest on which track, in which conditions, against which field.
The result is something you just don’t see in slide decks: behavioral evidence. Not what people said they preferred in a survey but what they actually did when presented with a real choice.
--
More pig races, please
Most companies compete in their markets: they track competitors’ moves, fight for tiny slivers of share, set aggressive growth targets. But internally, when it comes to the strategic decisions that determine how they'll compete, competition is extremely limited. A single team develops a single strategy. The strategy gets refined through internal review meetings. If the deck is convincing enough, the strategy gets approved and the money gets committed.
It’s a missed opportunity to create better strategy.
Developing the habit of putting strategic hypotheses into real-world competition before committing to them is one of the most undervalued advantages available to any organization right now. It's faster and more doable than it sounds. It's cheaper than the alternative. And because it’s based on experimentation, not argument, it generates a different conversation in a board meeting, a budget review, or a tough conversation with a CFO who wants proof that a strategy is really going to work.
May the best pig win. The only way to find it is to run the race.