In this era of unprecedented uncertainty, many factors are coming together to cause a revolution in the research and insights sector. These factors are coming together to influence Lean Research that is experiment-driven and focused on setting up experiments to see what consumers will do in response to an offer, a new product in their hands, or another behavioral way to test the critical assumption.
- First and foremost, the reality that the world is more volatile, uncertain, complex and ambiguous (VUCA) than ever before and that one test to validate an idea isn’t sufficient.
- Advances in behavioral science and behavioral economics have demonstrated that when consumers are asked what they want and if they’ll buy something in the future, that data is rarely sound
- Tech-driven solutions allow on-demand access to participants and lots of options that give tremendously valuable insights outside of “research” contexts (think Kickstarter, social media and more)
These factors are coming together to influence Lean Research that is experiment-driven, and breaks down the proposition into the riskiest pieces or assumptions, and tests those in order of riskiest to least risky. This prioritizes research time spent and gets ideas to market faster, or conversely, helps teams identify ideas to abandon more quickly because they haven’t been fully built out and brought through the launch cycle before realizing that they aren’t able to meet key hurdles of desirability, viability or feasibility. In this video, we unpack more about experiment-driven research and the testing of critical assumptions. Behavioral science advancements have caused the second mindset shift: as researchers, we’re responsible for looking at what people do, not what they say they’ll do. With this mindset, tests can be set up to see what consumers will do in response to an offer, a new product in their hands, or other behavioral ways to test the critical assumption.
As we talk with innovation and insights leaders, these shifts in research paradigms have provoked lots of questions about how exactly these types of experiments come to life, and what principles to follow when thinking about what sort of experiment to construct to test your riskiest assumption.
Coming from the traditional research world where specific methods are qualified for testing specific objectives, this experiment-driven research world can feel like a bit of a “wild west” where anything goes. Early on, everything in Lean Startup revolved around developing an MVP, or minimum viable product. That was a pretty linear solve; go make a low fidelity version of the product that a consumer can interact with and insert that prototype into their life to see what they do with it. While this is a great experiment type, it’s important to note that it’s only one potential experiment type. Things felt off for us at The Garage Group pretty early on in MVP thinking when building an MVP didn’t actually give data for the riskiest assumption, and/or building an MVP meant over-building and taking too much time or money, when another experiment type could have been leaner and still high rigor.
Here is a core set of principles that give guardrails when thinking about what sort of experiment to construct to test your riskiest assumption (or biggest risk to your business model’s desirability, viability, or feasibility.)
- In-the-Wild or Recruited? The reality of BigCo front-end-innovation is that it might not make sense right out of the gate to test your riskiest assumption in a public arena. The great news is that there are a variety of experiments that can be run in a recruited context. Ultimately, it’s about understanding the benefits and drawbacks of In-the-Wild vs Recruited research contexts and choosing the right scenario for the assumption that’s being tested and the place your team is at on their trajectory towards launch. Check out the table below for some of the ways we think about comparing the two options.
- Rigor, Cost, Time, & Context. Any given assumption will have a variety of ways that an experiment could be run. We’ve seen our BigCo clients’ experiments be less helpful to the ultimate decision-making process when the experiment type was too rogue for the growth board, leadership team, or other decision-making entity. We first heard the high rigor, low cost, low time idea from Strategyzer and David Bland. Giff Constable, writer of Testing with Humans, teaches teams to brainstorm at least six different potential experiment types. Then, stepping back, teams should ask themselves, “which of these is the highest rigor, lowest cost, lowest time experiments that will yield data that our organization will accept?” This is the context piece. Taking the context of what they’re willing to accept into consideration can save the team from headaches later on.
- Building a Body of Evidence. As we mentioned in the first post of this series, the beautiful thing about experiment-driven research is that the pressure is not on one singular test to be the end-all, be-all decision-making tool. Instead, it’s about building a body of evidence that helps inform decision-making along the way, as the product or service is iteratively de-risked. This is another big mindset shift for traditional market researchers, because we’re used to thinking of a singular test to validate the product or service, and we may even base forecasting off of that test. A massive benefit of this “building a body of evidence” approach is that uncertainty is reduced (quickly!) and based on the iterative arc of the product or service, not the version that was “validated” several months (or years!) prior to launch.
“You are looking for clues that help confirm or deny your assumptions….your goal is not to compile statistically significant answers. Instead you want to look for patterns that will help you make better decisions.” – Giff Constable
We’ll be back next month with another install into this Lean Research series: examples of types of experiments that we’ve run with clients over the past several years.
We don’t want to leave you hanging, though! If you’re on a BigCo team and are working to implement this experiment-driven Lean Research, check out our research capabilities to see how we might be able to help!
