Hi there -
Here is this week’s “1 principle, 2 strategies, and 3 actionable tactics” for running lean…
1 Universal Principle
“Learn qualitatively, verify quantitatively.”
(Share this on Twitter)
Consider the following scenario:
You pitch your product for the first time to 10 people over a week. Which result is statistically significant?
A) 6 out of 10 people buy
B) 7 out of 10 people buy
C) 8 out of 10 people buy
Answer: None of the above.
You need to get at least 9 out 10 people to buy to pass the mathematical definition of statistical significance!
But, a 60% conversion rate for a new product is an excellent result out of the gate (and even at scale). So what gives?
Aiming for validation through statistical significance is the wrong goal when dealing with small sample sizes. It helps to recalibrate your expectations and even remove “validation” from your vocabulary in favor of a two-stage approach: Learn qualitatively, verify quantitatively.
2 Underlying Strategies at Play
I. Founders and scientists have different goals.
While methodologies like Lean Startup draw a comparison between science and entrepreneurship, the goals of scientists and entrepreneurs are quite different.
Scientists chase perpetual truths (universal laws) and can afford to spend a lifetime evolving their models to get there. On the other hand, entrepreneurs chase temporal truths (insights) and must rush to build repeatable and scalable business models before running out of resources.
II. Qualitative learning is the fastest (and only practical) way to gather insights at the early stage.
Metrics can tell you what just happened, but getting to causality requires large sample sizes and good experiment design (testable hypothesis, A/B tests, statistical significance, etc.)
These conditions are rarely practical at the early stage of a product unless you already have a pre-established channel to many customers. Even so, qualitative experiments will typically reveal insights faster than quantitative experiments because of your ability to ask customers why.
Speed of learning is the new unfair advantage.
In the scenario above, each customer pitch is an opportunity to understand why customers buy or don’t buy your product. Conversations versus landing pages afford more freedom for exploration. And, patterns start to emerge quickly that shape your model of an ideal customer (insights).
Even though these insights aren’t yet statistically significant, the added context from deeper exploration boosts the confidence level or evidence strength of your insights.
More importantly, you can just as quickly test these insights in follow-on conversations and see if they stick.
3 Actionable Tactics
I. Establish a weekly baseline.
Repeatability is the first condition to establish. If you can build a system that lets you repeatably pitch to even 10 prospects a week (in person), you’re off to a great start.
II. Chase causality.
Don’t settle for random results or assume you know why customers behave the way they do. Err on the side of asking more why-directed questions — even for seemingly obvious behaviors.
Collecting signals is the first step; finding the right signals in the noise is the next.
III. Double down on signals that stick.
Quickly turn your insights into mini-campaigns that you (qualitatively) A/B test against your weekly baseline. If you measure a positive spike, don’t prematurely declare validation, but verify repeatability over subsequent weeks. Then scale the campaign.
That's all for today. See you next week.
Ash
.
.
P.S.
Repeatability is a pre-condition for scalability.