Stop Using A/B Tests to Validate Bad Product Decisions
The Cost of Learning Too Late
Many product teams spend months building features based on stakeholder requests, competitor pressure, intuition, or assumptions about customer needs. Then they launch (ideally behind a validation test) and hope for the best.
But the industry has learned a humbling lesson over the last few decades:
Most features don't materially change customer behavior.
Yet many product teams still assume their features will. So, their goals are related to feature velocity.
But once again, the real opportunity isn't who can get the most features implemented the most quickly. It's improving at building a problem-solution idea pipeline that consistently uncovers important problems and identifies the best solutions to test before feature development and eventual validation. Because experimentation isn't a single activity at the end of product development. It's two complementary systems. One system creates innovation. The other protects it.
1. The Sunk Cost Fallacy
In my experience, Product Managers often use experimentation too late. It’s used as insurance. Risk mitigation. ROI capture (STOP IT! That’s NOT what the stats are saying! It’s about whether you can be confident the treatment is different - not if you can be confident that an AMOUNT IS REAL. Just STOP IT!)
Using your test results to calculate ROI: RED CARD!
Common PM workflow:
Identify feature
Prioritize feature
Design feature
Build feature
Test feature
Analyze results
Scramble if it doesn’t “win” 😅
Launch feature anyway 🫣
Experimentation becomes a hurdle to clear to launch a feature that has had too much work put into it to abandon. A way to measure the financial impact of work already completed. STOP IT!
Incredibles: I wish I were Edna Mode sometimes. STOP IT!
The result is that teams start optimizing for the wrong outcomes:
Number of features shipped
Roadmap completion
Experiment win rate
Test velocity
Instead of:
Retention
Conversion rate
User experience
Customer value
Business impact
The irony is that by the time you're testing a fully built feature, most of the cost has already been incurred. The goal in the product world is: build → measure → learn. But while that’s the plan, it’s not often the outcome. What are you learning from risk mitigation testing?
I suggest flipping the script to more closely align with the scientific method:
What do we need to learn to make a decision?
What would we need to measure to learn that?
What is the smallest thing we need to build to measure that?
Build only that minimal thing
Measure
Learn!
Iterate (of course)
If we start there, we can build minimally viable versions of features to pressure test truly innovative ideas and find out what actually works. And we can do that BEFORE the investment to build new products or capabilities, only to learn that they do not resonate with your customer base (which frequently leads to the Sunk Cost Fallacy of Product Management: if they build it, we must find a metric or segment that it works for. This is product’s version of p-hacking. Instead of asking, “Did this solve the customer problem?”, teams ask, “Can we find a segment where it looked like it helped so we can ship?”).
Futurama: “not sure” meme with Sunk Cost Fallacy
There are two categories of experiments all product managers should be running.
Innovation experiments: “Should we build this?”
Lower certainty
Higher learning
Bigger opportunities
Validation experiments: “Did we build it correctly?”
Higher certainty
Limited learning
Risk mitigation
2. Innovation Experimentation for Growth
Experimentation shouldn't start after a product or feature is already built. It should begin when a problem or opportunity is identified. Innovation experiments should help a product team learn what works, what does not, and where they should truly invest.
Questions become:
Is this a real customer problem?
How painful is it?
Which part of the problem matters most?
Which solution elements create the most value?
Every innovation experiment should make the next experiment, the next feature, smarter.
3. Test the Idea Before You Build the Feature
In the product world, this type of innovation experimentation is not new. It’s often referred to as “Lean Experimentation”. And A/B tests are just one type. A few quick additional examples:
Fake Door Tests: Measure interest before development with a feature you don’t even develop. Just put a link on the website and count clicks to gauge interest.
Concierge Tests: Manually deliver a complex backend to make sure the cost of development will be worth it.
Rapid Prototypes: Test workflows before engineering with real customers, often using paper prototypes or anything easy to revise based on customer real-time feedback.
Component Testing: Instead of testing the full feature, test components to evaluate which components matter most (can be inside an A/B test).
In each of these cases, the goal is to evaluate a solution before you fully build it, and (here’s the important part) adjust your plans based on what you learn.
4. Experiment Your Way to Better Product Management
Traditional PM:
Problem → Feature requested → Build → Validation test
Experiment-led PM:
Problem → Innovation test 1 → Learn → Iterative testing → Learn → Build final feature → Validation test
By the time development on your full-functioning feature starts:
You know customers care
You know which parts matter most
You know what success looks like
You know exactly how to measure it
You're now using sprint cycles to innovate, learn, improve, and then validate that the features you know matter to your customers are built correctly.
5. The Goal Isn't Winning Experiments
Many leaders assume:
High win rate = good
Low win rate = bad
Reality is more nuanced. A team with a 60% win rate may simply be making tiny, low-risk changes. Or only using validation experiments to manage risk.
A team with a 10% win rate may be taking innovative, meaningful swings that create breakthroughs, but they are failing to run validation tests or simple iterations to scale a learning across regions, products, or audiences.
My proposal is not to stop mitigating risk with validation tests. Far from it!! I actually argue that validation testing is the most important part of product management. But:
By the time a feature reaches a validation test, we’ve already spent weeks or months learning our way toward the solution.
Before we get to the validation test, we have many failed innovation treatments to learn from, and we should expect a much higher win rate because we've tested our way to the solution. But we’ve also learned more about which features should receive investment and expansion, and which may not be worth further development.
6. Judgment is the New Scarce Resource
Historically, finding customer problems and solutions was more difficult. There was behavioral analysis, survey research, usability testing, competitive analysis…. But all of it required something many teams didn’t have the luxury of - time.
Today, AI has dramatically reduced the time required. Pull customer reviews from your site, from Reddit. Grab support tickets, chat transcripts, session recordings, survey responses, forums, even sales call transcripts. Throw it all into your AI tool of choice and, with solid prompt engineering, ideally including your history of test results and potentially information about your business goals, you have the ultimate customer-first problem-solution framework to build a hypothesis library of ideas to prioritize!
AI doesn't generate insights. It can only surface patterns. It takes humans to read, assess, prioritize, and build the experiments to validate the patterns that were surfaced. One more time for those in the back: AI can and should be your co-pilot. But humans working with AI will build better tests, write better copy, and see greater business outcomes than those who jump on the “vibe everything” wagon.
Given sufficient high-quality data inputs, AI can generate hundreds of high-quality hypotheses in a few hours. The scarce resource is no longer time. It’s judgment. And we shouldn’t be asking the AI to judge itself any more than we ask a team member to QA their own work.
7. Validation Tests Are Guardrails, Not Innovation
Most organizations see any experiment as innovation. And it's true that mitigating risk by putting every release behind a validation test gives you the cover to try innovative things.
However, in practice, that whole Sunk Cost Fallacy of Product Management tends to leave most companies somewhat stuck, often digging a deeper hole, attempting to validate the investment.
Innovation comes from innovation experiments. It's where you take big swings, find what matters, and iterate before final build.
Think of innovation tests as your GPS guiding you where you go next.
In contrast, validation tests are the check engine light. They can't tell you where to go. They just tell you when to stop the car and fix the engine.
Validation tests answer:
Did we break anything?
Did adoption occur?
Did unintended consequences emerge?
The innovation already happened earlier. The breakthrough insight was discovered before the feature was built (ideally). It's why we built the feature. Which is exactly why we expect higher win rates with validation tests.
If validation test win rates aren’t high, the organization probably isn't doing enough learning before development. If innovation test win rates are too high, the organization probably isn’t taking big enough swings.
8. A New PM Playbook
The best PMs of the next decade won't be the ones who ship the most features.
They'll be the ones who recognize the need to combine innovation with validation. The ones that take risks and mitigate them. The ones that accelerate their innovation and analysis with an AI co-pilot, but keep a human in the driver seat.
Experimentation doesn't slow product growth. Done right, experimentation fuels the acceleration.
