Framework · A/B Testing
Most A/B tests answer questions nobody asked.
A discipline for testing the things that actually move decisions — and reading the results without fooling yourself.
A/B testing is where good intentions go to produce noise. The framework isn’t statistical — it’s about testing hypotheses that matter, sizing them honestly, and resisting the urge to believe a flattering result.
A test you’d ignore if it lost isn’t a test. It’s a search for permission.
1 · Start with a real hypothesis
Not ‘let’s try a green button’ but ‘we believe people hesitate because X, so changing Y will help.’ If you can’t say why it might work, don’t test it.
2 · Size it honestly
Small traffic means only big effects are detectable. If your test could only ever produce a 2% lift on modest traffic, you’ll never see it — test something bigger or don’t bother.
3 · Decide the stopping rule first
Set the sample and duration before you start, and don’t peek-and-stop the moment it looks good. Stopping when you like the number is how noise becomes ‘results’.
4 · Read it against the hypothesis
A win that doesn’t match your reasoning is a coincidence until proven otherwise. Learning why beats a lift you can’t explain or repeat.
Keep a log of every test and its hypothesis. The compounding value isn’t any single win — it’s the pattern of what your audience keeps telling you.
Common questions
What makes a good A/B test?
A real hypothesis about why a change will work, enough traffic to detect the effect, a stopping rule set in advance, and a result read against the hypothesis rather than for a flattering number.
Why do most A/B tests fail?
They test trivial changes on too little traffic, stop the moment the number looks good, and can’t explain why the winner won — so the ‘win’ doesn’t repeat.
Related: the CRO framework · what people actually do.