A random positive subgroup can appear when many segments are inspected simultaneously.
Experiment audiences before splitting results
Define eligibility before starting. Mobile, market and new shoppers can be useful segments, but post-hoc hunting creates false winners.
Research context: Microsoft Research · Reliable online experiments ↗
Use a primary analysis and a few justified segments. Treat other patterns as hypotheses for later work.
Try the pattern
Task and expected result
- 01
Write eligibility before completing the decision rule.
- 02
Describe whether allocation uses shoppers or sessions.
- 03
Label new segment findings as hypotheses rather than confirmed gains.
How to investigate value
Define audience, exposure, denominator and primary outcome before starting. Record what could invalidate the data.
Watch for side effects
Data loss, incorrect variant allocation, returns/refunds, margin, accessibility and mobile performance.
This is a proposed investigation, not a measured result. Choose one hypothesis and define the audience before testing.
Build a measurement plan for this concept ↗