A variant failing before exposure logging may look better because affected shoppers disappear from the dataset.
Allocation imbalance: check SRM before interpreting effects
Check observed participant counts against planned allocation. Unexplained imbalance can make an experiment unreliable.
Research context: Microsoft Research · Reliable online experiments ↗
Plan data-quality checks before effect analysis. Never repair allocation by dropping inconvenient rows afterward.
Try the pattern
Task and expected result
- 01
Write the expected allocation and applicable events.
- 02
List causes such as bot filtering, logging failure and variant crashes.
- 03
Decide that unexplained quality issues block winner claims.
How to investigate value
Define audience, exposure, denominator and primary outcome before starting. Record what could invalidate the data.
Watch for side effects
Data loss, incorrect variant allocation, returns/refunds, margin, accessibility and mobile performance.
This is a proposed investigation, not a measured result. Choose one hypothesis and define the audience before testing.
Build a measurement plan for this concept ↗