🟢 Stats: A/B Testing Design Checklist
- Define hypothesis: "Changing X will increase metric Y"
- Pick metrics: Primary, secondary, guardrail
- Calculate sample size: n ≈ 16σ²/δ² (α=0.05, power=0.80, δ = min detectable effect)
- Set duration: Minimum 2 weeks (capture weekly patterns)
- Randomize: By user (not session) — hash user ID for consistency
- Analyze at predetermined end date — don't peek
- Check statistical AND practical significance
Practice Questions
Q: Why randomize by user ID, not by session?