How do you design a reliable A/B test?
Interviewer Expectations for this Question
01
Core Competency
Assesses fundamental understanding of Data Science & Statistics conventions, runtime behavior, and memory/performance considerations.
02
Evaluation Criteria
Hiring managers look for precision, avoidance of ambiguous jargon, and ability to explain trade-offs under real production conditions.
Comprehensive Model Answer
Verified Solution
Start with a clear hypothesis and a primary metric tied to the business, plus guardrail metrics for regressions.
Key steps:
- Define the unit of randomization, usually the user, and keep it stable across the test.
- Compute sample size from the minimum detectable effect, power of 0.8, alpha of 0.05 and baseline variance. Underpowered tests waste traffic.
- Randomize with a hash so assignment is consistent and balanced.
- Run for at least one full business cycle, avoiding peeking that inflates false positives.
- Analyze with the pre-declared metric and test, using intention-to-treat.
treatment = hash(user_id) % 100 < 50
Watch for novelty effects, sample ratio mismatch, and interactions between concurrent tests. Segment analyses are exploratory unless pre-registered. Combine statistics with judgment about practical significance before shipping.
Candidate Response Strategy & Interview Tips
- Start with a concise one-sentence summary: Deliver a direct, confident answer first before expanding into nuances.
- Demonstrate real-world trade-offs: Discuss where this approach excels and when you would avoid it in production systems.
- Discuss complexity & edge cases: Proactively explain time/space complexity or boundary conditions (null values, scale limits).
- Prepare for interviewer follow-ups: Technical hiring panels frequently probe deeper into concurrency, backward compatibility, or alternative libraries.