What is Simpson paradox?
Assesses fundamental understanding of Data Science & Statistics conventions, runtime behavior, and memory/performance considerations.
Hiring managers look for precision, avoidance of ambiguous jargon, and ability to explain trade-offs under real production conditions.
Simpson's paradox is when a trend appears in several groups but reverses or disappears when the groups are combined. It happens because a confounding variable is unevenly distributed across groups and is related to the outcome.
Classic example: a treatment appears to have a higher success rate overall, but within each severity level it actually performs better. The treatment group contains more mild cases, which inflates its overall rate.
df.groupby("severity").agg(success=("success", "mean"))
Implications: always examine the relevant subgroups and think about causal structure before aggregating. But do not automatically prefer the disaggregated view either, because conditioning on a collider or a mediator can create its own bias.
The lesson is that the unit of analysis and the adjustment set are modelling decisions that should follow from a causal diagram, not from whichever result looks better.
Candidate Response Strategy & Interview Tips
- Start with a concise one-sentence summary: Deliver a direct, confident answer first before expanding into nuances.
- Demonstrate real-world trade-offs: Discuss where this approach excels and when you would avoid it in production systems.
- Discuss complexity & edge cases: Proactively explain time/space complexity or boundary conditions (null values, scale limits).
- Prepare for interviewer follow-ups: Technical hiring panels frequently probe deeper into concurrency, backward compatibility, or alternative libraries.