Data Science & Statistics Interview Questions and Answers

When you run many hypothesis tests, the chance of at least one false positive grows quickly. With 20 independent tests at alpha 0.05, the probability of a false positive is about 1 - 0.95^20, roughly 64 percent. Reporting the single significant result is misleading.

Mitigations:

• Bonferroni correction: divide alpha by the number of tests, controlling the family-wise error rate but conservative.
• Holm-Bonferroni: a step-down improvement that is uniformly more powerful.
• Benjamini-Hochberg: controls the false discovery rate, the expected proportion of false positives among rejections, better for large exploratory screens such as genomics.

from statsmodels.stats.multitest import multipletests
reject, p_adj, _, _ = multipletests(p_values, method="fdr_bh")

Pre-register hypotheses and distinguish confirmatory from exploratory analyses. Segment mining in A/B tests is a common trap.

Simpson's paradox is when a trend appears in several groups but reverses or disappears when the groups are combined. It happens because a confounding variable is unevenly distributed across groups and is related to the outcome.

Classic example: a treatment appears to have a higher success rate overall, but within each severity level it actually performs better. The treatment group contains more mild cases, which inflates its overall rate.

df.groupby("severity").agg(success=("success", "mean"))

Implications: always examine the relevant subgroups and think about causal structure before aggregating. But do not automatically prefer the disaggregated view either, because conditioning on a collider or a mediator can create its own bias.

The lesson is that the unit of analysis and the adjustment set are modelling decisions that should follow from a causal diagram, not from whichever result looks better.