Run a mock round

Bayes theorem updates a belief given evidence: P(A|B) = P(B|A) * P(A) / P(B). The prior P(A) is your initial belief, the likelihood P(B|A) is how probable the evidence is under A, and the posterior P(A|B) is the updated belief.

Example: a disease affects 1 percent of people. A test is 99 percent sensitive and has a 5 percent false positive rate. If you test positive, what is the chance you have it?

p_disease = 0.01
sensitivity = 0.99
false_pos = 0.05
p_pos = sensitivity*p_disease + false_pos*(1-p_disease)
p_posterior = sensitivity*p_disease / p_pos
# 0.99*0.01 / (0.99*0.01 + 0.05*0.99) = 0.1667

Even with a positive test the probability is only about 17 percent, because the disease is rare. This base rate effect is why screening uses confirmatory tests and why base rates matter in fraud and spam detection.

Both test hypotheses about means, and the choice hinges on whether the population standard deviation is known and the sample size.

Use a z-test when the population standard deviation is known and the data are normal, or when the sample is large enough that the sample standard deviation is a good estimate thanks to the Central Limit Theorem, typically n greater than 30 or 50.

Use a t-test when the population standard deviation is unknown and the sample is small. The t-distribution has heavier tails that account for the extra uncertainty of estimating the variance from data; as n grows it converges to the normal.

from scipy import stats
stats.ttest_ind(group_a, group_b, equal_var=False)

Other choices matter too: paired tests for matched observations, Welch's t-test when variances differ, and non-parametric tests such as Mann-Whitney for heavily skewed or ordinal data.

A confidence interval gives a range of plausible values for a population parameter, along with the uncertainty in the estimate. A 95 percent interval means that if we repeated the study many times, about 95 percent of the intervals we constructed would contain the true parameter.

from scipy import stats
mean, se = data.mean(), stats.sem(data)
ci = stats.t.interval(0.95, len(data)-1, loc=mean, scale=se)

It is not the probability that the true value lies in this specific interval under a frequentist view, though that is the common shortcut. Wider intervals mean more uncertainty, coming from smaller samples or noisier data.

Overlapping intervals between two groups suggest no clear difference, but the correct test compares the difference directly rather than eyeballing overlap. Report the point estimate, the interval and the sample size together so stakeholders can judge practical significance.

Imagine we assume a change has no real effect, which is the null hypothesis. The p-value answers: if that were true, how surprising would the data we observed be? A p-value of 0.03 means that under the no-effect assumption we would see a result this extreme about 3 percent of the time by chance alone.

When the p-value is below a threshold, usually 0.05, we call the result statistically significant and reject the null hypothesis. It does not prove the effect is real, large or important; it only says the data are unlikely under the null. A tiny p-value with a trivial effect size can still be meaningless for the business.

Significance also depends on sample size and power. Always report the effect size and a confidence interval alongside the p-value, and remember that many tests increase the chance of a false positive.

Power is the probability of detecting a real effect of a given size, that is, correctly rejecting a false null hypothesis. The conventional target is 0.8, meaning an 80 percent chance of finding the effect if it exists.

Power depends on four linked quantities: significance level alpha, effect size, sample size and variance. Fix any three and the fourth follows. Larger effects, lower noise, higher alpha or more data all increase power.

from statsmodels.stats.power import TTestIndPower
n = TTestIndPower().solve_power(effect_size=0.3, alpha=0.05, power=0.8)

Consequences: an underpowered study may miss a real effect, and when it does find significance it tends to overestimate the magnitude. Always run a power calculation before collecting data, and acknowledge that small pilot studies rarely have enough power. Increasing duration or traffic is the usual remedy.

When you run many hypothesis tests, the chance of at least one false positive grows quickly. With 20 independent tests at alpha 0.05, the probability of a false positive is about 1 - 0.95^20, roughly 64 percent. Reporting the single significant result is misleading.

Mitigations:

• Bonferroni correction: divide alpha by the number of tests, controlling the family-wise error rate but conservative.
• Holm-Bonferroni: a step-down improvement that is uniformly more powerful.
• Benjamini-Hochberg: controls the false discovery rate, the expected proportion of false positives among rejections, better for large exploratory screens such as genomics.

from statsmodels.stats.multitest import multipletests
reject, p_adj, _, _ = multipletests(p_values, method="fdr_bh")

Pre-register hypotheses and distinguish confirmatory from exploratory analyses. Segment mining in A/B tests is a common trap.

Simpson's paradox is when a trend appears in several groups but reverses or disappears when the groups are combined. It happens because a confounding variable is unevenly distributed across groups and is related to the outcome.

Classic example: a treatment appears to have a higher success rate overall, but within each severity level it actually performs better. The treatment group contains more mild cases, which inflates its overall rate.

df.groupby("severity").agg(success=("success", "mean"))

Implications: always examine the relevant subgroups and think about causal structure before aggregating. But do not automatically prefer the disaggregated view either, because conditioning on a collider or a mediator can create its own bias.

The lesson is that the unit of analysis and the adjustment set are modelling decisions that should follow from a causal diagram, not from whichever result looks better.

A Type I error is a false positive: rejecting the null hypothesis when it is actually true. Its probability is the significance level alpha, usually 0.05. In a fraud system it means flagging a legitimate transaction.

A Type II error is a false negative: failing to reject the null when a real effect exists. Its probability is beta, and statistical power is 1 minus beta, the chance of detecting a true effect. In a medical test it means missing a real disease.

There is a tradeoff: lowering alpha reduces false positives but increases false negatives for a fixed sample size. Increasing sample size or effect size raises power.

from statsmodels.stats.power import TTestIndPower
n = TTestIndPower().solve_power(effect_size=0.2, power=0.8, alpha=0.05)

Choose alpha and power based on which error is more costly, and compute the sample size before the experiment.

Correlation means two variables move together; causation means one produces a change in the other. Correlation is symmetric and measurable with a coefficient, while causation is directional and requires a causal mechanism.

Why correlated variables may not be causal:

• Confounding: a third variable drives both, such as ice cream sales and drownings both rising in summer.
• Reverse causation: the outcome influences the predictor.
• Coincidence or selection effects.

df[["ad_spend", "revenue"]].corr()

To move toward causation, use randomized controlled experiments, or with observational data apply methods like difference-in-differences, instrumental variables, propensity score matching or regression discontinuity. Always plot the data and consider the mechanism. A high correlation coefficient is evidence of association, not proof of a causal relationship.