Run a mock round

Linear regression models a continuous target as a linear combination of predictors plus error: y = b0 + b1*x1 + ... + e. Coefficients are estimated by ordinary least squares, which minimizes the sum of squared residuals, or by maximum likelihood.

from sklearn.linear_model import LinearRegression
model = LinearRegression().fit(X, y)

Assumptions:

• Linearity between predictors and the mean of the target.
• Independence of errors, violated by time series or clustered data.
• Homoscedasticity: constant error variance.
• Normally distributed errors, mainly for inference and intervals.
• Little multicollinearity; check variance inflation factors.
• No influential outliers distorting the fit.

Violations affect inference more than prediction. Diagnose with residual plots, QQ plots and influence measures. Remedies include transformations, robust standard errors and adding interaction or polynomial terms.

Start with a clear hypothesis and a primary metric tied to the business, plus guardrail metrics for regressions.

Key steps:

1. Define the unit of randomization, usually the user, and keep it stable across the test.
2. Compute sample size from the minimum detectable effect, power of 0.8, alpha of 0.05 and baseline variance. Underpowered tests waste traffic.
3. Randomize with a hash so assignment is consistent and balanced.
4. Run for at least one full business cycle, avoiding peeking that inflates false positives.
5. Analyze with the pre-declared metric and test, using intention-to-treat.

treatment = hash(user_id) % 100 < 50

Watch for novelty effects, sample ratio mismatch, and interactions between concurrent tests. Segment analyses are exploratory unless pre-registered. Combine statistics with judgment about practical significance before shipping.

Imagine we assume a change has no real effect, which is the null hypothesis. The p-value answers: if that were true, how surprising would the data we observed be? A p-value of 0.03 means that under the no-effect assumption we would see a result this extreme about 3 percent of the time by chance alone.

When the p-value is below a threshold, usually 0.05, we call the result statistically significant and reject the null hypothesis. It does not prove the effect is real, large or important; it only says the data are unlikely under the null. A tiny p-value with a trivial effect size can still be meaningless for the business.

Significance also depends on sample size and power. Always report the effect size and a confidence interval alongside the p-value, and remember that many tests increase the chance of a false positive.

Correlation means two variables move together; causation means one produces a change in the other. Correlation is symmetric and measurable with a coefficient, while causation is directional and requires a causal mechanism.

Why correlated variables may not be causal:

• Confounding: a third variable drives both, such as ice cream sales and drownings both rising in summer.
• Reverse causation: the outcome influences the predictor.
• Coincidence or selection effects.

df[["ad_spend", "revenue"]].corr()

To move toward causation, use randomized controlled experiments, or with observational data apply methods like difference-in-differences, instrumental variables, propensity score matching or regression discontinuity. Always plot the data and consider the mechanism. A high correlation coefficient is evidence of association, not proof of a causal relationship.

Both test hypotheses about means, and the choice hinges on whether the population standard deviation is known and the sample size.

Use a z-test when the population standard deviation is known and the data are normal, or when the sample is large enough that the sample standard deviation is a good estimate thanks to the Central Limit Theorem, typically n greater than 30 or 50.

Use a t-test when the population standard deviation is unknown and the sample is small. The t-distribution has heavier tails that account for the extra uncertainty of estimating the variance from data; as n grows it converges to the normal.

from scipy import stats
stats.ttest_ind(group_a, group_b, equal_var=False)

Other choices matter too: paired tests for matched observations, Welch's t-test when variances differ, and non-parametric tests such as Mann-Whitney for heavily skewed or ordinal data.

When you run many hypothesis tests, the chance of at least one false positive grows quickly. With 20 independent tests at alpha 0.05, the probability of a false positive is about 1 - 0.95^20, roughly 64 percent. Reporting the single significant result is misleading.

Mitigations:

• Bonferroni correction: divide alpha by the number of tests, controlling the family-wise error rate but conservative.
• Holm-Bonferroni: a step-down improvement that is uniformly more powerful.
• Benjamini-Hochberg: controls the false discovery rate, the expected proportion of false positives among rejections, better for large exploratory screens such as genomics.

from statsmodels.stats.multitest import multipletests
reject, p_adj, _, _ = multipletests(p_values, method="fdr_bh")

Pre-register hypotheses and distinguish confirmatory from exploratory analyses. Segment mining in A/B tests is a common trap.

The Central Limit Theorem says that the sampling distribution of the sample mean approaches a normal distribution as the sample size grows, regardless of the shape of the population distribution, provided observations are independent and the variance is finite.

means = [np.mean(np.random.exponential(2, 50)) for _ in range(10000)]

Implications: it justifies normal-based confidence intervals and t-tests for means even when the underlying data are skewed, as long as n is reasonably large. The common rule of thumb is n greater than about 30, but heavily skewed or heavy-tailed data need more.

It also explains why the standard error shrinks with the square root of n: quadrupling the sample halves the margin of error. It does not apply directly to small samples, dependent data, or statistics other than means without care.

A Type I error is a false positive: rejecting the null hypothesis when it is actually true. Its probability is the significance level alpha, usually 0.05. In a fraud system it means flagging a legitimate transaction.

A Type II error is a false negative: failing to reject the null when a real effect exists. Its probability is beta, and statistical power is 1 minus beta, the chance of detecting a true effect. In a medical test it means missing a real disease.

There is a tradeoff: lowering alpha reduces false positives but increases false negatives for a fixed sample size. Increasing sample size or effect size raises power.

from statsmodels.stats.power import TTestIndPower
n = TTestIndPower().solve_power(effect_size=0.2, power=0.8, alpha=0.05)

Choose alpha and power based on which error is more costly, and compute the sample size before the experiment.

Bayes theorem updates a belief given evidence: P(A|B) = P(B|A) * P(A) / P(B). The prior P(A) is your initial belief, the likelihood P(B|A) is how probable the evidence is under A, and the posterior P(A|B) is the updated belief.

Example: a disease affects 1 percent of people. A test is 99 percent sensitive and has a 5 percent false positive rate. If you test positive, what is the chance you have it?

p_disease = 0.01
sensitivity = 0.99
false_pos = 0.05
p_pos = sensitivity*p_disease + false_pos*(1-p_disease)
p_posterior = sensitivity*p_disease / p_pos
# 0.99*0.01 / (0.99*0.01 + 0.05*0.99) = 0.1667

Even with a positive test the probability is only about 17 percent, because the disease is rare. This base rate effect is why screening uses confirmatory tests and why base rates matter in fraud and spam detection.