🧮 Hypothesis Testing

Hypothesis Testing#

A framework for deciding between a null and an alternative using data.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

Hypothesis testing is a formal procedure for deciding about a population parameter from sample data while controlling the risk of error. It is best read not as proof but as risk-managed decision-making.

The two hypotheses#

The null \(H_0\) is the status quo — no effect, zero difference, zero coefficient. The alternative \(H_1\) is a departure from it, either two-sided (\(\neq\)) or one-sided (\(>\) or \(<\)). The test never proves \(H_1\); it asks whether the data are inconsistent with \(H_0\).

The logic#

A proof-by-contradiction: assume \(H_0\), ask how likely data this extreme (or more) would be under it, and if that likelihood is tiny, reject \(H_0\). The machinery is a test statistic,

\[\text{test statistic} = \frac{\text{observed effect} - \text{effect under } H_0}{\text{standard error}},\]

which under \(H_0\) follows a known sampling distribution (normal for z, t for t-tests, \(\chi^2\), F) — that is what lets us turn it into a p-value, the probability under \(H_0\) of data as extreme or more.

The two errors#

Decisions can fail two ways: a Type I error (reject a true \(H_0\), rate \(\alpha\)) and a Type II error (fail to reject a false one, rate \(\beta\)), with power \(= 1 - \beta\). The two trade off — tightening \(\alpha\) raises \(\beta\). A two-sided test is more conservative; a one-sided test has more power but needs strong prior justification.

What “fail to reject” means#

Crucially, failing to reject \(H_0\) is not evidence that \(H_0\) is true or that no effect exists — only that the data are insufficient against \(H_0\) at the chosen \(\alpha\). It could be a true null, a small effect, or simply too little power. And because large samples make trivial effects significant, always read the p-value alongside an effect size and a confidence interval — significance for whether, effect size for how much.

Theme: Statistical Inference & Power · All terminology

Hint

Mind map — connected ideas

Statistical Significance · Significance Level (α) · P-Value (probability value) · Type I Error · Power (1 – β) · Effect Size (δ)

Hint

More in Statistical Inference & Power

A Priori Power Analysis · Chi-square (χ²) Test · Clopper–Pearson Interval · Compromise Power Analysis · Confidence Intervals (CIs) · Effect Size (δ) · Kolmogorov–Smirnov (KS) Test · Minimum Detectable Lift (MDL) · P-Value (probability value) · Post Hoc Power Analysis · Power (1 – β) · Power Analysis · Sample size · Significance Level (α)