🧮 Type I Error

Type I Error#

Rejecting a true null hypothesis — a false positive, controlled at level alpha.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

A Type I error is a false positive: you reject the null hypothesis \(H_0\) when it is actually true — concluding there is an effect or difference when in reality there is none.

Its probability is α#

The probability of a Type I error is exactly the significance level \(\alpha\), fixed before the test: \(\alpha = 0.05\) accepts a 5% chance of wrongly rejecting a true \(H_0\); \(\alpha = 0.01\) a 1% chance. Choosing \(\alpha\) is choosing how often you are willing to cry wolf.

Examples#

Medicine — \(H_0\): the drug has no effect. If it truly doesn’t, but the data happen to give \(p < 0.05\), you reject \(H_0\) and declare it works — a Type I error.
A/B testing — \(H_0\): conversion rates are equal. If they really are, but random variation produces a “significant” gap, you’ve made a Type I error.

Geometrically, with overlapping \(H_0\) and \(H_1\) distributions, \(\alpha\) is the rejection region in the tail; a statistic landing there while \(H_0\) holds is the error.

Type I vs Type II vs power#

There are two ways to be wrong and one way the test “works”:

Type I (false positive) — reject a true \(H_0\); probability \(\alpha\).
Type II (false negative) — fail to reject a false \(H_0\); probability \(\beta\).
Power — correctly reject a false \(H_0\); equals \(1 - \beta\) (and grows with sample size, effect size and \(\alpha\)).

Lowering \(\alpha\) reduces Type I errors but, all else equal, raises \(\beta\) — the two trade off.

Controlling it#

Use a stricter \(\alpha\); apply Bonferroni or other multiple-testing corrections when running many tests; stick to fixed-horizon testing (no peeking), or an α-spending design if you must look early; and replicate to confirm.

Theme: Statistical Inference & Power · All terminology

Hint

Mind map — connected ideas

Frequentist · Traditional A/B Test (Fixed-Horizon A/B Test) · Stopping Rules · Group Sequential Testing · A/B Testing · Bayesian Sequential Testing

Hint

More in Statistical Inference & Power

A Priori Power Analysis · Chi-square (χ²) Test · Clopper–Pearson Interval · Compromise Power Analysis · Confidence Intervals (CIs) · Effect Size (δ) · Hypothesis Testing · Kolmogorov–Smirnov (KS) Test · Minimum Detectable Lift (MDL) · P-Value (probability value) · Post Hoc Power Analysis · Power (1 – β) · Power Analysis · Sample size