🎲 Statistical Significance

Statistical Significance#

Evidence that an observed effect is unlikely under the null hypothesis, judged against a chosen threshold.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

A result is statistically significant when it is unlikely to have arisen by random chance alone, assuming the null hypothesis \(H_0\) is true — operationally, when the p-value \(\le \alpha\), the predefined significance level. It answers one narrow question: is this result sufficiently inconsistent with \(H_0\)?

What it does not tell you#

Significance says nothing about how large the effect is, whether it matters, or whether it will replicate. And the p-value is widely misread: it is the probability, under \(H_0\), of data as extreme or more extreme than observed — not the probability that \(H_0\) is true, nor the probability the result is “due to chance.”

Statistical vs practical significance#

These come apart. Statistical significance is about detectability and depends heavily on sample size; practical significance is about real-world importance and depends on effect size and context. With a large enough \(n\), a trivial effect becomes significant; with a small \(n\), a meaningful one may not — so a result can be significant yet practically meaningless.

Significance vs power#

Significance is a binary outcome (yes/no); power is the probability of achieving it when a real effect exists. High power makes a true effect likely to register; under low power, a non-significant result is ambiguous (it may just reflect too little data).

A decision rule, not a verdict#

Treat significance as a decision rule for controlling false positives under repeated use — part of a risk-management system, not a proof of truth. The \(\alpha = 0.05\) line is a convention, not a law: “significant” is not “important,” and “not significant” is not “no effect.” Good practice reports effect sizes, confidence intervals, and power alongside it, never significance alone.

Theme: Probability & Statistics Foundations · All terminology

Hint

Mind map — connected ideas

P-Value (probability value) · Type I Error · Statistical Power · Effect Size (δ) · Frequentist · A/B Testing

Hint

More in Probability & Statistics Foundations

Beta Distribution · Confidence Level · Correlation · Critical Value · Cumulative Distribution Function (CDF) · Frequentist · IID (Independent and Identically Distributed) · Likelihood · Margin of Error (MoE) · Mean · Median · Normal Distribution · Outlier · Population Proportion