🎲 Frequentist

Frequentist#

The school of statistics treating probability as long-run frequency and parameters as fixed unknowns.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

The frequentist approach defines probability as the long-run frequency of an event over infinitely many repeated trials. Parameters — a population mean, a conversion rate, a treatment effect — are treated as fixed but unknown constants, and all the uncertainty comes from the randomness of the data (the sample), never from the parameter itself.

Key principles#

Probability = frequency — “P(heads) = 0.5” means about half of many flips land heads.
Parameters fixed, data random — the true mean \(\mu\) is a fixed constant; the sample mean \(\bar{x}\) varies from sample to sample.
Inference via repeated sampling — p-values, confidence intervals and tests are all defined by what would happen if the experiment were repeated many times.

The toolkit#

Point estimates (the sample mean \(\bar{x}\) for \(\mu\)), confidence intervals (an interval procedure that captures the true parameter in, say, 95% of repeated experiments), hypothesis tests (a null \(H_0\) and a p-value — the probability of data at least as extreme as observed, assuming \(H_0\)), and maximum likelihood estimation.

A subtlety worth stating#

A 95% confidence interval does not mean “95% probability the true value is in this interval” — under frequentism the parameter is fixed, so it’s either in or out. It means the procedure covers the truth 95% of the time across repetitions. The “95% probability the mean lies here” reading is the Bayesian credible interval.

Example — an A/B test#

To ask whether variant B beats A, treat the true rates \(p_A, p_B\) as fixed, estimate them with sample proportions \(\hat{p}_A, \hat{p}_B\), run a two-proportion z-test, and reject \(H_0\) if the p-value is below 0.05.

Frequentist vs Bayesian#

The two paradigms differ along clear lines: probability as long-run frequency vs degree of belief; parameters as fixed constants vs random variables; uncertainty from data only vs prior + data; inference via p-values and confidence intervals vs posteriors and credible intervals; and on sequential data, frequentist peeking inflates error (needs corrections) while Bayesian updating is continuously valid. Frequentist methods remain dominant in medicine, the social sciences and classic A/B testing.

Theme: Probability & Statistics Foundations · All terminology

Hint

Mind map — connected ideas

Bayesian Inference. · A/B Testing · Traditional A/B Test (Fixed-Horizon A/B Test) · Sequential Settings · Posterior

Hint

More in Probability & Statistics Foundations

Beta Distribution · Confidence Level · Correlation · Critical Value · Cumulative Distribution Function (CDF) · IID (Independent and Identically Distributed) · Likelihood · Margin of Error (MoE) · Mean · Median · Normal Distribution · Outlier · Population Proportion · Probability