Post Hoc Power Analysis#
Computing achieved power after a study from the observed effect — widely criticised as uninformative.
Important
✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.
What it is#
Post-hoc power analysis computes the statistical power of a test after the study is finished, plugging in the observed sample size \(n\), the observed effect size \(\delta\), and the chosen \(\alpha\). It asks: given what we actually saw, what was the probability we could have detected an effect?
Why people run it#
Usually to interpret a non-significant result (“was it real-but-missed, or genuinely null?”), to satisfy a journal asking about sensitivity, or to judge older studies in a meta-analysis. The calculation is the a-priori one with the observed effect size substituted in:
Example#
If \(H_0\) is a 10% conversion rate, the treatment shows a tiny 10.2%, with 1,000 per group at \(\alpha = 0.05\), post-hoc power might be only 12% — the study was underpowered to detect so small a lift.
The tautology problem#
The deep flaw: post-hoc power is a deterministic function of the p-value, so it adds nothing. A non-significant result always yields low post-hoc power, and a significant one always high — it merely restates the test. Worse, it invites the fallacy “non-significant + low power ⇒ \(H_0\) is true,” when it only means “this study wasn’t sensitive enough.”
Report this instead#
Rather than post-hoc power, report the observed effect size (Cohen’s d, a difference in proportions, an odds ratio) and a confidence interval for the effect — these convey the strength and precision of the result without the circularity.
Theme: Statistical Inference & Power · All terminology
Hint
Mind map — connected ideas
A Priori Power Analysis · Compromise Power Analysis · Statistical Power · Effect Size (δ) · P-Value (probability value) · Frequentist
Hint
More in Statistical Inference & Power
A Priori Power Analysis · Chi-square (χ²) Test · Clopper–Pearson Interval · Compromise Power Analysis · Confidence Intervals (CIs) · Effect Size (δ) · Hypothesis Testing · Kolmogorov–Smirnov (KS) Test · Minimum Detectable Lift (MDL) · P-Value (probability value) · Power (1 – β) · Power Analysis · Sample size · Significance Level (α)
See also
Source article Adapted (context, re-expressed) in our own words from: Post Hoc Power Analysis (insightful-data-lab.com).