Statistics Reference
P-Value Explained: What p < 0.05 Actually Means
The p-value is the most used and most misread number in statistics. Its definition takes one sentence; the misreadings fill retraction notices. This guide gives the precise meaning, walks a complete example from data to decision, and corrects the five misinterpretations that cause most of the damage.
The Definition, Precisely
The p-value is the probability of observing data at least as extreme as yours, assuming the null hypothesis is true.
Every word carries weight. “Assuming the null hypothesis is true” means the whole calculation lives in a hypothetical world where nothing is going on — the coin is fair, the drug does nothing, the groups are equal. “At least as extreme” means the p-value is a tail area, not the probability of your exact result. And because the assumption is about the hypothesis, the p-value can never tell you the probability of the hypothesis — that inversion is the root of nearly every misreading.
Small p-values are evidence against the null hypothesis by a simple logic: either the null is true and something rare just happened, or the null is false. The smaller the p, the less comfortable the first option becomes.
Worked Example: Is This Coin Fair?
A coin is flipped 100 times and lands heads 60 times. Is that evidence of bias?
- Null hypothesis: the coin is fair, p = 0.5. Under it, the proportion of heads has mean 0.5 and standard error √(0.5 × 0.5 / 100) = 0.05.
- Test statistic: z = (0.60 − 0.50) / 0.05 = 2.0 — the observed 60% sits two standard errors above fairness.
- Tail area: results at least this extreme in either direction (≤ 40 or ≥ 60 heads) have probability 2 × (1 − Φ(2.0)) ≈ 0.0455.
- Decision at α = 0.05: 0.0455 ≤ 0.05, so the result is statistically significant — a fair coin would produce a split this lopsided only about once in 22 experiments.
Note what the p-value did not say: it did not say there is a 95.45% chance the coin is biased, and it did not say the bias is large. It said only that 60-of-100 is fairly unusual for a fair coin. The z-test calculator reproduces every number above in one-proportion mode, and the p-value calculator converts any test statistic into its tail area.
How the Decision Rule Works
Before testing, you choose a significance level α — the false-alarm rate you can live with. The rule is then mechanical: reject the null hypothesis when p ≤ α. The two numbers play different roles: α is a policy set in advance; p is a measurement made from data. Comparing them decides whether the data landed in the rejection region, which is why p ≤ α and “the test statistic exceeds the critical value” are always the same verdict.
The same evidence can also be read against a stricter policy: the coin's p = 0.0455 clears α = 0.05 but fails α = 0.01. Nothing about the coin changed — only the tolerance for false alarms. The trade-offs behind that choice (and what happens to missed effects when α shrinks) are the subject of the Type I vs Type II errors guide.
The Five Misreadings — and the Correct Version of Each
| Misreading | What is actually true |
|---|---|
| “p = 0.05 means a 5% chance the null hypothesis is true.” | The p-value conditions on the null being true; it cannot also measure its probability. Hypothesis probabilities require Bayesian methods and a prior. |
| “p = 0.05 means a 95% chance the effect is real.” | The share of significant findings that are real depends on how often tested hypotheses are true and on power — it can be far below 95% in low-power, long-shot research. |
| “A smaller p means a bigger effect.” | p mixes effect size with sample size. A trivial effect in a huge sample beats a large effect in a small one. Judge magnitude with effect sizes and confidence intervals. |
| “p > 0.05 proves there is no effect.” | Absence of evidence is not evidence of absence — the study may simply lack power. A wide confidence interval spanning zero says “undetermined,” not “zero.” |
| “p = 0.049 and p = 0.051 mean different realities.” | Evidence is continuous; the cliff at 0.05 is administrative. Two studies straddling it agree almost perfectly about the world. |
Using P-Values Well
- Decide the test before seeing the data — the hypothesis, the tail, and α. Switching any of them afterward silently inflates false alarms.
- Report exact values with context: the estimate, its confidence interval, the effect size, and n — not just a star next to p < 0.05.
- Mind multiplicity. Twenty independent true-null tests have about a 64% chance of producing at least one p < 0.05 — the at-least-one calculator makes that arithmetic vivid. Correct for multiple comparisons or treat lone significant results from big screens as leads.
- Replicate. One p < 0.05 is a single noisy vote, not a verdict; effects that matter show up again.
Try the P-Value Calculator
Convert z, t, chi-square, or F statistics into exact one- and two-tailed p-values, with the distribution logic shown.
Frequently Asked Questions
What does a p-value of 0.05 actually mean?
It means that if the null hypothesis were true, data as extreme as yours (or more extreme) would occur 5% of the time by chance alone. It is a statement about the data given the hypothesis - not about the hypothesis given the data. It does not mean there is a 5% chance the null hypothesis is true, and it does not mean the finding has a 95% chance of being real.
Is p < 0.05 the same as 'the result is true'?
No. Crossing the 0.05 threshold means the data would be unusual under the null hypothesis - nothing more. The result can still be a false alarm (expected for about 1 in 20 true-null tests), the effect can be real but trivially small, and the finding still needs replication. Significance is a screening verdict, not a truth certificate.
Why is 0.05 the standard cutoff?
Convention, traceable to Ronald Fisher's 1925 textbook, which suggested one-in-twenty as a convenient benchmark. Nothing in probability theory privileges 0.05: particle physics demands about 0.0000003 (five sigma) for discoveries, while exploratory screens may accept 0.10. Treat 0.05 as a community default whose appropriateness depends on the cost of false alarms in your context.
What is the difference between one-tailed and two-tailed p-values?
A two-tailed p-value counts extreme results in both directions; a one-tailed p-value counts only the pre-specified direction and is half the two-tailed value for symmetric distributions. In the coin example, z = 2.0 gives p = 0.0455 two-tailed but 0.0228 one-tailed. Choosing one-tailed after seeing the data points in your favor is a form of p-hacking.
Does a smaller p-value mean a bigger or more important effect?
No. The p-value mixes effect size with sample size: a tiny, unimportant effect yields an arbitrarily small p-value if the sample is large enough, while a large effect can miss significance in a small study. To judge magnitude, look at the effect size (like Cohen's d) and the confidence interval, not the number of zeros in p.
What should I report alongside a p-value?
The estimate itself, a confidence interval, the effect size, and the sample size - enough for a reader to judge magnitude and precision, not just the significance verdict. Report the exact p-value (p = 0.03) rather than an inequality (p < 0.05), and state whether the test was one- or two-tailed and whether it was planned before seeing the data.