Statistics Reference
Type I vs Type II Errors
Every hypothesis test can fail in exactly two ways: sound the alarm when nothing is happening, or stay silent when something is. Those are the Type I and Type II errors, and everything about test design — significance levels, sample sizes, power calculations — is a negotiation between them. This guide defines both, quantifies them with a worked example, and shows which levers reduce each one.
The Four Possible Outcomes
A test ends in a decision (reject the null hypothesis or not) and reality is one of two states (the null is true or false). Crossing them gives the fundamental 2×2 table:
| H₀ is actually true | H₀ is actually false | |
|---|---|---|
| Test rejects H₀ | Type I error — false positive, probability α | Correct — true positive, probability 1 − β (power) |
| Test fails to reject | Correct — true negative, probability 1 − α | Type II error — false negative, probability β |
α is chosen before the test — it is the false-alarm rate you are willing to tolerate, enforced through the critical value. β is not chosen directly: it falls out of the design (sample size, noise) and the size of the true effect, which is why it must be computed rather than declared.
The Same Two Errors, Everywhere
- Medical screening. Type I: a healthy patient flagged as ill — anxiety and unnecessary follow-ups. Type II: a sick patient cleared — a missed diagnosis. Screening programs deliberately accept more Type I errors to shrink Type II.
- Spam filtering. Type I: a real message sent to spam (the null “this is legitimate mail” wrongly rejected). Type II: spam reaching the inbox. Losing one job offer outweighs seeing ten spam messages, so filters run conservative.
- Quality control. Type I: halting a production line that is actually fine — lost output. Type II: shipping a bad batch — recalls and reputation. The relative costs set the control limits.
- Criminal trials. Type I: convicting the innocent. Type II: acquitting the guilty. “Beyond reasonable doubt” is a demand for a very small α, knowingly paid for with a larger β.
Worked Example: Computing β and Power
A school claims its program raises IQ scores above the national mean of 100 (σ = 15, known). It tests H₀: μ = 100 against a two-sided alternative at α = 0.05 with n = 25 students, so the standard error is 15/√25 = 3.
- Rejection region. Reject when the sample mean falls outside 100 ± 1.96 × 3, i.e. below 94.12 or above 105.88. By construction, a true-null sample lands there 5% of the time — that is the Type I error rate.
- Suppose the truth is μ = 106. A Type II error happens when the sample mean still lands inside (94.12, 105.88). Standardizing against the true mean: z = (105.88 − 106) / 3 = −0.04, and the lower bound is nearly 4 standard errors below, contributing essentially 0.
- Type II error rate. β = Φ(−0.04) ≈ 0.4840 — a 48% chance of missing a genuine 6-point effect.
- Power. 1 − β ≈ 0.516. Barely better than a coin flip, despite a real effect being present.
Tightening α to 0.01 moves the cutoff to 100 + 2.576 × 3 = 107.73, and β rises to Φ((107.73 − 106)/3) = Φ(0.58) ≈ 0.72 — power collapses to about 28%. Same data, same true effect: fewer false alarms bought strictly more missed effects. The only way out is better design — most directly, more students per group.
How to Reduce Each Error
Reducing Type I (α)
- Choose a stricter significance level before testing.
- Correct for multiple comparisons when running many tests.
- Pre-register the hypothesis and tail direction — switching tails after seeing data inflates α silently.
Reducing Type II (β)
- Increase the sample size — the one lever that lowers β without raising α.
- Reduce measurement noise (better instruments, paired designs).
- Target effects large enough to matter, and use one-tailed tests only when the direction is genuinely known in advance.
The standard planning workflow fixes α (usually 0.05), demands a power target (usually 80% or 90%) for the smallest effect worth detecting, and solves for n — which is exactly what the sample size calculator does. Reporting an effect size alongside the p-value then tells readers whether a significant result is also a meaningful one.
Where the Errors Live in a Real Test
Run the numbers yourself: the z-test calculator reports the critical value that defines the Type I error rate, and the t-test calculator does the same when the standard deviation is estimated. The p-value calculator converts any test statistic into the tail probability that gets compared against α. None of these report β — power is a property of the design, not of one sample, which is why it belongs in planning rather than in the results section.
Try the Z-Test Calculator
Run one-sample, two-sample, and proportion z-tests with the critical values and p-values that define the Type I error rate.
Frequently Asked Questions
What is a simple way to remember Type I vs Type II errors?
A Type I error is a false alarm: the null hypothesis was true, but you rejected it (false positive). A Type II error is a missed alarm: the null hypothesis was false, but you failed to reject it (false negative). One mnemonic: the boy who cried wolf caused a Type I error first (villagers believed a wolf that wasn't there), then a Type II error (they ignored a wolf that was).
Which is worse, a Type I or a Type II error?
Neither is universally worse - it depends on the costs. Convicting an innocent person (Type I) is usually judged worse than acquitting a guilty one (Type II), so courts demand proof beyond reasonable doubt (a tiny alpha). But in cancer screening, missing a real tumor (Type II) can be deadlier than a false alarm that triggers follow-up testing (Type I). Set alpha and power to match the real costs of each mistake.
How are alpha and the Type I error rate related?
Alpha IS the Type I error rate you choose in advance: testing at alpha = 0.05 means accepting a 5% chance of rejecting a true null hypothesis. It is enforced by the critical value - the test only rejects when data lands in a region that true-null data would reach 5% of the time. That is also why p < alpha is the rejection rule: the p-value measures how extreme the data is on the same scale.
What is statistical power and how does it relate to Type II errors?
Power is the probability of correctly rejecting a false null hypothesis, and it equals 1 - beta, where beta is the Type II error rate. A study with 80% power misses a real effect of the assumed size 20% of the time. Power grows with larger samples, bigger true effects, less noisy data, and a larger alpha - the four levers of study design.
Does lowering alpha from 0.05 to 0.01 make my test better?
It makes false alarms rarer and missed effects more common - at a fixed sample size, cutting alpha raises beta. Whether that trade is 'better' depends on which error is costlier. If you need both a strict alpha and high power, the price is sample size: the worked example in this article shows power falling from 52% to 28% when alpha drops to 0.01, unless n increases.
If my result is significant at p < 0.05, is there still a chance it is wrong?
Yes, two ways. First, even with everything done correctly, 5% of true null hypotheses tested at alpha = 0.05 will produce false alarms. Second, the fraction of significant results that are wrong (the false discovery rate) can be far above 5% when true effects are rare or power is low - a field testing mostly null hypotheses at low power produces many more false than true alarms.