Tukey HSD Calculator

Tukey's honestly significant difference (HSD) test compares every pair of group means after a one-way ANOVA and keeps the chance of at least one false positive across all the pairs at α. Testing each pair with its own t-test would let that chance grow with the number of pairs. Tukey's test instead refers each difference to the studentized range distribution, the distribution of the gap between the largest and the smallest of k means measured in standard errors, so the whole family of comparisons holds its level exactly when the groups have equal sizes. With unequal sizes the same formula is called the Tukey-Kramer method. It errs on the safe side: Hayter (1984) proved that its error rate never exceeds α.

Use it when the group variances are similar and you want to know which means differ, usually after a significant F in the one-way ANOVA calculator. The reasoning behind adjusted p-values is in multiple comparisons explained, and ANOVA explained covers the test that comes first.

1. Raw data: enter each group's observations separated by commas, spaces or new lines, at least 2 per group. Add another group gives you up to 10 groups. An empty group is left out and the others keep their own numbers in the results.
2. From an ANOVA table: choose Means + MSE and df (from ANOVA) and enter the group means, the sample sizes, the within-group mean square (MSE, called Mean Square Within or Error in the output) and its degrees of freedom N − k. This also works for the marginal means of a two-way ANOVA.
3. Set the family-wise significance level α, usually 0.05. It fixes the critical q and the confidence level 1 − α of the intervals.
4. Press Calculate Tukey HSD. Read the critical q, the HSD (only when all groups have the same size) and the table with the difference, standard error, q, adjusted p-value and interval of every pair.
5. Load example enters the PlantGrowth data used below in the current input mode, and Copy link to this calculation shares your exact analysis.

The critical value q(α; k, N − k) is the upper α point of the studentized range distribution. The calculator computes it, and every p-value, by numerical integration of that distribution (relative error below 1e-12, checked against 30-digit reference values and SciPy) instead of reading a printed q table, so any number of groups up to 50 and any error df from 2 to 10 million work.

• Critical q and HSD. With equal group sizes any two means further apart than the HSD are significantly different at the family-wise level; with unequal sizes each pair has its own threshold and the HSD card shows n/a.
• Diff. The mean of the first group minus the mean of the second (the SciPy and SPSS convention). R's TukeyHSD subtracts the earlier group from the later one, so its differences have the opposite sign.
• SE and q. SE is the standard error of the difference and q = √2 · |Diff| / SE is the studentized range statistic for the pair.
• Adj. p. The probability of a gap at least this large between two of the k means when all population means are equal. It is already adjusted for the number of pairs, so compare it with α directly.
• Confidence interval. A simultaneous interval: all the pair intervals together cover the true differences with probability 1 − α, so each is wider than an ordinary t interval. It excludes 0 exactly when the pair is significant.
• Not significant is not equal. A pair whose interval contains 0 is compatible with no difference; with small groups it is compatible with a large one too.

The groups 4, 5, 6 (group 1), 6, 7, 8 (group 2) and 8, 9, 10 (group 3) are what the calculator holds when the page opens. The means are 5, 7 and 9, each group has n = 3, and the within-group sum of squares is 2 + 2 + 2 = 6 on 9 − 3 = 6 degrees of freedom, so MSE = 1.

The PlantGrowth data set that ships with R (Dobson, 1983) holds the dry weight of 30 plants, ten controls and ten for each of two treatments. The one-way ANOVA calculator gives F(2, 27) = 4.846 and p = 0.0159 with MSE = 0.3886 on 27 degrees of freedom, so the means differ somewhere. Press Load example to enter the weights, or switch to the ANOVA-table input, where the example holds the means 5.032, 4.661 and 5.526, n = 10, MSE = 0.3886 and df = 27.

The critical value is q(0.05; 3, 27) = 3.5064, so HSD = 3.5064 × √(0.3886 / 10) = 0.6912. The three comparisons:

Only treatment 1 against treatment 2 (groups 2 and 3) is significant: its gap of 0.865 exceeds the HSD of 0.6912, its adjusted p-value is 0.012 and its interval excludes 0. The control differs from neither treatment at α = 0.05 with ten plants per group. R's TukeyHSD(aov(weight ~ group, data = PlantGrowth)) reports the same comparisons as trt1-ctrl = −0.371, trt2-ctrl = 0.494 and trt2-trt1 = 0.865 with adjusted p = 0.3908711, 0.1979960 and 0.0120064: R subtracts the earlier group from the later one, so its signs are the opposite of the Diff column and its lower and upper limits are the negated upper and lower limits here. Typed from the rounded ANOVA table, the example differs from the raw-data result only in the sixth decimal of the p-values.

With unequal sample sizes the standard error of each pair uses its own two sizes, and there is no single HSD. Take four groups from an ANOVA table with means 10, 12, 15 and 11, sizes 6, 8, 5 and 7, MSE = 4 and 22 error degrees of freedom. The critical value is q(0.05; 4, 22) = 3.927. Three of the six comparisons:

Groups 1 and 2 are two units apart and not significantly different, while the larger gaps between groups 1 and 3 (5 units) and between groups 3 and 4 (4 units) are. Hayter (1984) proved that the Tukey-Kramer intervals are conservative for every choice of sizes: the simultaneous coverage is at least 1 − α, and its minimum, exactly 1 − α, occurs when the sizes are equal.

For a chosen set of p-values from any source use the Bonferroni correction calculator. The Kruskal-Wallis test is the rank-based alternative to the ANOVA that comes before Tukey's test: Kruskal-Wallis test.

• Independent observations within and between the groups. This comes from the design.
• Roughly normal values within each group. Moderate departures do little harm; check with the Shapiro-Wilk test or a histogram.
• Equal variances. The MSE pools all the groups. With equal sizes, a group with a much larger spread makes the comparisons that involve it too liberal and the comparisons among the other groups too conservative; with unequal sizes the direction depends on which groups are larger. Test the spreads with the Levene test calculator and switch to Games-Howell if it fails.
• One factor. For a factor in a two-way ANOVA enter its marginal means, the number of observations behind each mean, and the error mean square and error df of the two-way table.