Questions: One-Way ANOVA: Theory and F-Test

Question 1

An ANOVA comparing four teaching methods yields F(3, 76) = 4.2, p = .008. What can you legitimately conclude?

• A. All four teaching methods produce significantly different outcomes from each other
• B. At least one pair of group means differs more than expected by chance
• C. The methods with the highest and lowest means are significantly different
• D. The average score across all methods is significantly above zero

Answer: (B)

A significant F-test rejects H₀: μ₁ = μ₂ = μ₃ = μ₄, meaning the group means are not all equal — at least one pair differs. It does NOT identify which pairs differ, does NOT conclude all pairs differ, and does NOT rank or compare specific groups. Post-hoc tests (Tukey, Bonferroni, etc.) are needed to determine which specific groups differ. Concluding that the two extreme groups differ (option C) is a common but invalid move that inflates Type I error.

Question 2

Two researchers analyze the same dataset: three groups of n=15 each. Researcher A selects the highest and lowest-scoring groups and runs a t-test on just those two. Researcher B runs a one-way ANOVA across all three groups. Which approach is statistically preferable?

• A. Researcher A — testing only the most extreme groups is the most powerful test of group differences
• B. Researcher B — ANOVA tests all groups simultaneously without the inflated Type I error of selecting which groups to compare after seeing the data
• C. They are equivalent — a t-test and ANOVA produce identical p-values when comparing exactly two groups
• D. Researcher A — ANOVA is only valid when all group means are expected to be different

Answer: (B)

Researcher A is performing a post-hoc comparison without correction — selecting the most extreme groups after observing data inflates the Type I error rate far above α=.05. One-way ANOVA tests H₀: all means are equal in a single omnibus test, maintaining the overall α level. Option C is true when you specify two groups in advance without cherry-picking, but Researcher A's selection procedure invalidates this equivalence. ANOVA is designed precisely to avoid the multiple-comparison inflation that arises from running separate t-tests.

Question 3

The F-statistic in one-way ANOVA increases when between-group variability is large relative to within-group variability.

• T. True
• F. False

Answer: True

F = MS_Between / MS_Within. MS_Between measures how much the group means vary around the grand mean; MS_Within measures average variability within groups (the pooled error). When groups have very different means, MS_Between is large; when observations within each group cluster tightly, MS_Within is small. A large F thus indicates that the signal (group mean differences) is large relative to the noise (within-group spread), which is exactly the evidence we need to reject the null hypothesis that all group means are equal.

Question 4

A statistically significant one-way ANOVA F-test allows you to conclude which specific pairs of group means differ significantly.

• T. True
• F. False

Answer: False

This is the most common misinterpretation of ANOVA results. The omnibus F-test answers only one question: are all group means equal? If the answer is no (p < α), you know at least one pair differs — but you do not know which one(s). Identifying which pairs differ requires post-hoc tests (e.g., Tukey's HSD, Bonferroni correction) that control the family-wise error rate across all pairwise comparisons. Skipping post-hoc tests and simply comparing the two groups with the largest mean difference is statistically invalid.

Question 5

Explain what the F-statistic in one-way ANOVA is actually measuring. What goes in the numerator, what goes in the denominator, and why does a large F-ratio provide evidence against the null hypothesis?

Model answer: The numerator is MS_Between (mean square between groups): the variance of group means around the grand mean, scaled by group size. It measures how spread out the group means are. The denominator is MS_Within (mean square within groups): the pooled average variance within each group, measuring random sampling error. Under H₀ (all means equal), both MS_Between and MS_Within estimate the same underlying population variance, so F ≈ 1. When some means differ, MS_Between becomes inflated (true mean differences add to the variance of group means), while MS_Within remains an unaffected estimate of error variance. A large F therefore indicates that group means vary more than pure chance predicts.

The key intuition is that F is a signal-to-noise ratio. MS_Between captures both true differences between groups and sampling error; MS_Within captures only sampling error. If there are no true group differences, the ratio is approximately 1. Real group differences push the numerator up without affecting the denominator, increasing F. The F-distribution under H₀ tells you how large F would need to be by chance, and a sufficiently large observed F leads us to reject H₀.