Questions: Multiple Comparisons Problem and Correction Methods

Question 1

A researcher runs an ANOVA with 4 groups, finds a significant omnibus F-test, then examines all 6 possible pairwise comparisons to locate the effect. What correction is most appropriate for these comparisons?

• A. No correction — the overall F-test already controlled for familywise error across the six comparisons
• B. Bonferroni correction dividing α by 6 — any post-hoc comparison requires this specific correction
• C. A post-hoc correction designed for exhaustive pairwise comparisons, such as Tukey's HSD
• D. FDR control — all post-hoc comparisons are exploratory, and FDR is the standard for exploration

Answer: (C)

The omnibus F-test tells you that *some* difference exists among the groups, but not which pairs differ — that requires further testing. These follow-up pairwise tests are post-hoc comparisons conducted after observing data, which capitalizes on chance and inflates false positive risk. Tukey's HSD is specifically designed for this situation: all pairwise comparisons after a significant ANOVA. Simple Bonferroni (option B) is valid but more conservative than necessary here. The F-test (option A) does not protect against the pairwise comparisons — it answers a different question.

Question 2

A researcher conducts 100 statistical tests at α = .05. Assuming all null hypotheses are true (no real effects exist), approximately how many tests are expected to yield a 'significant' result?

• A. 0 — with proper alpha control, no false positives should occur under the null
• B. 5 — the expected number of Type I errors is α × number of tests = .05 × 100
• C. 1 — one test per family is the conventional allowance
• D. 0.05 — that is the probability of a false positive, not the expected count

Answer: (B)

Each test independently has a 5% false positive rate. With 100 tests and no true effects, the expected number of false positives is 0.05 × 100 = 5. This is the core of the multiple comparisons problem: the more tests you run, the more false positives you expect by chance alone, even when nothing is real. The familywise error rate — the probability of at least one false positive — approaches 1 − (0.95)^100 ≈ 99.4% with no correction.

Question 3

The Bonferroni correction controls the familywise error rate by making each individual test harder to pass, but this comes at the cost of reduced statistical power to detect true effects.

• T. True
• F. False

Answer: True

Bonferroni divides α by the number of tests k: each test uses α/k instead of α. For k = 20 tests, the per-test threshold drops from .05 to .0025. This dramatically reduces the probability of any false positive, but equally reduces power: true effects that would have been significant at α = .05 may no longer clear the stricter threshold. This power reduction is why Bonferroni is considered conservative, especially when tests are positively correlated — in that case, it over-corrects.

Question 4

A researcher who specifies exactly two theoretically motivated comparisons before data collection needs to apply the same stringent correction as a researcher who conducts 100 post-hoc comparisons on the same dataset.

• T. True
• F. False

Answer: False

The appropriate correction depends critically on whether comparisons were planned a priori or discovered post-hoc. Two pre-specified planned contrasts constitute a family of 2 tests with a coherent theoretical basis — the inflation of Type I error is modest and may not require correction beyond the two tests themselves. One hundred post-hoc comparisons represent a fundamentally different situation: the researcher is hunting for significance, and the probability of a false positive among the findings is high. The number of tests is only one factor; whether they were planned or exploratory determines the correct accounting.

Question 5

Why does the multiple comparisons problem arise when conducting many statistical tests, and why can't it be fully fixed by applying corrections after data collection has occurred?

Model answer: The multiple comparisons problem arises because each test has a nonzero false positive rate (α), so running many tests guarantees some false positives will occur by chance. With k independent tests at α = .05, the familywise error rate is 1 − (1 − .05)^k, which approaches 1 as k grows. Post-hoc corrections like Bonferroni or FDR can mathematically adjust thresholds to control error rates, but they cannot undo the deeper problem: when a researcher selects which comparisons to report after seeing the data (or runs many comparisons to find significance), the reported p-values no longer have their nominal meaning. The selection process itself introduces bias that no correction can fully repair — only pre-registration and honest reporting of all tests conducted can preserve the validity of the inference.

This is why the distinction between planned and post-hoc comparisons matters so much. A Bonferroni correction applied after exploring all possible subgroup analyses does reduce the nominal false positive rate, but the researcher's decision to report only significant findings, or to stop analyzing once significance was found, cannot be corrected statistically. The solution is procedural (preregistration, full reporting) not purely mathematical.