Questions: Exploratory and Confirmatory Analysis Strategies and Their Distinct Roles

Question 1

A researcher collects data on 50 psychological variables, examines all pairwise correlations, finds that 'optimism correlates with creativity' at p = .04, and reports it as a significant discovery. What is the primary statistical problem with this conclusion?

• A. The p-value of .04 does not meet the conventional .05 threshold for significance
• B. With 50 variables there are 1,225 correlations; running all of them and selecting the significant one inflates the Type I error rate far above 5%, so the reported p-value does not mean what it appears to mean
• C. Correlations are not valid for psychological variables — only experimental designs produce valid p-values
• D. The finding is invalid because it was not preregistered before data collection began

Answer: (B)

At α = .05, about 5% of tests will return a false positive by chance. With 1,225 correlations, roughly 61 will appear 'significant' even when there is no real relationship. Selecting the most interesting-looking ones and reporting them with unconditional p < .05 claims makes those p-values meaningless as guarantees of 5% Type I error — the calculation was done as if one pre-specified test was run. Preregistration is the solution, not the original problem: the problem is presenting exploratory results as confirmatory.

Question 2

Which statement correctly describes the relationship between exploratory and confirmatory analysis?

• A. Exploratory analysis is scientifically inferior and its findings should never be published
• B. Confirmatory analysis guarantees true findings; exploratory analysis is unreliable
• C. Both have legitimate scientific roles: exploratory analysis generates hypotheses with honest uncertainty; confirmatory analysis tests pre-specified hypotheses with controlled Type I error rates
• D. The distinction is merely procedural — any p-value computed correctly has the same evidential meaning regardless of when the hypothesis was formulated

Answer: (C)

Exploratory analysis is scientifically essential — you cannot discover unexpected patterns without looking for them. The issue is not exploration itself but misrepresentation: presenting exploratory findings as confirmatory violates the statistical guarantee that makes p-values meaningful. Exploratory findings are valuable leads. Confirmatory findings are controlled tests. Treating them as equivalent is one mechanism behind the replication crisis. Option D captures the common error: p-values computed after data inspection are formally identical to pre-specified p-values but carry entirely different epistemic weight.

Question 3

A p-value computed after a researcher examines the data and selects the most interesting comparison carries the same Type I error guarantee as a p-value from a preregistered hypothesis test.

• T. True
• F. False

Answer: False

False. A p-value's guarantee that the false positive rate is controlled at α (e.g., 5%) holds only when the test was specified in advance — before seeing the data. When a researcher inspects data first and then chooses which test to report, the selection process itself capitalizes on chance: the analyst unconsciously or consciously picks tests that 'worked.' The resulting p-value is calculated using a formula that assumes a single pre-specified test, but the effective number of comparisons considered was much larger. The guarantee is void.

Question 4

The replication crisis in psychology is partly caused by researchers reporting exploratory findings as if they were confirmatory, leading readers to overestimate the strength of evidence.

• T. True
• F. False

Answer: True

True. When exploratory analyses — run after seeing the data, with multiple comparisons and flexible analysis choices — are reported with the language and statistics of confirmatory tests, readers interpret the p-values as evidence of controlled Type I error rates. But those rates are inflated. Studies built on this inflated evidence then fail to replicate when independent researchers run pre-specified confirmatory tests. Transparent reporting (labeling what was exploratory vs. confirmatory) and preregistration are the primary correctives.

Question 5

A researcher finds a surprising pattern with p = .03, but the hypothesis was not preregistered. Explain why this p-value cannot be interpreted the same way as a p-value from a preregistered test.

Model answer: The p-value's interpretation as a Type I error rate assumes the test was the one the researcher always planned to run, regardless of the data. Without preregistration, we cannot know whether this was the only test considered or the most interesting result selected from many. If the researcher examined multiple potential patterns and reported the most significant, the true false-positive rate for that finding could be much higher than 5% — even though p = .03. The p-value formula assumes a single pre-specified test; undisclosed exploration inflates error rates without changing the formula.

This is the HARKing problem (Hypothesizing After Results are Known). Even honest, good-faith researchers are subject to motivated reasoning: they explore many comparisons and gravitate toward reporting ones that 'worked.' Preregistration prevents this by creating a verifiable record that the hypothesis existed before data collection. Without that record, p = .03 is best interpreted as an interesting exploratory finding worth testing confirmatorily on new data — not as strong evidence with a 3% false-positive rate.