Questions: Analysis Planning and Preregistration of Hypotheses

Question 1

A researcher collects data, then tries 20 different combinations of exclusion criteria, covariates, and outcome measures until finding p < 0.05. This is problematic primarily because:

• A. Using multiple analysis methods is always a violation of APA ethical guidelines
• B. The p-value's meaning as a false positive rate assumes the analysis was prespecified; selecting the best result from many alternatives inflates the true false positive rate far above 5%
• C. The study should have collected more data before running any analyses
• D. Researchers must use identical methods to those used in prior studies on the same topic

Answer: (B)

The p < 0.05 threshold is meaningful only when the test was specified in advance. When a researcher searches through analytic space and reports the most favorable result, the probability of obtaining p < 0.05 by chance is far higher than 5% — the nominal error rate collapses. This is the structural problem (researcher degrees of freedom / p-hacking) that preregistration addresses, regardless of whether the researcher did it intentionally.

Question 2

A preregistered study finds a statistically significant relationship that was not included in the analysis plan. The correct course of action is:

• A. Report it as a confirmed finding since it was statistically significant
• B. Discard the finding entirely since it wasn't preregistered
• C. Report it as exploratory or hypothesis-generating, noting it requires a future confirmatory study
• D. Add it to the preregistration retroactively before publishing to maintain transparency

Answer: (C)

Preregistration does not prohibit exploration — it requires that exploration be labeled as such. An unexpected finding is genuinely interesting and worth reporting, but it counts as a hypothesis to be confirmed, not as established knowledge. Only a future study that preregisters this finding as its primary hypothesis can provide confirmatory evidence. Discarding it entirely would waste valuable information; calling it confirmed would repeat the original problem.

Question 3

Preregistration prevents researchers from conducting any analyses beyond what was specified in the preregistration document.

• T. True
• F. False

Answer: False

This is a common misunderstanding. Preregistration does NOT prohibit exploratory analyses — it requires that they be clearly distinguished from confirmatory ones. Curiosity and hypothesis generation are essential to science; the problem was never exploration itself, but presenting exploratory findings as confirmatory hypothesis tests with their associated error-rate interpretations. A preregistered study can and should report unexpected findings, clearly labeled as exploratory.

Question 4

A researcher who unconsciously p-hacks — making analytic choices without realizing they are being influenced by how those choices affect results — is still inflating the false positive rate.

• T. True
• F. False

Answer: True

Researcher degrees of freedom inflate false positive rates whether or not the researcher is aware of the bias. The problem is structural, not motivational. Analytic choices like 'this outlier looks like a data entry error' or 'this covariate improves model fit' often seem locally justified, but when they are consistently made in the direction of better results, the cumulative effect is the same as deliberate p-hacking. This is why structural solutions like preregistration are more effective than appeals to researcher conscientiousness.

Question 5

Why does preregistration make a p-value more interpretable, even when the data and statistical methods are identical to an unregistered study?

Model answer: The p-value's interpretation as a false positive rate (e.g., 5% chance of rejecting a true null at alpha = 0.05) holds only when the analysis was specified before seeing the data. Preregistration creates a timestamped record verifying this commitment. Without preregistration, a reported p < 0.05 may be the best result from dozens of analytic alternatives, in which case the actual false positive rate is far higher than 5%. Preregistration restores the statistical guarantee by ensuring the test was a genuine prospective prediction, not retrospective pattern-matching.

This is the mathematical heart of why preregistration matters. The frequentist guarantee — 'if the null is true, we'll reject it only 5% of the time' — applies to a single prespecified test. When researchers perform many tests and report the most favorable, the math breaks down. Preregistration doesn't change the data or the methods; it changes whether the reported p-value means what it claims to mean.