Questions: Research Integrity and Open Science: Transparency and Reproducibility

Question 1

A researcher collects data and finds p = .06. They remove two participants flagged as potential outliers, find p = .049, and publish with no mention of the original analysis. This practice is best described as:

• A. Appropriate data cleaning — outliers distort results and must be removed
• B. P-hacking — using post-hoc, result-contingent decisions to push p below the significance threshold, exploiting researcher degrees of freedom
• C. Acceptable because p = .049 technically falls below .05
• D. Standard practice in social science because all outliers must be removed

Answer: (B)

This is a textbook example of exploiting researcher degrees of freedom. Decisions about outlier exclusion made after seeing results — and contingent on whether they produce significance — are not legitimate data cleaning. They inflate the false-positive rate because the researcher is effectively running multiple analyses and reporting only the one that worked. Preregistration would have required the exclusion criteria to be specified before data collection, preventing this.

Question 2

A researcher preregisters the hypothesis that 'nature exposure reduces anxiety,' with a specific anxiety scale as the outcome. After running the study, they notice a second anxiety measure also shows a trend and report it prominently. How should this second finding be classified?

• A. As strong confirmation of the preregistered hypothesis, since both measures reflect the same construct
• B. As exploratory — a pattern not pre-specified that requires independent replication before being treated as confirmed
• C. As more important than the preregistered result, since it was an unexpected discovery
• D. As disconfirmatory, since it was not the primary outcome

Answer: (B)

Preregistration creates a clean distinction between confirmatory analysis (testing a pre-specified hypothesis with full inferential weight) and exploratory analysis (discovering post-hoc patterns that require replication). Both types of analysis are legitimate and valuable, but they differ in inferential status. Calling an exploratory finding 'confirmation' conflates hypothesis-testing with hypothesis-generation and is one of the mechanisms driving false positives in the literature.

Question 3

The replication crisis in social science was primarily caused by deliberate fraud by individual researchers who fabricated or falsified their data.

• T. True
• F. False

Answer: False

While fraud exists, it is not the primary cause of the replication crisis. The crisis arises from structural features: researcher degrees of freedom (many legitimate-seeming analytical choices that can be steered toward significance, even unconsciously) combined with publication bias (journals preferentially publish significant results, making null results invisible). These mechanisms produce inflated false-positive rates even when every researcher is acting in complete good faith. The crisis would persist even with zero fraud.

Question 4

Preregistration prevents researchers from conducting any analyses beyond those specified in advance, eliminating most researcher flexibility in data analysis.

• T. True
• F. False

Answer: False

Preregistration does not prohibit additional analysis — it creates transparency by distinguishing pre-specified (confirmatory) analyses from post-hoc (exploratory) ones. Researchers remain free to explore their data fully; they must simply label exploratory findings as such. The value of preregistration is accountability and clarity about inferential status, not restriction of scientific creativity. A preregistered study can have a rich exploratory section, as long as it is clearly labeled.

Question 5

Explain why a publication bias problem could mislead the scientific literature even if every individual researcher is acting in complete good faith and reporting their results accurately.

Model answer: Even honest researchers preferentially write up and submit significant results. Journals preferentially publish significant results. Non-significant results — even from well-designed studies — go unreported or are rejected. Over time, the published literature becomes a biased sample that overrepresents positive findings, many of which are false positives or inflated effect sizes, because the filtering mechanism systematically excludes the null results that would correct the picture.

This is why the replication crisis is a structural problem, not primarily an ethics problem. Individual virtue cannot fix a selection process that filters the literature. Remedies must be structural too: preregistered reports (where journals commit to publish before seeing results), registered replication reports, data sharing requirements, and meta-analytic correction techniques like trim-and-fill. The problem is architectural — and so must be the solution.