Questions: Peer Review: Origins, Evolution, and Critiques

Question 1

The replication crisis in psychology and medicine revealed a specific statistical manipulation that peer review failed to catch. What is 'p-hacking'?

Model answer: P-hacking refers to manipulating the analysis of a dataset -- adding or removing variables, stopping data collection when results are significant, reporting only significant subgroup analyses -- until a p-value below 0.05 is achieved, then reporting the result as if the analysis had been pre-specified. Because peer reviewers rarely see raw data or analysis code, they cannot detect this. The result is published findings that appear statistically rigorous but are actually exploratory patterns inflated by selective reporting. Large replication studies, such as the 2015 Open Science Collaboration project that reproduced only 39% of published psychology findings, exposed how widespread this problem was.

P-hacking is not necessarily conscious fraud -- researchers may genuinely believe the manipulation is justified. The problem is structural: publication bias rewards positive results, peer review cannot detect exploratory data mining, and statistical thresholds (p < 0.05) create a bright line that incentivizes crossing it by any means.

Question 2

When did peer review become the standard mechanism for evaluating scientific manuscripts before publication?

• A. Since the founding of scientific journals in the 17th century -- the Royal Society used peer review from the start
• B. Peer review became widespread only in the mid-20th century; earlier journals relied on editorial judgment, and some major journals adopted it only in the 1970s
• C. Peer review was introduced by Vannevar Bush in 1945 as part of the NSF grant process
• D. Peer review has existed as long as science itself -- Greek philosophers reviewed each other's arguments

Answer: (B)

The history of peer review is often misunderstood. The Royal Society's Philosophical Transactions (1665) had editorial oversight but not systematic peer review. External referee review became standard gradually: Nature adopted it only in the 1970s. The assumption that peer review is a timeless feature of science is itself a historical myth -- understanding its emergence helps evaluate its shortcomings.

Question 3

What are the main arguments for and against anonymous (double-blind) peer review compared to open review?

Model answer: Anonymous review argues that anonymity protects junior researchers and non-prestigious institutions from reviewer bias -- known authors from elite universities get easier reviews. Critics argue that anonymity enables irresponsible reviewing (reviewers have no accountability for unfair or superficial reviews), that it doesn't prevent reviewers from guessing authors' identities, and that it hides conflicts of interest. Open review (reviewers' names known to authors, or reviews published alongside papers) increases accountability and allows readers to evaluate reviewers' credibility and potential biases. Some journals and preprint servers have adopted open review, finding it reduces gatekeeping while maintaining quality.

The debate about peer review structure is active and unresolved. Different fields have adopted different practices, and evidence on which works best is limited -- peer review is itself insufficiently peer reviewed.

Question 4

Peer review effectively prevents publication of fraudulent research.

• T. True
• F. False

Answer: False

Several high-profile fraud cases -- Hwang Woo-suk's fabricated stem cell papers (2004-2005), Diederik Stapel's decade of fabricated psychology data, Jan Hendrik Schon's falsified physics -- all passed peer review. Peer reviewers assess manuscripts, not original data; they cannot detect fabrication without access to raw files. Peer review's purpose is to evaluate logical coherence, methodological soundness, and adequacy of evidence -- it is not designed to detect deliberate fraud. Fraud detection typically comes from anomalies noticed by other researchers, whistleblowers, or post-publication data analysis.

Question 5

What was the 2015 Open Science Collaboration study, and what were its main findings about psychology research?

Model answer: The 2015 Open Science Collaboration, involving over 270 researchers, attempted to replicate 100 studies published in prominent psychology journals. Only 36-39% of the replications produced statistically significant results matching the original findings, and effect sizes were on average about half those in the original papers. The study did not prove fraud; it showed that standard scientific practices (flexible analysis, publication of positive results only, small samples) produced inflated and unreliable findings at scale. It triggered a reform movement: pre-registration of hypotheses, open data sharing, registered reports (where journals commit to publish regardless of results).

The OSC study was methodologically sophisticated and controversial -- some psychologists disputed its conclusions. The reform movement it triggered (open science practices, pre-registration) represents one of the most significant changes in scientific practice in recent decades.