Questions: External Validity and Generalizability to Populations

Question 1

A randomized controlled trial enrolls 3,000 undergraduate psychology students and randomly assigns them to conditions. What is the primary threat to external validity?

• A. The sample is too large, making the results overly sensitive to small, meaningless differences
• B. The sample may not represent the broader population — undergraduates are younger, more educated, and more Western than most target groups
• C. Random assignment undermines external validity by making conditions too artificial
• D. There is no threat — large sample sizes guarantee that findings will generalize

Answer: (B)

External validity is about representativeness, not sample size. Three thousand undergraduates still generalizes only to undergraduate-like populations — they are disproportionately young, educated, Western, and female compared to most general populations (the WEIRD problem). A well-stratified sample of 500 could generalize better to a broader population than 3,000 convenience-sampled undergraduates. Option D is the core misconception this topic addresses: large N improves statistical power and internal precision, but cannot substitute for representative sampling.

Question 2

Researchers often face a tradeoff: increasing internal validity (experimental control, random assignment) can reduce external validity.

• T. True
• F. False

Answer: True

High internal validity requires controlled settings that remove extraneous variables — but these controlled settings often diverge sharply from real-world conditions, limiting generalizability. A fear-conditioning study in a quiet laboratory cubicle with tones and mild shocks may produce clean causal estimates (high internal validity) that tell us little about fear learning in naturalistic, socially meaningful contexts (low external validity). The tradeoff is not absolute — some effects replicate across settings — but the assumption that experimental control implies generalizability is false.

Question 3

External validity is a binary property: a study either has it or doesn't, depending on whether the sample was randomly drawn from the target population.

• T. True
• F. False

Answer: False

External validity is better understood as a map — a conditional description of where findings travel well and where they don't. A finding might generalize across cultures but not across age groups, or across laboratory and field settings but not across historical eras. Meta-analysis reveals this through moderator analysis: when effect sizes are heterogeneous across studies, identifying which features (population characteristics, methodological choices, setting type) explain the variation produces a conditional generalization rather than a binary verdict. 'This effect holds when X is present and weakens when Y is present' is more scientifically honest than claiming universal applicability.

Question 4

What is the difference between population validity and ecological validity? Give an example of a study that could have high population validity but low ecological validity.

Model answer: Population validity concerns whether findings generalize from the study's sample to the intended target population. Ecological validity concerns whether the study's procedures and settings resemble the conditions under which the phenomenon naturally occurs. A study with high population validity but low ecological validity: a nationally representative probability sample of adults (high population validity) measuring aggression by having participants blast loud noise at a confederate in a laboratory cubicle (low ecological validity, since real-world aggression occurs in specific relational and emotional contexts not captured by this proxy measure).

These two components of external validity vary independently and are frequently conflated. You can have a representative sample studying an artificial phenomenon, or a naturalistic study with a convenience sample. The best external validity requires both components to be strong, but identifying which is weak helps diagnose how and where findings might fail to replicate.

Question 5

Cross-cultural replications found that many psychological findings from WEIRD samples did not hold in other populations. This is primarily a failure of which type of validity?

• A. Internal validity — confounds were not controlled in the original studies
• B. External validity — findings from non-representative samples were incorrectly assumed to generalize universally
• C. Construct validity — the measures used in the original studies were invalid
• D. Statistical conclusion validity — the original studies lacked sufficient statistical power

Answer: (B)

The WEIRD problem is a failure of external validity — specifically population validity. The original studies likely measured what they intended to measure (internal validity was often adequate), but decades of findings on conformity, moral reasoning, perception, and cognitive biases were derived from non-representative convenience samples and assumed to describe universal human psychology. Cross-cultural failures reveal that the findings did not travel beyond the populations sampled. This is the canonical illustration of why external validity requires explicit attention: a real effect in one population may not be a universal human effect.