Questions: Effect Sizes, Practical Significance, and Results Reporting

Question 1

A study with 50,000 participants finds that people born in January score 0.3 IQ points higher than those born in July (p < 0.0001, d = 0.02). What is the most accurate interpretation?

• A. The effect is both real and important because the p-value is extremely small
• B. The effect is likely real but has negligible practical significance given the tiny effect size
• C. The p-value is too small to trust; very small p-values indicate measurement error
• D. The effect size is too small to be statistically significant, so we should withhold judgment

Answer: (B)

This scenario illustrates the core lesson: statistical significance and practical significance are independent. With 50,000 participants, even a trivially small difference produces an extremely small p-value because significance is a function of sample size, effect size, and variability. A d of 0.02 (two-hundredths of a standard deviation) has no meaningful practical consequence. Option A commits the most common error — equating statistical significance with importance.

Question 2

What does a confidence interval around an effect size communicate that the p-value alone does not?

• A. Whether the effect size is large enough to be practically significant
• B. The probability that the null hypothesis is true given the data
• C. Both the magnitude of the estimated effect and the precision (uncertainty) of that estimate
• D. The minimum sample size needed to replicate the finding

Answer: (C)

A p-value tells you whether the result is unlikely under the null hypothesis. An effect size tells you how large the difference is. A confidence interval around the effect size adds information about precision: a narrow CI around d = 0.6 indicates a well-estimated medium-large effect; a wide CI [0.05, 1.15] around the same point estimate signals that the true effect size is highly uncertain. Neither the p-value nor the effect size alone conveys this uncertainty; the confidence interval does.

Question 3

A larger p-value indicates a larger effect size.

• T. True
• F. False

Answer: False

P-values and effect sizes are logically independent. A p-value reflects not just the effect magnitude but also sample size and variability. A tiny effect can produce p < 0.0001 with a huge sample, and a large effect can produce p = 0.20 with a small sample. The p-value answers 'Is this real?' Effect size answers 'How big is it?' Confusing the two is one of the most common errors in interpreting psychological research.

Question 4

A study can find a statistically significant effect that is practically meaningless.

• T. True
• F. False

Answer: True

Yes — and this is increasingly common in large-scale studies. With very large samples, even trivially small differences produce statistically significant results because significance is sensitive to sample size. A drug that reduces blood pressure by 0.1 mmHg might reach p < 0.001 in a trial with 100,000 patients, but this effect is clinically irrelevant. This is why reporting both p-values and effect sizes (with confidence intervals) is required by modern publication standards.

Question 5

Explain why statistical significance and effect size are logically independent, using an example to illustrate.

Model answer: Statistical significance depends on three things: effect magnitude, sample size, and variability. Effect size measures only the magnitude, standardized by variability (e.g., Cohen's d). These are independent because sample size can be varied independently of the true effect. A small true effect (d = 0.05) will be statistically significant with n = 100,000 because the standard error is tiny. A large true effect (d = 0.80) will be non-significant with n = 10 because low power means high sampling variability.

The formula for a t-test makes this concrete: t = d × √(n/2). You can make t large (and p small) either by having a large d or a large n. Effect size and significance are two separate quantities measuring two separate things. This is why well-reported results always include both: the p-value is evidence the effect is real; the effect size is evidence the effect matters.