A pharmaceutical trial with n = 500,000 participants per group finds a statistically significant reduction in cholesterol (p < 0.0001, Cohen's d = 0.04). What is the correct interpretation?
AThe drug is highly effective because the p-value is very small
BThe result is probably a false positive despite the significant p-value
CThe drug produces a real but negligibly small effect — statistical significance does not imply practical importance
DThe sample size is too large for p-values to be meaningful
This is the classic disconnect between statistical and practical significance. A p-value of < 0.0001 means we are very confident the effect is real — the data would be extraordinarily unlikely under the null. But Cohen's d = 0.04 means the difference is only 0.04 standard deviations, negligible by any clinical standard. The enormous sample size is what drove a trivially small effect to statistical significance. The correct conclusion: real, but practically meaningless.
Question 2 Multiple Choice
What does Cohen's d measure that a p-value cannot?
AWhether the observed result would be unlikely by chance
BThe probability that the null hypothesis is true
CThe magnitude of the difference between groups, in units of pooled standard deviations
DThe confidence level at which we can reject the null hypothesis
Cohen's d = (μ₁ − μ₂) / σ_pooled standardizes the difference by the pooled standard deviation, expressing 'how many standard deviations apart are the two groups?' This magnitude measure does not change with sample size. A p-value answers 'how unlikely is this data under the null?' — which depends heavily on n. Cohen's d answers 'how large is the effect?' — which is independent of how many people you measured.
Question 3 True / False
A study can produce a statistically significant result (p < 0.05) even when the true effect size is negligibly small.
TTrue
FFalse
Answer: True
Statistical significance depends on both effect size and sample size. With a large enough sample, even an infinitesimally small effect will eventually achieve significance — the test has enough power to detect essentially any departure from zero. A study with n = 10,000,000 could find p < 0.001 for an effect of d = 0.001, which is practically invisible. Reporting only the p-value can therefore be deeply misleading without an accompanying effect size.
Question 4 True / False
A p-value of 0.001 tells us that the observed effect is large enough to be practically important.
TTrue
FFalse
Answer: False
P-values and effect sizes measure entirely different things. A p-value of 0.001 tells us there is very strong evidence against the null hypothesis — it says nothing about the size of the effect. The effect could be real but minuscule (d = 0.01) and still yield a tiny p-value given sufficient sample size. Practical importance depends on effect size, not on how confident we are that an effect exists.
Question 5 Short Answer
Why is it insufficient to report only a p-value when presenting hypothesis test results? What additional information is needed, and what does it tell us?
Think about your answer, then reveal below.
Model answer: A p-value only answers 'are we confident the effect isn't zero?' It says nothing about how large the effect is. An effect size measure (such as Cohen's d, r, or R²) answers 'how large is the effect, and does it matter?' The p-value establishes that an effect exists; the effect size establishes whether it is worth caring about. Together they provide a complete picture: confident it's real (p-value) and knowing whether it's meaningful (effect size).
A drug that achieves p < 0.001 with d = 0.03 has a real but clinically useless effect; prescribing it based on the p-value alone would be a mistake. Conversely, a study with d = 1.5 and p = 0.09 found a potentially large effect that the small sample couldn't confirm at conventional thresholds — dismissing it as 'not significant' would also be wrong. Neither piece of information is complete without the other.