A company surveys 500,000 customers via opt-in email and reports 88% satisfaction. A competitor surveys 2,000 randomly selected customers by phone and reports 71% satisfaction. Which survey more reliably estimates the true satisfaction level?
AThe company's survey, because 500,000 is a vastly larger sample
BThe competitor's survey, because random selection reduces selection bias more than raw sample size does
CThey are equally reliable since both used real customers
DNeither can be reliable without knowing the full population size
Representativeness beats size when a sample is biased. The opt-in email survey systematically over-represents satisfied customers (dissatisfied ones are less likely to respond), so adding more biased observations just reinforces the wrong answer with greater precision. A well-drawn random sample of 2,000 gives each customer equal inclusion probability, producing an unbiased estimate. The 1936 Literary Digest poll (10 million responses, catastrophically wrong) is the canonical example of size failing to fix bias.
Question 2 Multiple Choice
A medical test is 99% accurate. The disease it screens for affects 1 in 1,000 people. If a randomly selected person tests positive, approximately what is the probability they actually have the disease?
AAbout 99%, since the test is 99% accurate
BAbout 50%, since a test is either right or wrong
CAbout 9%, because the low base rate means most positives are false positives
DAbout 0.1%, equal to the disease prevalence
This is the base rate neglect problem. Out of 1,000 people: roughly 1 has the disease (true positive with 99% chance ≈ 1 person) and 999 do not (false positive rate of 1% ≈ 10 people). So about 11 people test positive, and only 1 of those ~11 actually has the disease — roughly 9%. The test's accuracy sounds impressive but is swamped by the rarity of the disease. Ignoring the base rate leads to wildly overestimating what a positive result means.
Question 3 True / False
A study with 100,000 participants finds a statistically significant difference between two groups (p < 0.001). This means the difference is practically important.
TTrue
FFalse
Answer: False
False — statistical significance only means the result is unlikely to have arisen by chance at the chosen threshold. With 100,000 participants, even a trivially small difference (say, a 0.01% improvement) can reach p < 0.001. The p-value says nothing about whether the effect is large enough to matter in the real world. Effect size — how big the difference actually is — determines practical importance. 'Significant' in statistics means 'detectable,' not 'meaningful.'
Question 4 True / False
A random sample of 1,000 people is generally more reliable than a convenience sample of 100,000 people for estimating a population proportion.
TTrue
FFalse
Answer: True
True — a convenience sample systematically excludes or over-represents certain groups, producing a biased estimate regardless of size. Adding more observations from the same biased pool does not correct the bias; it reinforces it. A well-drawn random sample of 1,000 gives each population member equal inclusion probability, making it representative and its error quantifiable (margin of error ≈ 1/√1000 ≈ ±3%). Larger but biased samples give precise estimates of the wrong thing.
Question 5 Short Answer
Explain why a biased sample of one million people may give worse results than a random sample of one thousand people.
Think about your answer, then reveal below.
Model answer: Bias is a systematic error — the sample consistently mis-represents the population regardless of size. Adding more biased observations doesn't reduce the error; it just makes you more confident in the wrong answer. A random sample of 1,000 has only random (unsystematic) sampling error, which averages out and is quantifiable via margins of error. The biased sample produces a precise estimate of the wrong quantity. Size reduces random error; it cannot fix systematic error.
The Literary Digest example illustrates this perfectly: 10 million responses from car and telephone owners in 1936 systematically missed the voters who actually elected Roosevelt. More data from the same biased pool just compounded the false confidence. Representativeness determines whether your sample asks the right question; size determines how precisely you answer it.