Survey A on voting intentions achieves an 85% response rate but recruits participants primarily through civic associations, overrepresenting politically engaged citizens. Survey B on the same topic achieves a 30% response rate, but analysis shows that non-respondents are similar to respondents across income, age, and prior voting behavior. Which survey is likely more biased?
ASurvey B, because a 30% response rate is too low to support valid inference regardless of who responds
BSurvey A, because an 85% response rate is suspiciously high and likely reflects poor sampling
CSurvey A, because its non-response is systematic — the missing 15% are not randomly distributed on the variable of interest
DBoth equally — response rate and non-response patterns are independent sources of error that always matter equally
Non-response bias depends on whether those who don't respond differ from those who do on the variable of interest — not on response rate alone. Survey A's high response rate does not protect it from bias if the recruitment mechanism systematically excludes low-engagement voters. Survey B's 30% rate could be unbiased if the non-respondents are genuinely random with respect to voting intention. The key diagnostic question is always: do non-respondents differ systematically from respondents on the outcome variable?
Question 2 Multiple Choice
A researcher surveys 1,000 adults about their use of social media and gets a 60% response rate. She later finds that heavy social media users were significantly more likely to complete the survey. What is the most accurate description of the situation?
AThe results are reliable because 600 respondents is a large sample
BThere is likely non-response bias — heavy users are overrepresented because they differ systematically from non-respondents on the key variable
CThe 40% non-response rate guarantees bias; the only solution is to re-survey all non-respondents
DSince response rate is above 50%, statistical inference is valid and no adjustment is needed
Non-response bias is present when non-respondents differ from respondents on the variable being measured. Here, social media use itself predicts survey completion — exactly the situation where bias is most severe. Sample size is irrelevant: a large biased sample is still biased. The 40% non-response rate is not automatically damning (it depends on who the 40% are), but finding that they differ systematically on the key variable confirms bias. No response-rate threshold automatically guarantees validity.
Question 3 True / False
A double-barreled question ('Do you support raising the minimum wage and strengthening unions?') can produce misleading data even if every respondent answers completely honestly.
TTrue
FFalse
Answer: True
A double-barreled question asks about two distinct issues in a single item. Respondents who support one but not the other must choose a single answer that misrepresents their view. The resulting data conflates agreement with two separate policies, making it impossible to know which sub-question the respondent was answering. Even perfectly honest responses produce uninterpretable data. This is a question construction error, not a respondent error — it is caught through cognitive interviewing before fieldwork begins.
Question 4 True / False
A survey with a low response rate is typically more biased than one with a high response rate, because fewer respondents means less representation of the population.
TTrue
FFalse
Answer: False
Response rate alone does not determine bias. Bias depends on whether non-respondents differ systematically from respondents on the variables of interest. A survey with a 25% response rate may produce unbiased estimates if non-response is random. A survey with an 80% response rate may be severely biased if the 20% who did not respond share a systematic characteristic relevant to the outcome. Researchers assess non-response bias by comparing respondents to known population benchmarks and by characterizing non-respondents through follow-up — not simply by checking the response rate.
Question 5 Short Answer
How can a researcher assess whether non-response has biased a survey, even when the non-respondents themselves cannot be surveyed?
Think about your answer, then reveal below.
Model answer: Compare the demographic and behavioral profile of respondents to known population benchmarks from census or administrative data. If respondents match the population on observable characteristics (age, income, education, geography), bias on unobservables is less likely. Additionally, follow up with a random subsample of non-respondents using a shorter instrument or incentive, and compare their responses to those of original respondents on key variables. Large differences signal bias; similarity suggests the non-response may be ignorable.
This strategy — benchmark comparison plus non-respondent follow-up — is the standard approach precisely because you cannot survey people who decline to participate. Neither method is perfect: population benchmarks only cover observable variables, and non-respondent follow-up reaches only the more cooperative non-respondents. But together they give a much more informative picture than the response rate alone. The core principle is: assess bias by examining what you know about the non-respondents, not just how many of them there are.