Questions: Survey Design, Construction, and Administration

Question 1

Researcher A surveys 10,000 people from a carefully constructed probability sample and gets a 40% response rate. Researcher B posts a survey link on social media and gets 2,000 voluntary responses — a much higher effective rate. Whose results are more generalizable to the general population?

• A. Researcher B, because more responses and a higher participation rate always mean better data
• B. Researcher A, because probability sampling — not response rate — determines generalizability
• C. Neither, because 40% is too low to draw any conclusions regardless of sampling method
• D. They are equivalent because sample size and response rate compensate for each other

Answer: (B)

Generalizability depends on whether every member of the target population had a known, nonzero chance of being selected — that is, probability sampling. Researcher A used probability sampling, so statistical inference to the population is valid despite the moderate response rate. Researcher B's volunteers are self-selected; there is no way to calculate or correct for selection bias because the sampling frame is undefined. A high response rate from a biased pool is less valuable than a moderate rate from a proper probability sample.

Question 2

A survey item reads: 'Don't you agree that the current administration has done a poor job managing the economy?' What is the primary flaw in this question?

• A. Acquiescence bias — the yes/no format causes respondents to agree regardless of their true opinion
• B. Double-barreled question — it asks about two separate issues in one item
• C. Leading question — it embeds an evaluative frame that pulls responses toward a predetermined answer
• D. Social desirability bias — respondents will answer based on what they think the researcher wants to hear

Answer: (C)

A leading question contains language that signals the 'correct' or expected answer, distorting responses away from genuine opinion. 'Don't you agree' presupposes agreement, and 'poor job' is an explicit negative evaluation embedded in the question stem. Acquiescence bias (A) is a respondent tendency to agree with any statement, which interacts with this flaw but is not the flaw itself. Social desirability bias (D) applies to sensitive self-disclosures, not to politically framed evaluations of others.

Question 3

Acquiescence bias — the tendency to agree with survey statements regardless of content — can be partially controlled by including reverse-scored items in the instrument.

• T. True
• F. False

Answer: True

Reverse-scored items state the opposite of what the positive items state. If a respondent agrees with everything, their agreement on the reverse item contradicts their agreement on the positive item, revealing the bias statistically. By comparing scores on positively and negatively worded versions of the same construct, researchers can detect and partially correct for acquiescence. This is why well-designed attitude scales often include both 'I feel confident in social situations' and 'I often feel uncomfortable in social situations.'

Question 4

A high survey response rate is the most important indicator of data quality because it ensures the respondents represent the target population.

• T. True
• F. False

Answer: False

Response rate measures how many of the contacted people responded — it says nothing about whether the contacted people were the right people. A 95% response rate from a non-representative sampling frame produces systematically biased data. Representativeness is determined by the sampling method (probability vs. non-probability) and whether the sampling frame covers the target population. What matters is not how many responded, but whether the people who responded are representative of the people the researcher wanted to describe.

Question 5

Why is 'a high response rate' insufficient to guarantee that a survey's results are valid or generalizable to the target population?

Model answer: Response rate only tells you what fraction of the people you contacted actually responded. It says nothing about whether those contacted people were the right people — whether the sampling frame represents the target population. If you survey a convenience sample (e.g., social media followers, mall intercepts) and get a 90% response rate, you have a complete picture of a biased subset, not the population. Generalizability requires probability sampling, where every member of the target population has a known, nonzero selection probability. A moderate response rate from a probability sample produces valid inferences; a high response rate from a poorly defined frame does not.

The misconception equates participation with representation. These are independent dimensions: participation (response rate) measures effort and engagement; representation (sampling method) measures whether the right people were contacted in the first place. A truly random sample of 60% respondents is more valuable than a self-selected 90% because statistical theory can quantify uncertainty around the random sample — no such theory applies to unknown selection biases in the volunteer sample.