Questions: Big Data Collection and Analysis in Social Science

Question 1

A researcher scrapes 50 million tweets to study public opinion on immigration policy. She finds 62% of tweets express negative views and concludes that most people oppose more permissive immigration policy. What is the most significant flaw in this reasoning?

• A. The sample size is too small — she needs at least 100 million tweets for reliable conclusions
• B. Twitter users are not representative of the general population, so the data suffers from selection bias unrelated to sample size
• C. Sentiment analysis tools cannot accurately classify tweets, introducing measurement error
• D. She should have used a different platform with more users, like Facebook, for better coverage

Answer: (B)

This is the selection bias problem endemic to big data. Twitter users are systematically younger, more urban, more politically engaged, and more extreme in expressed opinions than the general population. No amount of additional tweets fixes this — the bias comes from who generates Twitter data in the first place, not how much of it you collect. Switching to Facebook (option D) would have different but equally systematic biases. This is fundamentally different from survey sampling bias: you cannot correct for platform selection bias by collecting more data from the same platform.

Question 2

A study using social media data finds that people who post frequently about social activities report higher loneliness on follow-up surveys. A researcher concludes that active social media use causes loneliness. What is the primary methodological concern?

• A. The study should have used a control group of people with no social media accounts
• B. Posting frequency is a behavioral metric that may not validly represent social engagement or loneliness as theoretical constructs
• C. The sample size is insufficient for causal claims about such a complex phenomenon
• D. Longitudinal designs cannot establish causation — only experimental designs can

Answer: (B)

This is the construct validity problem. Posting frequency is a digital behavior metric, not a direct measure of social engagement or loneliness. People post frequently for many reasons — performance, boredom, professional necessity — that have no connection to how socially connected they feel. The gap between what the data records (posting behavior) and what the researcher wants to measure (loneliness) is a construct validity problem that cannot be fixed by having more data. Big data provides scale but not automatic validity — connecting digital behavior metrics to underlying social constructs requires careful theoretical justification.

Question 3

Collecting a larger dataset in big data research eliminates selection bias by including more observations from the target population.

• T. True
• F. False

Answer: False

False. Selection bias in big data operates at the platform level, not the sample level. If Twitter users are systematically different from the general population, then collecting every single tweet ever posted still gives you a biased picture of that general population — because the bias comes from who uses Twitter, not from sampling within the Twitter population. More observations from a biased source gives you a more precise measure of the biased population, not a less biased measure of the target population. This is different from traditional random sampling, where increasing N does reduce sampling error.

Question 4

Big data's scale can reveal patterns impossible to detect in smaller datasets, but it amplifies the consequences of poor research design rather than compensating for it.

• T. True
• F. False

Answer: True

True. Scale is a double-edged property. With millions of observations, researchers can detect tiny effect sizes that would be statistically undetectable in smaller samples — including trivially small effects that are theoretically meaningless. Statistical significance becomes nearly guaranteed at large N even for noise. Meanwhile, systematic biases (selection bias, measurement invalidity) are not reduced by scale — they are replicated consistently across millions of observations. The principle that a large N does not substitute for a clear research question or valid measurement instrument is fundamental to responsible big data social science.

Question 5

Explain why selection bias in big data research is different from selection bias in traditional survey research, and why increasing the dataset size cannot fix it.

Model answer: In traditional survey research, selection bias arises from who responds to an invitation — and you can potentially reduce it by improving response rates or weighting by known population characteristics. In big data, selection bias arises from who uses the platform at all — who generates the digital traces in the first place. Twitter users, credit card holders, and smartphone location data sources are systematically non-representative subpopulations defined by age, income, geography, and technology access. No amount of additional data from the same platform closes this gap, because the unrepresented people are simply absent from the data-generating process entirely.

This distinction matters because it means the traditional 'more data = less bias' intuition breaks down for big data. A 50-million-tweet dataset has exactly the same structural selection bias as a 1,000-tweet dataset on Twitter. The fix is not to collect more data from the same source but to understand the population that source represents, be explicit about that population in claims, or supplement with data from other sources that cover the underrepresented groups.