Questions: Hypothesis Testing: Framework and Logic

Question 1

A researcher tests whether a new drug reduces blood pressure and finds p = 0.02 with α = 0.05. Which interpretation is correct?

• A. There is a 2% chance the drug has no effect — the p-value is the probability H₀ is true
• B. If the drug truly had no effect, there is only a 2% chance of observing data this extreme or more extreme
• C. The researcher can be 98% confident the drug is effective
• D. The drug reduces blood pressure in 98% of patients

Answer: (B)

The p-value is P(data this extreme | H₀ true) — a conditional probability with H₀ in the condition. It is NOT P(H₀ true | data). Option A is the most common and consequential misinterpretation of p-values. The p-value tells you how surprising your data would be in a world where the null is true, not how likely the null is to be true given your data.

Question 2

A study comparing two teaching methods finds p = 0.30, with α = 0.05. The researchers fail to reject H₀. What is the most accurate conclusion?

• A. The null hypothesis is proven true — the two methods are equally effective
• B. The alternative hypothesis is false
• C. The data are not surprising enough under H₀ to cross the pre-set rejection threshold
• D. The study was conducted incorrectly and should be repeated

Answer: (C)

Failing to reject H₀ is not the same as proving H₀ true. The data are simply insufficiently surprising under H₀ — the result is consistent with both H₀ being true and H₀ being false but the study being underpowered. This asymmetry is fundamental: hypothesis tests can produce evidence against H₀, but 'no evidence against' is not the same as 'evidence for.'

Question 3

A p-value of 0.03 means there is a 3% chance the null hypothesis is true.

• T. True
• F. False

Answer: False

This is the single most common p-value misinterpretation. The p-value is P(observing data this extreme or more | H₀ is true) — it conditions on H₀ being true. Computing P(H₀ is true | data) requires knowing the prior probability of H₀, which the frequentist hypothesis testing framework does not provide. The p-value tells you how unusual your data are under H₀, not how likely H₀ is.

Question 4

The significance level α must be chosen before seeing the data, not after, in order to maintain a valid Type I error rate.

• T. True
• F. False

Answer: True

Choosing α after seeing the data — for example, setting α = 0.06 to just barely reject a null you already know produced p = 0.05 — is a form of p-hacking that inflates the true Type I error rate above the nominal α. The error-rate guarantee of hypothesis testing only holds when the decision rule is fixed independently of the data. Pre-registration of α is the mechanism that prevents this form of circular reasoning.

Question 5

What is a p-value, and why does it not tell you the probability that the null hypothesis is true?

Model answer: A p-value is the probability of observing a test statistic as extreme as (or more extreme than) the one computed, assuming the null hypothesis is true. It conditions on H₀ — it tells you how surprising your data would be in a world where H₀ holds. The probability that H₀ is true given the data (P(H₀|data)) is a different quantity that requires a prior probability for H₀, which frequentist hypothesis testing does not specify.

The confusion arises from inverting the conditional. P(data | H₀) is what the p-value measures. P(H₀ | data) is what people want it to mean. These are related by Bayes' theorem but are not equal unless P(H₀) = P(data), which is rarely the case. Recognizing this distinction is essential for correctly interpreting any statistical test.