Questions: Absence of Evidence Is Evidence of Absence

Question 1

A company claims its supplement boosts immune function. Researchers conduct a large, well-designed double-blind trial and find no statistically significant effect. The company responds: 'Absence of evidence is not evidence of absence — you just haven't proven it doesn't work.' What is the most accurate Bayesian reply?

• A. The company is correct — a null result only means the study lacked statistical power
• B. The null result does lower the probability that the supplement works, in proportion to how reliably a real effect would have been detected
• C. Absence of evidence only matters if the study found positive evidence of harm
• D. We cannot update our probability estimate in either direction from a null result

Answer: (B)

If the supplement works, a well-designed large trial should detect the effect with high probability. The null result is therefore more likely under 'supplement doesn't work' than under 'supplement works.' By Bayes' theorem, this shifts probability toward the hypothesis that the supplement doesn't work. The company's defense ('you haven't proved absence') conflates logical proof with probabilistic evidence. The null result is genuine evidence against efficacy — its strength depends on the trial's power to detect a real effect.

Question 2

Hypothesis H predicts that observable event E will occur with probability 0.95 if H is true, and E occurs with probability 0.05 if H is false. You look for E and do not find it. How should you update your belief in H?

• A. Do not update — absence of evidence is never informative
• B. Update weakly against H — since E is expected, not finding it is only mildly surprising
• C. Update strongly against H — the likelihood ratio of not-E is P(¬E|H false)/P(¬E|H true) = 0.95/0.05 = 19:1 in favor of ¬H
• D. Update in favor of H — the rarity of not-E under H-false means H is more likely

Answer: (C)

The likelihood ratio for not-E is P(¬E | H false)/P(¬E | H true) = (1−0.05)/(1−0.95) = 0.95/0.05 = 19. This means observing ¬E is 19 times more likely if H is false than if H is true — a very strong update against H. When a hypothesis strongly predicts we should see evidence and we don't see it, that absence is powerful evidence against the hypothesis. This is the Bayesian cash-out of 'absence of evidence is evidence of absence.'

Question 3

If a hypothesis predicts observable consequences that we look for and fail to find, failing to find them is evidence against the hypothesis.

• T. True
• F. False

Answer: True

This is the central claim. The strength of the evidence depends on the likelihood ratio: how much more probable is the absence of evidence if the hypothesis is false versus if it is true? When the hypothesis strongly predicts the evidence, a failure to find it is a large update against the hypothesis. The popular saying 'absence of evidence is not evidence of absence' is, in the probabilistic sense, simply wrong in this case.

Question 4

The evidential weight of failing to find expected evidence is the same regardless of how thoroughly and carefully we searched.

• T. True
• F. False

Answer: False

The strength of absence-as-evidence depends entirely on P(evidence | hypothesis true) — how likely we were to find the evidence if the hypothesis were actually true. A cursory search that would miss most evidence even if the hypothesis were true yields a weak update. A thorough search that would reliably detect evidence if the hypothesis were true yields a strong update. 'We looked under the couch' is weak absence-of-evidence for 'the keys aren't in this house.' A full building sweep is strong absence-of-evidence.

Question 5

Under what conditions does failing to find evidence strongly support absence, and when does it barely matter? Explain using the concept of likelihood ratios.

Model answer: Absence of evidence strongly supports absence when the hypothesis predicts that observable evidence would appear with high probability — i.e., P(evidence | hypothesis true) is large. In that case, not finding the evidence gives a large likelihood ratio in favor of the hypothesis being false: P(¬E | H false)/P(¬E | H true) = (1 − P_base)/(1 − P_predicted). When the hypothesis makes a weak prediction — the evidence might not appear even if the hypothesis is true — not finding it barely updates us at all, because the likelihood ratio approaches 1.

The classic example is Sherlock Holmes's 'the dog that did not bark in the night.' If the dog would reliably bark at an intruder, and the dog did not bark, this is strong evidence no intruder came. If the dog only sometimes barks at intruders, silence is weak evidence. The evidential weight of absence scales with how confidently the hypothesis predicts the evidence's presence.