Questions: Joint and Conditional Entropy

Question 1

If X and Y are independent random variables, which relationship holds?

• A. H(X,Y) = H(X) * H(Y)
• B. H(X,Y) = H(X) + H(Y) and H(Y|X) = H(Y)
• C. H(X,Y) = max(H(X), H(Y))
• D. H(Y|X) = 0 because knowing X fully determines Y

Answer: (B)

When X and Y are independent, knowing X tells you nothing about Y, so H(Y|X) = H(Y). The chain rule then gives H(X,Y) = H(X) + H(Y|X) = H(X) + H(Y). Joint entropy is additive for independent variables. Option 3 (H(Y|X) = 0) describes the opposite extreme: perfect dependence, where X completely determines Y.

Question 2

A dataset contains patient records with variables Disease (D) and Symptom (S). A researcher finds H(S|D) = 0.2 bits and H(S) = 3.1 bits. What does this tell you?

• A. Symptoms are nearly useless for diagnosing disease
• B. Knowing the disease leaves very little residual uncertainty about symptoms — the disease almost completely determines which symptoms appear
• C. The entropy of disease is 2.9 bits
• D. The symptom variable has very low entropy overall

Answer: (B)

H(S|D) = 0.2 bits means that once you know the disease, only 0.2 bits of uncertainty about symptoms remains (out of the original 3.1 bits). The disease explains most of the symptom variation. This does NOT directly tell us H(D) — the chain rule gives H(D,S) = H(D) + H(S|D), not H(S) - H(S|D) = H(D). The quantity H(S) - H(S|D) = 2.9 bits is the mutual information I(D;S), not H(D).

Question 3

Conditioning always reduces entropy: H(Y|X) <= H(Y). This means that for every specific value x, H(Y|X=x) <= H(Y).

• T. True
• F. False

Answer: False

The inequality H(Y|X) <= H(Y) holds ON AVERAGE — it says the expected conditional entropy is at most the marginal entropy. But for specific values of x, H(Y|X=x) can be greater than H(Y). For example, if X usually gives a lot of information about Y but for one rare value x* it creates ambiguity, then H(Y|X=x*) could exceed H(Y). The inequality is about the weighted average across all x, not about every individual x.

Question 4

Derive the chain rule for entropy H(X,Y) = H(X) + H(Y|X) from the definition of joint and conditional entropy, and explain why the decomposition is asymmetric.

Model answer: Starting from H(X,Y) = -sum_{x,y} p(x,y) log p(x,y), use p(x,y) = p(x)*p(y|x). Then log p(x,y) = log p(x) + log p(y|x), so H(X,Y) = -sum_{x,y} p(x,y) log p(x) - sum_{x,y} p(x,y) log p(y|x). The first sum simplifies to H(X) (summing out y gives the marginal). The second sum is H(Y|X) by definition. The decomposition is asymmetric: H(X,Y) = H(X) + H(Y|X) = H(Y) + H(X|Y), but H(Y|X) != H(X|Y) in general. Knowing X may reduce uncertainty about Y by a different amount than knowing Y reduces uncertainty about X.

Both orderings are valid chain rules and give the same joint entropy. The asymmetry reflects a real phenomenon: in a teacher-student pair, knowing the teacher's grade assignment might almost determine the student's grade (low H(student|teacher)), but knowing the student's grade may leave substantial uncertainty about the teacher's specific rubric (high H(teacher|student)).