A system has 100 possible microstates. A measurement reveals with certainty that it is in microstate #47. What is the Shannon entropy of this distribution?
Aln 100 — because there are 100 possible states in the system
B1/100 — the probability of any one state in the uniform distribution
C0 — complete knowledge of the microstate means zero uncertainty
D100 × (1/100) × ln 100 = ln 100 — by summing over all states
If the system is certainly in microstate #47, then p₄₇ = 1 and all other pᵢ = 0. Shannon entropy H = −Σ pᵢ ln pᵢ = −(1 × ln 1) − (99 × 0 × ln 0) = 0. (The 0 ln 0 terms are zero by convention.) Zero entropy means zero uncertainty — you know exactly which microstate the system occupies. Options A and D both give ln 100, which is the entropy of the *uniform* distribution over 100 states — the maximum-ignorance case. Option B gives a single probability value, not entropy.
Question 2 Multiple Choice
Jaynes' maximum entropy principle says the correct statistical mechanical ensemble is the distribution that maximizes Shannon entropy subject to known constraints. For the canonical ensemble (fixed mean energy ⟨E⟩), this yields:
AThe uniform distribution over all microstates, because maximum entropy always means maximum uniformity
BA distribution concentrated on the single lowest-energy microstate
CThe Boltzmann distribution pᵢ ∝ e^{−βEᵢ}, where β is a Lagrange multiplier enforcing the mean energy constraint
DA distribution proportional to the energy of each microstate
When you maximize H = −Σ pᵢ ln pᵢ subject to the constraint Σ pᵢ Eᵢ = ⟨E⟩ (and normalization), the Lagrange multiplier method yields pᵢ ∝ e^{−βEᵢ} — the Boltzmann distribution. β = 1/kT is the Lagrange multiplier for the energy constraint. The uniform distribution (option A) is the maximum-entropy solution only when there are *no* constraints beyond normalization — the microcanonical case where all accessible states are equiprobable. Options B and D are incorrect; the Boltzmann distribution naturally weights lower-energy states more heavily but is not concentrated on any single state.
Question 3 True / False
The statement 'entropy increases in an isolated system' is equivalent to saying that our knowledge of the system's precise microstate increases over time.
TTrue
FFalse
Answer: False
This reverses the information-theoretic meaning. Entropy is a measure of *uncertainty* or *ignorance* — it is maximized when the distribution is most spread out (maximum uncertainty). 'Entropy increases' means our knowledge of the microstate *decreases* — the system evolves into a larger space of accessible microstates, and our information about which one it occupies diminishes. The correct restatement is: isolated systems evolve toward states of maximum uncertainty (maximum Shannon entropy), meaning minimal knowledge of the precise microstate.
Question 4 True / False
Boltzmann's formula S = k ln Ω is a special case of Shannon entropy, arising when all accessible microstates are equally probable.
TTrue
FFalse
Answer: True
If all Ω microstates are equally probable, then pᵢ = 1/Ω for all i. Shannon entropy H = −Σ pᵢ ln pᵢ = −Ω × (1/Ω) × ln(1/Ω) = ln Ω. Multiplying by Boltzmann's constant k gives S = k ln Ω — exactly Boltzmann's formula. This is the microcanonical ensemble result. The Shannon entropy formula is the general case; Boltzmann's formula applies when the system is isolated and all accessible microstates are equiprobable. This equivalence confirms that statistical mechanical and information-theoretic entropy are the same concept.
Question 5 Short Answer
Why does 'entropy increases' mean the same thing as 'our knowledge of the microstate decreases,' and what does this imply about the direction of time?
Think about your answer, then reveal below.
Model answer: Shannon entropy measures uncertainty — how spread out a probability distribution is. When entropy increases, the distribution over microstates becomes more spread out, meaning we know less about which specific microstate the system is in. A gas expanding into a vacuum has more accessible microstates; the same energy is consistent with exponentially more arrangements, so our knowledge of where the molecules are diminishes. This interpretation implies that the arrow of time is the direction in which information about the microstate is lost. We experience time as directed from past to future because high-entropy (high-ignorance) states vastly outnumber low-entropy ones, so random evolution almost always increases entropy — and thus decreases our knowledge.
This connection between time and information loss was developed by Maxwell, Boltzmann, and later Jaynes. The past feels different from the future because we retain memories (information) of past states but not future ones — and memories are low-entropy records. The second law says that isolated systems evolve toward maximum ignorance. The 'arrow of time' is not a feature of the fundamental laws (which are time-symmetric) but emerges from the statistical tendency toward higher entropy — toward states about which we know less.