Questions: Mixture Models and Gaussian Mixture Models

Question 1

Two clusters overlap significantly in a 2D dataset. A GMM and k-means are both fitted with K=2. For a point located exactly between the two cluster centers, which statement best describes the difference in outputs?

• A. K-means and GMM give identical cluster assignments; they differ only in how they handle outliers
• B. K-means assigns the point to exactly one cluster; the GMM assigns it fractional membership in both components, reflecting genuine uncertainty
• C. The GMM assigns the point to the component with the higher covariance; k-means uses only distance to the nearest centroid
• D. The GMM will identify more clusters than k-means in overlapping regions

Answer: (B)

This is the fundamental conceptual difference. K-means makes hard assignments — every point belongs to exactly one cluster, with no expression of uncertainty. A GMM computes responsibilities: the posterior probability that each component generated the point. A boundary point might have responsibilities of 0.55 and 0.45 — the GMM captures that genuine ambiguity. K-means would force a binary choice. Option C conflates responsibility computation with covariance magnitude, which is incorrect.

Question 2

Why is the EM algorithm used to fit GMMs rather than directly maximizing the likelihood?

• A. EM is faster because it applies gradient descent to the log-likelihood using automatic differentiation
• B. Direct optimization is intractable because the log of a sum of Gaussian densities has no closed-form solution; EM sidesteps this by alternating between computing responsibilities and updating parameters
• C. The GMM likelihood surface is convex, and EM is the standard solver for convex problems
• D. EM imposes hard assignments in the E-step that make the M-step analytically tractable by reducing to a k-means update

Answer: (B)

The log-likelihood of a GMM is log Σₖ πₖ N(x; μₖ, Σₖ) — a log of a sum, which has no closed-form derivative that sets to zero. EM circumvents this by introducing latent component assignments: the E-step computes soft responsibilities using current parameters (effectively completing the data), and the M-step re-estimates parameters given those responsibilities, yielding clean closed-form updates. Option D is wrong — EM's E-step gives soft (fractional) responsibilities, not hard assignments; hard assignments would reduce it to k-means, which is a degenerate limiting case.

Question 3

K-means clustering is a special case of Gaussian Mixture Models where the covariance of each component is fixed as the identity matrix and responsibilities are forced to be 0 or 1.

• T. True
• F. False

Answer: True

This is a precise and important connection. When all GMM components are constrained to spherical equal-variance Gaussians (Σₖ = σ²I for the same σ) and the EM soft responsibilities are replaced by hard 0/1 assignments (winner-takes-all), the EM algorithm reduces exactly to the k-means algorithm. The E-step becomes nearest-centroid assignment and the M-step becomes centroid recomputation. GMMs generalize k-means by allowing elliptical clusters (full covariance matrices) and probabilistic membership.

Question 4

A GMM with K components assigns each data point to the single component whose mean is closest to that point, analogous to k-means centroid assignment.

• T. True
• F. False

Answer: False

This describes k-means, not GMMs. A GMM assigns each point a responsibility for every component based on the full Gaussian density — which depends on both the mean distance AND the covariance shape of each component. A point may be close to one component's mean but have low responsibility if that component has a small variance (tight cluster) while another component's covariance stretches toward the point. The GMM accounts for cluster shape and uncertainty; k-means considers only distance to the nearest centroid.

Question 5

What does it mean for a GMM to perform 'density estimation,' and why is k-means fundamentally incapable of this?

Model answer: Density estimation means modeling the probability density of the data — being able to evaluate p(x) for any new data point x. A GMM is a proper probabilistic model: it defines p(x) = Σₖ πₖ N(x; μₖ, Σₖ), so you can compute the likelihood of any new observation under the fitted model. K-means produces only a partition of space into Voronoi regions; it assigns labels but assigns no probability to locations. It cannot tell you how likely a new point is, whether it comes from any cluster, or how the overall data is distributed.

The distinction matters for tasks like anomaly detection (flagging new points with low density), generative modeling (sampling new plausible data points), and model selection (using likelihood-based criteria like BIC to choose K). None of these are possible with k-means. The GMM's probabilistic foundation is not just a mathematical nicety — it unlocks capabilities that fundamentally require knowing p(x), and k-means discards exactly that information by using hard, non-probabilistic assignments.