Mixture Models and Gaussian Mixture Models

Core Idea

Mixture models represent data as weighted combinations of K component distributions. Gaussian Mixture Models (GMM) use Gaussian components fit via EM. GMMs provide soft assignments (membership probabilities) unlike k-means' hard assignments. GMMs enable principled model selection via likelihood and provide density estimation.

Explainer

You already know k-means clustering: assign each data point to its nearest centroid, recompute centroids, repeat. K-means works well when clusters are roughly spherical and equally sized, but it has a fundamental limitation — every point belongs to exactly one cluster with no uncertainty. Real data is messier. A data point sitting between two clusters might genuinely belong to either one, and k-means gives you no way to express that ambiguity. Mixture models fix this by treating clustering as a probability problem.

A Gaussian Mixture Model (GMM) assumes your data was generated by K Gaussian (normal) distributions, each with its own mean, covariance, and mixing weight (the prior probability that a random point came from that component). The mixing weights sum to 1. For any data point, you can compute a responsibility — the posterior probability that component k generated this point, using Bayes' theorem with the component densities you studied in probability. A point near the boundary of two clusters might have responsibilities of 0.6 and 0.4, capturing genuine uncertainty that k-means throws away.

Fitting a GMM means finding the means, covariances, and mixing weights that maximize the likelihood of the observed data. This is where your knowledge of expectation-maximization (EM) becomes essential. Direct optimization of the likelihood is intractable because the log of a sum has no clean closed form. EM sidesteps this by alternating between two steps: the E-step computes responsibilities using current parameters (which component likely generated each point?), and the M-step updates parameters using those responsibilities as soft weights (recompute each component's mean and covariance, weighted by how much each point "belongs" to it). Each iteration increases the likelihood, and the algorithm converges to a local maximum. Notice the parallel to k-means: k-means is actually a special case of GMM where covariances are fixed as identity matrices and responsibilities are forced to 0 or 1.

Because GMMs are proper probabilistic models, they unlock capabilities that k-means cannot offer. You can evaluate the likelihood of new data points, enabling density estimation — modeling the overall shape of the data distribution, not just cluster assignments. You can use information criteria like BIC or AIC to select the number of components K in a principled way, rather than relying solely on heuristics like the elbow method. And because each component has its own covariance matrix, GMMs naturally handle elliptical, elongated, or rotated clusters that would confuse k-means. The cost is computational: each EM iteration requires computing responsibilities across all points and components, and the result depends on initialization — running multiple restarts helps avoid poor local optima.