Information Geometry Advanced

Core Idea

Advanced information geometry explores the dually flat structure of statistical manifolds, where exponential families and mixture families sit in geometric duality. The alpha-connection (a family of connections parameterized by alpha in [-1, 1]) interpolates between the exponential (e) connection (alpha=+1) and mixture (m) connection (alpha=-1). The KL divergence D_KL(p||q) is the canonical divergence of the dually flat structure, with the generalized Pythagorean theorem providing a fundamental decomposition: for any m-projection of p onto a submanifold, D_KL(p||r) decomposes into the KL from p to the projection plus the KL from the projection to r. Natural gradient descent in parameter space becomes a geodesic flow in the manifold, with convergence rates determined by the manifold's curvature. Variational inference can be understood geometrically as alternating projections on dual spaces. These structures have profound implications for optimization, machine learning, and understanding why certain algorithms (EM, natural gradient) converge efficiently.

Explainer

Information geometry is the study of probability distributions as points on a Riemannian manifold, with the Fisher information matrix as the metric tensor. The basics — using the Fisher metric to measure distances between distributions, understanding geodesics — provide tools for statistical inference. Advanced information geometry goes deeper into the remarkable dually flat structure, a mathematical property unique to information-geometric spaces.

The Dual Connection Structure:

A standard Riemannian manifold has one natural connection (the Levi-Civita connection). A statistical manifold admits two dual connections: the e-connection (exponential) and the m-connection (mixture). The e-connection makes exponential families flat (zero curvature in natural parameter coordinates). The m-connection makes mixture families flat (zero curvature in mixture weight coordinates). The two connections are dual with respect to the Fisher metric, and KL divergence is the canonical divergence associated with this duality.

This duality is the source of many deep insights. For instance, the e-geodesic from p to q in natural parameters is a straight line in the natural parameter space — exponential families are "straight" in one coordinate system. Similarly, m-geodesics (mixture interpolations) are straight in mixture weights. Any distribution lies in both coordinate systems, and the geometry of the space is captured by how these two flatnesses interact.

Generalized Pythagorean Theorem:

In Euclidean geometry, if c is the orthogonal projection of a onto line b, then ||a||^2 = ||a-c||^2 + ||c||^2 (Pythagorean theorem). Information geometry admits a precise analog: for a submanifold S that is m-flat, and q the m-projection of p onto S,

D_KL(p||r) = D_KL(p||q) + D_KL(q||r) for all r in S.

This is the "generalized Pythagorean theorem" in the information-geometric sense. It states that KL divergence from p to any point in the submanifold separates into the error (p to q) and the distance within the submanifold (q to r). This has profound algorithmic implications: if you want to minimize D_KL(p||r) over r in S, first project p onto S (m-projection), and you have solved the optimization problem. No further search within S is needed — the projection is the global minimizer.

Natural Gradient Descent:

Gradient descent in Euclidean space moves in the direction of the negative gradient: theta_{t+1} = theta_t - eta * grad L(theta). This is coordinate-dependent: different parameterizations lead to different convergence rates. Natural gradient descent accounts for the Fisher metric:

theta_{t+1} = theta_t - eta * F(theta)^-1 * grad L(theta)

Geometrically, this is gradient descent in the statistical manifold where distances are measured via the Fisher metric. The update is coordinate-invariant — changing how you parameterize the probability family doesn't change the algorithm's behavior. Information-geometrically, natural gradient traces geodesics in the manifold, which are the "shortest paths" between distributions. This leads to faster convergence than Euclidean gradient descent, especially on exponential families.

The EM Algorithm:

The EM algorithm is a prime example of dually flat geometry in action. Given observed data X, unknown latents Z, and parameters theta, EM alternates:

1. E-step: Find q(Z) that minimizes D_KL(p(Z|X; theta)||q(Z)) — this is an m-projection.

2. M-step: Find theta that maximizes E_q[log p(X, Z; theta)] — this is an e-projection.

These projections are orthogonal in the dually flat space. By the generalized Pythagorean theorem, each step monotonically decreases the KL divergence between the true posterior and the model. This geometric understanding explains EM's remarkable property: it converges without explicit line search, without convexity assumptions, and without knowing the true posterior. The geometry guarantees it.

Variational Inference:

Variational inference approximates an intractable posterior p(Z|X) with a tractable variational family q(Z | phi) by minimizing D_KL(q||p). This is an e-projection (finding the closest distribution in the variational family). The dual m-projection would be to approximate with the mixtures of the exact posterior — intractable but conceptually clean. Mean-field variational inference further restricts q to factorized form, which is an additional m-projection. The algorithm alternates between updating the factorized form and each factor, which are alternating projections in the dually flat space.

Advanced information geometry transforms our understanding of statistical algorithms: they are not ad-hoc optimization procedures but geometric operations on manifolds. Natural gradient, EM, variational inference, and many others are revealed as projections, geodesic flows, or combinations thereof. This perspective enables new algorithm designs, convergence analysis, and deep insights into why these methods work. The framework continues to shape machine learning and Bayesian inference, providing both theoretical understanding and practical algorithmic guidance.