A topic in the Open Knowledge Graph — a free, open map of 15,290 topics and the order to learn them in.
Information bottleneck (IB) theory, introduced by Tishby et al. (2000), characterizes optimal representations by balancing two competing information-theoretic objectives: (1) maximizing information about the target Y that the representation T preserves (I(T; Y)), and (2) minimizing information about the input X that the representation retains (I(T; X)). The IB principle states that the optimal representation is the one that compresses the input (low I(T; X)) while maintaining predictive power (high I(T; Y)). This provides a unified framework for understanding generalization, dimensionality reduction, and representation learning in neural networks, revealing deep learning as implicit information compression.
The information bottleneck principle provides a beautiful information-theoretic lens on representation learning. Given data X with labels Y, the goal is to find a representation T such that T compresses X (is as independent of X as possible while remaining a deterministic function of X) while preserving predictive power for Y (T contains all task-relevant information about Y).
The formal setup uses the information bottleneck (IB) objective: I_IB(beta) = I(T; Y) - beta * I(T; X). For each value of beta, optimizing this defines an optimal representation on the Pareto frontier. Beta plays the role of Lagrange multiplier: beta = 0 recovers the original input (no compression), and large beta enforces aggressive compression. The Pareto frontier maps out the achievable trade-off: for every level of compression, what is the maximum mutual information with Y? Conversely, for every level of predictive power, what is the minimum necessary I(T; X)?
A remarkable insight from IB theory is the law of diminishing returns: beyond a certain compression level, you cannot increase I(T; Y) much further — the information about Y that is compressible away is precisely the noise and spurious correlations. This explains why simple models often generalize better than complex ones: they are forced to compress input into minimal representations, discarding the data-specific noise that would memorize.
IB theory also provides a window into neural network training. The empirical observation is that deep networks exhibit two training phases: early training focuses on fitting (I(T; Y) increases), while later training focuses on compression (I(T; X) decreases). This was termed the "fitting and forgetting" phases. The compression phase is where generalization emerges — by forgetting irrelevant details, the network prevents overfitting. This is automatic and implicit, requiring no explicit regularization, because the network's finite capacity and gradient descent dynamics naturally drive toward compression.
In practice, variational information bottleneck (VIB) replaces the intractable I(T; X) with a tractable variational bound, enabling optimization via gradient descent. The VIB objective becomes: I(T; Y) - beta * KL(q(t|x) || p(t)), where q is a learned encoder and p is a prior. This allows neural networks to optimize an information-theoretic objective directly. Applications include unsupervised representation learning, semi-supervised learning, and domain adaptation.
The limitations of IB are worth noting: it assumes a clear distinction between information about X versus Y (which may blur in practice), assumes you can compute or bound mutual information accurately (computationally hard for high-dimensional X), and the optimal representation under IB may not align with downstream task performance. Additionally, IB theory applies most cleanly to deterministic mappings; for stochastic representations, the analysis is more complex. Despite these limitations, IB provides invaluable intuition: good representations compress input while preserving target information, a principle now central to modern representation learning.