A topic in the Open Knowledge Graph — a free, open map of 15,290 topics and the order to learn them in.
Deep learning theory seeks to explain three mysteries of modern neural networks: why depth helps (depth-separation results show deep networks can represent functions exponentially more efficiently than shallow ones), why optimization succeeds (over-parameterized networks have benign loss landscapes where SGD finds global minima), and why generalization occurs despite over-parameterization (implicit regularization, where SGD's dynamics bias toward simple solutions). The neural tangent kernel (NTK) theory connects infinitely wide networks to kernel methods, providing one tractable theoretical framework, though it does not fully capture the feature-learning capabilities of finite-width networks.
Deep learning theory confronts the three biggest gaps between classical learning theory and modern practice. Classical theory predicts that over-parameterized models should overfit, non-convex optimization should get stuck in local minima, and complex models should need proportionally more data. Deep networks violate all three predictions and work spectacularly well. Understanding why is the central project of modern learning theory.
The first mystery is depth separation: why are deep networks more powerful than shallow ones, beyond the universal approximation guarantee? Depth-separation results provide a crisp answer: there exist functions that deep networks with polynomial parameters can represent exactly, but that shallow networks need exponentially many parameters to approximate. The key mechanism is composition — each layer applies a nonlinear transformation that interacts with previous layers, creating an exponentially growing space of representable functions as depth increases. For hierarchical functions (where the output is computed by composing simpler operations), deep networks match the hierarchy naturally, while shallow networks must "flatten" the computation at enormous cost.
The second mystery is optimization: the loss landscape of a deep network is non-convex, with potentially many local minima, saddle points, and plateaus. Yet SGD reliably finds solutions with very low training loss. Over-parameterization theory provides a partial answer: when the network has many more parameters than training examples, the loss landscape becomes "benign" — local minima are also global minima (or very close to them), and saddle points are easily escaped. The NTK theory formalizes this for infinitely wide networks: in the infinite-width limit, training with gradient descent becomes equivalent to kernel regression with a fixed kernel, making the optimization convex. For finite-width networks, the picture is more complex, but the empirical observation is robust: wider and deeper networks are easier to optimize, not harder.
The third mystery is generalization: networks with millions of parameters, trained to zero training error on thousands of examples, should overfit according to classical bounds — yet they achieve excellent test performance. The explanation involves implicit regularization (SGD selects among the many interpolating solutions for ones with low complexity), norm-based generalization bounds (which depend on weight magnitudes and margins rather than parameter counts), and the structure of real-world data (which lies on low-dimensional manifolds that the network's effective complexity adapts to). The Zhang et al. (2017) experiment — showing that the same network architecture can memorize random labels but generalize on real labels — proved definitively that generalization depends on the interaction between model, algorithm, and data, not on the model alone.
These three threads — expressiveness, optimization, and generalization — are deeply interconnected, and a unified theory that explains all three simultaneously remains the grand challenge of deep learning theory. The neural tangent kernel provides one unifying framework (at the cost of ignoring feature learning), PAC-Bayes bounds provide another (at the cost of loose constants), and the study of implicit regularization promises to bridge optimization and generalization. The field is rapidly evolving, with new results regularly reshaping the theoretical landscape.