A topic in the Open Knowledge Graph — a free, open map of 15,290 topics and the order to learn them in.
During backpropagation, gradients multiply across layers; with saturating activation functions like sigmoid, gradients near zero cause deep layers to learn very slowly (vanishing gradients) or gradients can grow uncontrollably (exploding gradients). Solutions include careful weight initialization (Xavier, He initialization), gradient clipping, non-saturating activations (ReLU), and architectural innovations like skip connections and gating mechanisms.
Train deep networks with sigmoid activations and observe layer-wise gradient magnitudes, then compare with ReLU networks to see how activation choice affects gradient flow.
From backpropagation, you know that training a neural network means computing the gradient of the loss with respect to every weight, then nudging each weight in the direction that reduces the loss. The chain rule makes this possible: the gradient at any layer is the product of local gradients along the path from the output back to that layer. The vanishing gradient problem is what happens when that product shrinks to near zero, effectively cutting off learning for the earlier layers of a deep network.
To see why this happens, consider a network with sigmoid activations. The sigmoid function squashes its input to the range (0, 1), and its derivative peaks at 0.25 and drops toward zero for large or small inputs. During backpropagation, the gradient at each layer is multiplied by the local sigmoid derivative. If that derivative is 0.2 at each layer, then after 10 layers the gradient has been multiplied by 0.210 ≈ 0.0000001. The gradient reaching the first layer is astronomically smaller than the gradient at the last layer. Those early layers — which learn fundamental, low-level features — barely update their weights at all. The network appears to train (the last few layers adjust), but the deep layers remain near their random initialization, and the network never learns the hierarchical representations that make deep learning powerful.
The mirror problem is exploding gradients: if local gradients are consistently greater than 1, the product grows exponentially, causing weight updates so large that training becomes numerically unstable (weights oscillate wildly or overflow to infinity). Vanishing and exploding gradients are two sides of the same coin — the instability inherent in multiplying many factors together.
The solutions attack the problem from multiple angles. ReLU (Rectified Linear Unit) activations have a derivative of exactly 1 for positive inputs, so gradients pass through without shrinking. Careful initialization (Xavier for tanh/sigmoid, He for ReLU) sets initial weights so that the variance of activations and gradients stays stable across layers. Gradient clipping caps the gradient norm to prevent explosions. Most fundamentally, skip connections (as in ResNets) add shortcut paths that let gradients flow directly to earlier layers, bypassing the multiplicative chain entirely. These architectural innovations are what made training networks with hundreds of layers feasible — not more data or compute, but solving the gradient flow problem that had bottlenecked deep learning for decades.