Backpropagation Algorithm

Core Idea

Backpropagation computes gradients efficiently via the chain rule in two phases: forward pass computes activations, backward pass propagates error signals layer-by-layer. Complexity is O(n) for n parameters, enabling large-scale neural network training.

How It's Best Learned

Implement backpropagation from scratch, deriving gradients by hand and verifying with numerical gradients.

Common Misconceptions

Backpropagation applies to any differentiable computation, not just neural networks. Vanishing/exploding gradients require normalization techniques.

Explainer

You already know from neural networks that a model makes predictions by passing inputs through layers of weighted connections and activation functions. Training the model means adjusting those weights so the predictions improve — but with potentially millions of weights across dozens of layers, how do you know which direction to adjust each one? That is the problem backpropagation solves.

The algorithm runs in two phases. In the forward pass, inputs flow through the network layer by layer, and you store the intermediate activations at each layer. At the end, you compute the loss — a scalar measuring how wrong the prediction was. In the backward pass, you work in reverse: starting from the loss, you use the chain rule to compute the gradient of the loss with respect to each weight. The chain rule says that ∂L/∂w = (∂L/∂a) · (∂a/∂w), where a is the activation that w influences. Because each layer feeds into the next, the error signals propagate back through the network by repeated application of the chain rule. Crucially, the stored activations from the forward pass are needed here — they are part of the local gradient at each layer.

The efficiency of backpropagation comes from the order of computation. A naïve approach to computing ∂L/∂w for every weight would recompute many of the same partial derivatives repeatedly. Backpropagation avoids this by computing error signals once per layer in a single backward sweep, reusing results as they propagate. This gives O(n) time for n parameters — the same cost as a single forward pass — which is why training large networks is computationally feasible.

A common misconception is that backpropagation is specific to neural networks. It is not — it is reverse-mode automatic differentiation applied to a particular kind of computation graph. Any software that computes a differentiable function (not just neural nets) can use the same technique. Modern deep learning frameworks like PyTorch build a dynamic computation graph during the forward pass and then traverse it in reverse to compute all gradients automatically.

The main practical hazard is the vanishing gradient problem: when gradients pass through many layers, they are repeatedly multiplied by local derivatives. If those derivatives are consistently small (as with saturating activations like sigmoid, whose derivative peaks at 0.25), the product shrinks exponentially. Early layers receive nearly zero gradient and fail to learn. The opposite — exploding gradients from derivatives greater than 1 — causes numerical instability. Remedies include ReLU activations (gradient is 1 for positive inputs), batch normalization (keeps activations in healthy ranges), and gradient clipping for recurrent networks.