Gradient Descent and Optimization

Core Idea

Gradient descent iteratively moves toward minima by stepping in the negative gradient direction. Step size (learning rate) controls convergence: too small is slow, too large diverges. Momentum and adaptive methods improve convergence.

How It's Best Learned

Implement vanilla gradient descent on a convex function, visualizing iterations and comparing with Adam.

Common Misconceptions

Gradient descent finds global minima only for convex functions; non-convex problems may converge to local minima. Smaller learning rates are not always better.

Explainer

The core idea of gradient descent is simple: if you know the slope of a function at your current location, you can decrease the function's value by stepping in the downhill direction. For a scalar parameter θ minimizing loss L, the update is θ ← θ − η · (∂L/∂θ), where η (eta) is the learning rate. In higher dimensions, the gradient ∇L is a vector pointing in the direction of steepest ascent — so the negative gradient points downhill, and you step that way.

The learning rate η controls how far you step each iteration. Setting it too small means you make tiny, cautious moves and convergence takes an enormous number of steps — particularly painful in high-dimensional spaces with flat regions. Setting it too large causes you to overshoot: instead of landing near the minimum at the bottom of a valley, you jump past it, climb the far wall, and bounce back. The learning rate is often the most critical hyperparameter to tune. Common practice is to start with a moderate value (e.g., 0.01) and use a learning rate schedule that decays it over time, or to use an adaptive optimizer.

Vanilla gradient descent computes the gradient using the entire dataset before each update. For modern deep learning with millions of training examples, this is prohibitively expensive. Stochastic gradient descent (SGD) instead estimates the gradient from a single randomly chosen example (or a small mini-batch of ~32–256 examples), making updates far more frequently with noisier estimates. Mini-batch SGD combines the best of both: the noise helps escape sharp local minima, the averaging reduces variance enough to take stable steps, and modern hardware (GPUs) parallelizes mini-batch computation efficiently.

For non-convex loss surfaces — which is essentially every interesting problem in deep learning — gradient descent has no guarantee of finding the global minimum. It follows the local slope and stops wherever it cannot descend further. Empirically, this turns out to be far less of a problem than it sounds, because in high-dimensional parameter spaces most local minima and saddle points have similar loss values to the global minimum. The geometry of high-dimensional loss landscapes is fundamentally different from low-dimensional intuition.

More advanced optimizers build on the gradient descent idea by incorporating momentum and adaptive learning rates. Momentum accumulates a running average of past gradients, effectively giving the optimizer "inertia" that smooths oscillations and accelerates progress in consistent directions. Adam (adaptive moment estimation) maintains per-parameter estimates of both the first moment (mean gradient) and second moment (uncentered variance), using them to normalize the step size for each parameter independently. These methods generally converge faster and more robustly than vanilla SGD on deep learning problems, though SGD with momentum remains competitive for well-tuned training regimes.