Questions: Optimization Algorithms: SGD, Adam, RMSprop

Question 1

A team trains a deep neural network with Adam and achieves fast training convergence but poor generalization to the test set. Switching to well-tuned SGD with momentum achieves similar training loss but significantly better test accuracy. What best explains this pattern?

• A. Adam is fundamentally broken for deep learning and should be replaced in all cases
• B. Adam's per-parameter adaptive learning rates can converge to sharp minima that interpolate the training data but generalize poorly; SGD with momentum may find flatter minima that generalize better
• C. Adam uses too much memory, corrupting gradient estimates and causing poor generalization
• D. SGD with momentum escapes local minima more easily because it lacks adaptive learning rates

Answer: (B)

Adam's adaptive scaling can cause it to converge to sharp minima in the loss landscape — regions with narrow, steep valleys that fit training data well but generalize poorly. SGD with momentum, despite being slower to converge, can find flatter minima associated with better generalization. This is a known empirical pattern and motivates AdamW (Adam with decoupled weight decay) and learning rate schedules as partial remedies.

Question 2

What specific problem does RMSprop solve that vanilla SGD with momentum does not?

• A. Oscillation caused by the learning rate being too high in all parameter directions simultaneously
• B. Different parameters having vastly different gradient magnitudes, so a single learning rate is too large for some and too small for others
• C. The inability of gradient descent to escape saddle points in the loss landscape
• D. The computational cost of computing full-batch gradients on large datasets

Answer: (B)

RMSprop tracks the running average of squared gradients per parameter. Parameters whose gradients are consistently large get their effective learning rate divided by a large number (dampened); parameters whose gradients are small get their learning rate scaled up. This per-parameter adaptation solves the problem of heterogeneous gradient scales that a single global learning rate cannot handle. Momentum addresses a different problem: smoothing noisy gradient direction.

Question 3

Adam's bias correction step is necessary because the first and second moment estimates are initialized at zero and would underestimate true gradient statistics early in training without it.

• T. True
• F. False

Answer: True

The exponential moving averages for m (first moment) and v (second moment) start at 0. Early in training, when relatively few gradient steps have been taken, these averages are biased toward zero regardless of actual gradient magnitudes. Bias correction divides each estimate by (1 − β^t), where t is the time step, scaling the estimates upward to remove this initialization bias and ensuring numerically stable step sizes from the first iteration.

Question 4

Because Adam adapts the learning rate individually for each parameter, the global learning rate hyperparameter α becomes irrelevant and does not need to be tuned.

• T. True
• F. False

Answer: False

Adam adapts learning rates per-parameter by dividing the global step by a parameter-specific scale factor, but the global learning rate α still multiplies the final update. It remains the most important hyperparameter to tune — the adaptive mechanism makes Adam less sensitive to α than vanilla SGD, but the right order of magnitude still matters significantly for convergence speed and final solution quality.

Question 5

Explain how Adam combines the properties of SGD with momentum and RMSprop, and what problem each component solves.

Model answer: SGD with momentum maintains a running average of past gradients (first moment), which smooths noisy gradient direction and accelerates movement through flat regions of the loss landscape. RMSprop tracks the running average of squared gradient magnitudes (second moment) to scale each parameter's learning rate by its recent gradient size — large gradients get dampened, small gradients get amplified. Adam combines both: the first moment provides directional inertia, the second moment provides per-parameter scaling, and bias correction ensures both estimates are accurate early in training when the running averages would otherwise be biased toward zero.

The combination makes Adam robust: it is fast through flat regions (momentum), adapts to curvature per parameter (adaptive scaling), and is stable from the first step (bias correction). Its default hyperparameters (β₁=0.9, β₂=0.999, ε=10⁻⁸) work acceptably across diverse problems, which is why it became the default optimizer for neural network training.