Questions: Batch Normalization

Question 1

A model with batch normalization performs well during training but gives poor results at deployment. Training used batch size 64; deployment processes one image at a time. What is the most likely cause?

• A. Batch normalization degrades all models at deployment regardless of batch size
• B. The model was not switched to evaluation mode, so it uses noisy single-sample batch statistics instead of the stored running averages
• C. Batch size 64 was too large, causing overfitting in the normalization layers
• D. The learnable parameters γ and β are discarded when a model is deployed

Answer: (B)

This is the classic batch normalization deployment bug. During training, BN computes mean and variance from the minibatch. At inference with batch size 1, the 'batch' statistics are just that single sample — the mean is the sample value itself, and the variance is zero, producing nonsensical normalization. The fix is to switch to eval mode, which uses the running averages of population statistics accumulated during training. Forgetting to call model.eval() (PyTorch) or setting training=False (TensorFlow) is a common and silent bug.

Question 2

A researcher removes the learnable scale (γ) and shift (β) parameters from all batch normalization layers, leaving only the normalization step. What is the likely consequence?

• A. No effect; γ and β are redundant because weights in the next layer can compensate
• B. The network loses the ability to represent identity transforms or unnormalized distributions, constraining what functions it can learn
• C. Training accelerates because there are fewer parameters to optimize
• D. Regularization increases because normalization is applied more strictly

Answer: (B)

Without γ and β, every layer's output is permanently locked to zero mean and unit variance. This severely restricts expressiveness — for example, a sigmoid activation layer that works best with larger-magnitude inputs cannot receive them. The γ and β parameters are precisely what give BN zero representational cost: the network can recover any scale and offset if needed by learning γ = original std and β = original mean. Removing them imposes a hard constraint that limits function space.

Question 3

Batch normalization cannot reduce a network's representational capacity because the learnable parameters γ and β allow the network to recover any unnormalized distribution if gradient descent finds it useful.

• T. True
• F. False

Answer: True

This is a key design property. If the network learns γ = σ (the original standard deviation) and β = μ (the original mean), the output y = γx̂ + β recovers exactly the unnormalized pre-BN values. BN never forces normalization — it gives the network the option to normalize. This means BN adds flexibility without restricting expressiveness, which is why it can be inserted into networks without risking degradation in principle.

Question 4

During training, batch normalization uses population statistics computed over the entire training dataset to normalize each layer's inputs.

• T. True
• F. False

Answer: False

During training, BN uses minibatch statistics (mean and variance computed over the current batch only). Population-level statistics (running averages accumulated via exponential moving average across minibatches) are stored during training but only used at inference time. Using true population statistics during training would require a pass over the entire dataset at every step, which is computationally infeasible. The minibatch statistics also introduce beneficial noise, acting as a mild regularizer.

Question 5

Why does batch normalization behave differently at training time versus inference time, and what bug does this difference commonly cause?

Model answer: During training, BN normalizes using the current minibatch's mean and variance, which are noisy estimates of the true population statistics. At inference, there is often no meaningful batch (or only a single sample), so BN switches to using stored running averages of population statistics accumulated during training. The common bug is forgetting to switch the model to evaluation mode before inference, causing the model to use single-sample batch statistics that are meaningless — often producing outputs that look normal during training but fail completely at deployment.

This train/test discrepancy is a footgun in every major deep learning framework. In PyTorch, model.train() and model.eval() toggle this behavior; in Keras, the training= flag controls it. It is especially insidious because the model may produce outputs that look plausible (no crash, no NaN) while being systematically wrong, making it hard to debug without knowing to look for this issue.