Questions: Activation Functions in Neural Networks

Question 1

A researcher builds a 10-layer neural network using only linear transformations between layers — no activation functions. What is the effective expressive power of this network compared to a single-layer linear model?

• A. It is 10 times more powerful because each layer adds an independent linear transformation
• B. It is equivalent to a single-layer linear transformation
• C. It can approximate nonlinear functions through the interaction of many linear layers
• D. It has exponentially more capacity because of its large number of parameters

Answer: (B)

A composition of linear transformations is always itself a linear transformation — the product of any number of matrices is still a matrix. No matter how many linear layers are stacked without activation functions, the entire network collapses to a single matrix multiplication, capable only of linear mappings. Activation functions are not optional decorations; they are the entire source of a deep network's ability to model nonlinear functions. Without them, depth adds zero expressive power.

Question 2

A multi-class classifier with 5 output classes uses ReLU as its output layer activation. What is the primary problem with this design?

• A. ReLU blocks gradient flow backward through the output layer
• B. ReLU outputs can be any non-negative number and will not sum to 1, making them invalid as class probabilities
• C. ReLU is too computationally expensive for output layers
• D. ReLU causes the dying ReLU problem, zeroing out all output neurons

Answer: (B)

Output activation choices are driven by the task, not by gradient-flow considerations. For multi-class classification, outputs must be non-negative and sum to 1 — exactly what softmax provides. ReLU can output any non-negative value with no normalization constraint, so the outputs cannot be interpreted as probabilities and cannot be used with cross-entropy loss correctly. This is one of the most common neural network design errors: it produces no error message but silently makes training and interpretation incorrect.

Question 3

Stacking more linear layers without activation functions allows a neural network to model increasingly complex, nonlinear decision boundaries.

• T. True
• F. False

Answer: False

No matter how many linear layers are stacked, the composition remains equivalent to a single linear transformation. Linear models can only produce hyperplane decision boundaries. Arbitrary and nonlinear decision boundaries — XOR patterns, spirals, concentric rings — require nonlinearity. Only nonlinear activation functions between layers create the representational capacity needed for complex patterns. More linear layers change the effective weight matrix but do not add any expressive power beyond the linear function class.

Question 4

ReLU avoids the vanishing gradient problem for positive inputs because its derivative is exactly 1, allowing gradient signals to flow backward through layers without shrinking.

• T. True
• F. False

Answer: True

For x > 0, ReLU computes f(x) = x, so f'(x) = 1 exactly. During backpropagation, the chain rule multiplies gradients through each layer; a factor of exactly 1 at a ReLU unit means no attenuation of the gradient signal at that unit. In contrast, sigmoid and tanh derivatives are always between 0 and 1, approaching 0 for large inputs, causing gradients to shrink exponentially as they propagate backward through many layers. ReLU's unit derivative for positive inputs is the direct reason it enabled training of much deeper networks.

Question 5

Why is a nonlinear activation function necessary between layers of a neural network? What happens if all activations are removed?

Model answer: Without nonlinear activations, any composition of layers reduces to a single linear transformation, because the product of any number of matrices is still a matrix. The network could only learn linear functions of its input — straight-line decision boundaries in classification, linear relationships in regression. Nonlinear activations break this collapse: after a nonlinear transformation, subsequent layers operate on a nonlinearly transformed space, making the overall mapping a composition of nonlinear functions with far greater expressive power.

The proof is elementary linear algebra: W₂(W₁x) = (W₂W₁)x, and W₂W₁ is just another matrix. So 10 linear layers = one linear layer, always. Activation functions interrupt this collapsing by introducing operations that cannot be expressed as matrix multiplication. After a nonlinear activation, the subsequent weight matrix no longer simply combines features linearly — it operates on a nonlinearly transformed representation. This is why activation functions are the architectural element that makes neural networks universal function approximators rather than just expensive linear models.