Questions: Self-Attention and Multi-Head Attention

Question 1

Suppose you remove the positional encodings from a Transformer model and train it on a sentence classification task. What is the most fundamental consequence?

• A. The model can no longer compute attention scores because the Q, K, V projections depend on position
• B. The model treats every permutation of the same words as identical, losing all sensitivity to word order
• C. Multi-head attention stops working because heads require positional information to specialize
• D. The model attends only to adjacent tokens, losing the ability to model long-range dependencies

Answer: (B)

Self-attention is permutation-invariant: the computation treats every position symmetrically, so shuffling the input tokens produces the same output (just rearranged). The sentences 'the cat chased the dog' and 'the dog chased the cat' would produce identical representations after self-attention without positional encodings. Positional encodings inject sequence-order information into the token embeddings *before* attention operates, giving the mechanism something to attend to that encodes position. This is why positional encoding is not optional in Transformers — it is a necessary fix for a fundamental property of the attention mechanism itself.

Question 2

Why are the raw dot-product attention scores divided by √dₖ before applying the softmax in self-attention?

• A. To normalize scores to the range [0, 1] before softmax can be applied
• B. To prevent very large dot products from pushing the softmax into a near-zero-gradient region, which would slow training
• C. To ensure that the output of each attention head has the same variance as the input
• D. To make attention scores independent of the model dimension, allowing the same architecture to work at any scale

Answer: (B)

When dₖ (the key/query dimension) is large, random dot products have variance proportional to dₖ. Large-magnitude dot products push the softmax toward a near-one-hot distribution — one position gets almost all the weight and the rest get almost nothing. In this regime the softmax gradient is nearly zero, making learning very slow. Dividing by √dₖ keeps the dot products in a range where the softmax gradient remains healthy. This scaling is subtle but critical — without it, multi-head attention with high-dimensional projections trains poorly.

Question 3

Self-attention inherently captures the order of tokens in a sequence, which is why Transformers can model word order without needing positional encodings.

• T. True
• F. False

Answer: False

This is the central counterintuitive property of self-attention. The mechanism computes dot products between every pair of positions and takes weighted sums — operations that are completely symmetric with respect to position. If you feed in 'A B C' versus 'C B A,' the self-attention operation (without positional encodings) produces the same output values, just reindexed. Word order information must be injected externally via positional encodings added to the input embeddings. This is a fundamental architectural constraint, not an implementation detail.

Question 4

In multi-head attention, different attention heads can specialize in capturing different types of relationships — such as syntactic dependencies and coreference — without any explicit supervision about which head should learn which pattern.

• T. True
• F. False

Answer: True

This specialization emerges naturally from training. Because each head has its own learned Q, K, V projection matrices mapping into a lower-dimensional subspace, different heads learn to attend to different aspects of the input that are useful for the training objective. Empirical studies of trained Transformers have found heads that attend to adjacent words, heads that resolve coreference (e.g., 'it' → 'cat'), and heads that track syntactic roles — even though none of this was prescribed during architecture design. The parallel structure makes this division of labor both possible and computationally efficient.

Question 5

Why is self-attention described as having O(n²) computational cost with respect to sequence length, and what does this imply for very long sequences?

Model answer: Self-attention requires computing a score between every pair of positions: position 1 attends to positions 1 through n, position 2 attends to positions 1 through n, and so on. With n positions, this yields n × n = n² score computations. Both the memory required to store the attention matrix and the compute required to fill it scale quadratically with n. For short sequences (sentences of ~512 tokens), this is manageable. For very long sequences (documents, high-resolution images, genomic sequences), it becomes prohibitively expensive — motivating architectures like sparse attention, linear attention, and hierarchical approaches that reduce this cost at the expense of some modeling flexibility.

This quadratic cost is the main limitation of standard self-attention. Unlike RNNs, which process tokens sequentially with O(n) compute but struggle to propagate information across long distances, self-attention directly connects all pairs but pays a quadratic price. This is the fundamental trade-off that drives ongoing research into efficient attention variants.