Questions: Orthogonal Projections and Least Squares Approximation

Question 1

A student is told the least-squares solution x* minimizes ||Ax − b||². She concludes that Ax* must equal b. What is wrong with this reasoning?

• A. There is no error — minimizing the squared error means the minimum value is zero
• B. Ax* is the orthogonal projection of b onto the column space of A, which equals b only if b already lies in that column space
• C. She confused the row space with the column space of A
• D. Minimizing ||Ax − b||² requires calculus, not linear algebra

Answer: (B)

The least-squares problem arises precisely when Ax = b has no exact solution — meaning b lies outside the column space of A. The minimizer x* gives Ax* = proj_{col(A)}(b), the closest point to b in col(A). This point is not equal to b unless b happens to be in col(A) already. The minimum value of ||Ax − b||² is the squared distance from b to its projection, not zero.

Question 2

When does the least-squares problem Ax = b have a unique solution x* = (AᵀA)⁻¹Aᵀb?

• A. When b lies in the column space of A
• B. When A is a square matrix
• C. When A has linearly independent columns, ensuring AᵀA is invertible
• D. When the rows of A are orthonormal

Answer: (C)

Linear independence of the columns of A is precisely the condition that makes AᵀA invertible, giving a unique minimizer x*. Option A describes when the exact solution exists (Ax = b is consistent), not when least squares is unique. Option B is insufficient — a square singular matrix has linearly dependent columns and AᵀA is still not invertible.

Question 3

The normal equations AᵀAx = Aᵀb arise because we want the residual vector b − Ax* to be perpendicular to the column space of A.

• T. True
• F. False

Answer: True

This is the geometric heart of least squares. The closest point in a subspace to b is the point where the error vector is orthogonal to the subspace. 'Perpendicular to every column of A' means Aᵀ(b − Ax*) = 0, which rearranges directly to AᵀAx* = Aᵀb — the normal equations. The word 'normal' in 'normal equations' refers to this perpendicularity condition.

Question 4

The projection matrix P = A(AᵀA)⁻¹Aᵀ satisfies P² = P because applying the projection twice is equivalent to applying it once.

• T. True
• F. False

Answer: True

Once a vector is projected onto a subspace, the result already lies in that subspace. Projecting it again does not move it. Algebraically: P² = A(AᵀA)⁻¹Aᵀ · A(AᵀA)⁻¹Aᵀ = A(AᵀA)⁻¹(AᵀA)(AᵀA)⁻¹Aᵀ = A(AᵀA)⁻¹Aᵀ = P. This idempotent property P² = P, together with symmetry Pᵀ = P, completely characterizes orthogonal projection matrices.

Question 5

Explain geometrically why the least-squares solution minimizes ||Ax − b||².

Model answer: The set of all vectors of the form Ax (as x varies over all vectors) is the column space of A. The problem asks: which point in col(A) is closest to b? The closest point to b in any subspace is its orthogonal projection onto that subspace — the unique point where the error vector b − Ax* is perpendicular to the entire subspace. Minimizing the squared distance ||Ax − b||² is equivalent to finding this projection. The normal equations encode the perpendicularity condition: Aᵀ(b − Ax*) = 0.

This geometric view unifies the algebra. The normal equations are not an arbitrary algebraic trick — they encode exactly one geometric condition: the error must be orthogonal to the subspace we're projecting onto. This is why least squares appears everywhere from linear regression to signal processing: in each case, you want the best approximation to something you can't represent exactly, and orthogonal projection gives the unique closest point.