Questions: Principal Component Analysis

Question 1

You run PCA on a 100-feature dataset. The first 3 principal components explain 82% of total variance. A colleague says 'PCA found the 3 most important features.' What is wrong with this statement?

• A. Nothing is wrong — PCA selects the 3 features with the highest variance
• B. PCA found 3 new axes that are linear combinations of all 100 original features, not a subset of 3 features
• C. PCA selects features by correlation, not by variance, so 82% refers to correlation explained
• D. The colleague is right, except the number should be higher — PCA typically retains at least 10 features

Answer: (B)

Principal components are new directions in the original feature space — linear combinations of all 100 features — not selections from them. The first PC is the direction of maximum variance across all features simultaneously. Feature selection (keeping original features) and feature extraction (creating new axes via PCA) are fundamentally different operations. After PCA, each component may load on all 100 original features with different weights, so 'the most important features' is a category error.

Question 2

A dataset's true structure lies on a two-dimensional Swiss roll (a curved, spiral surface) embedded in three-dimensional space. You apply PCA to reduce to 2 dimensions. What will most likely happen?

• A. PCA will perfectly recover the 2D structure, since the data truly lives in 2 dimensions
• B. PCA will fail to capture the intrinsic structure because it can only find flat (linear) subspaces, and no flat plane efficiently aligns with a curved manifold
• C. PCA will fail because it cannot handle 3D data — it only works on high-dimensional datasets
• D. PCA will succeed if you first normalize the features, since normalization linearizes the structure

Answer: (B)

PCA finds the best flat (affine) subspace to project onto, as determined by the covariance matrix eigendecomposition. A Swiss roll has meaningful geometric structure, but it is curved — no 2D plane captures both the separation across the roll and the variation along it. The result is that PCA will project points that are far apart on the manifold (but happen to be close in 3D Euclidean space) onto the same location. Non-linear methods like UMAP or t-SNE, which can 'unfold' curved manifolds, are needed for this type of data.

Question 3

The first principal component is the eigenvector of the covariance matrix corresponding to the largest eigenvalue, and it points in the direction of maximum variance in the data.

• T. True
• F. False

Answer: True

This is the exact definition. PCA solves for the directions (eigenvectors) along which the projected data has the most spread (variance), ranked by their eigenvalues. The covariance matrix Σ is symmetric positive semi-definite, so it has an orthogonal eigendecomposition. The eigenvector with the largest eigenvalue λ₁ defines the direction of maximum variance — the first PC. Successive PCs are orthogonal to all previous ones and capture decreasing amounts of variance. The eigenvalue itself quantifies the variance along that direction.

Question 4

PCA removes noise from a dataset by keeping mainly the principal components with large eigenvalues and discarding the rest.

• T. True
• F. False

Answer: False

PCA is agnostic about what constitutes 'signal' versus 'noise' — it retains directions of high variance and discards directions of low variance. If noise happens to be high-variance and signal low-variance, PCA will do the opposite of what is intended. True noise reduction requires either domain knowledge about which directions are meaningful, or methods that explicitly model noise (like probabilistic PCA or factor analysis). PCA is a dimensionality-reduction technique that preserves variance, not a denoising technique in the general sense.

Question 5

Why must data be centered (mean-subtracted) before applying PCA, and what artifact arises if this step is skipped?

Model answer: PCA finds the directions of maximum variance in the data. If the data is not centered, the first principal component will point toward the mean of the data (the direction with the largest squared distances from the origin), not the direction of greatest spread. This is because variance is defined relative to the mean; without centering, the 'variance' PCA maximizes is actually the second moment about the origin, which conflates the data's spread with its offset from zero. After centering, every PC captures genuine spread in the data rather than a combination of spread and mean direction.

The covariance matrix Σ = (1/n)XᵀX requires centered data (X having zero mean columns). Without centering, the computed matrix is the second-moment matrix (1/n)XᵀX, whose top eigenvector points toward the mean. For data far from the origin (e.g., heights in centimeters, which are all ~170), this artifact dominates all others. The practical fix is always to subtract the column means before computing the covariance matrix.