Questions: Linear Regression in Machine Learning

Question 1

Which statement best describes what the ordinary least squares (OLS) objective minimizes?

• A. The sum of absolute differences between predicted and actual values
• B. The sum of squared differences between predicted and actual values
• C. The maximum single prediction error across all training examples
• D. The difference between the largest and smallest predicted values

Answer: (B)

OLS minimizes the sum of squared residuals (SSR = Σ(yᵢ - ŷᵢ)²). Squaring errors penalizes large errors disproportionately and makes the objective differentiable, enabling the closed-form normal equation solution β = (XᵀX)⁻¹Xᵀy and gradient-based optimization. Minimizing absolute differences (L1) leads to median regression, not OLS.

Question 2

Linear regression can mainly model truly linear relationships between input features and the target variable.

• T. True
• F. False

Answer: False

Linear regression is linear in its parameters, not necessarily in the input features. By adding polynomial features (x², x³) or interaction terms (x₁·x₂) to the feature matrix, linear regression can model curved and complex relationships. The model remains 'linear' because ŷ is a linear combination of the (now transformed) features, and all OLS theory still applies.

Question 3

When would you choose gradient descent over the normal equations to fit a linear regression model?

Model answer: When the number of features (p) is very large, because the normal equations require inverting a p×p matrix, which is O(p³) in time and O(p²) in memory. Gradient descent scales much better with large feature sets and is the standard approach when p can be thousands or millions.

The normal equation β = (XᵀX)⁻¹Xᵀy requires forming and inverting the p×p matrix XᵀX. For small datasets with few features this is fast and exact. But for high-dimensional problems, matrix inversion is computationally prohibitive. Gradient descent iteratively updates parameters using the loss gradient, costing only O(n·p) per step, making it far more scalable.