Questions: F-Statistic for Overall Model Significance

Question 1

A researcher starts with a regression of wages on 2 relevant predictors (education, experience) and then adds 20 noise variables with no true relationship to wages. Compared to the original 2-variable model, the 22-variable model is MOST likely to have:

• A. A much higher F-statistic, because more regressors explain more variation
• B. A similar or lower F-statistic, because the degrees-of-freedom penalty punishes irrelevant predictors
• C. The same F-statistic, since F is determined only by sample size
• D. A higher F-statistic and higher R², confirming the larger model is better

Answer: (B)

The F-statistic divides explained variation by its degrees of freedom (k) and unexplained variation by (n-k-1). Adding 20 noise variables increases ESS only trivially (random predictors capture a little variation by chance) but dramatically increases k, shrinking ESS/k. Meanwhile, RSS/(n-k-1) also changes as n-k-1 falls. The net effect: irrelevant predictors reduce F even while R² mechanically increases. This is exactly why raw R² is a misleading goodness-of-fit measure and the F-statistic's degrees-of-freedom adjustment matters.

Question 2

A regression of quarterly sales on 5 variables yields F = 21.4 (p < 0.001). What does this tell you?

• A. All five variables individually have statistically significant effects on sales
• B. The regression has identified a causal relationship between the predictors and sales
• C. The five predictors collectively explain significantly more variation in sales than a model with no predictors
• D. The model has high R² and therefore strong out-of-sample predictive accuracy

Answer: (C)

The overall F-test asks one question: do the regressors collectively explain anything, relative to a null model with no predictors? A significant F answers 'yes' to that narrow question. It does NOT indicate which individual regressors are significant (some may be useless), does NOT imply causation (correlated variables can produce enormous F statistics with zero causal content), and says nothing about out-of-sample prediction or R². Each of those requires separate analysis.

Question 3

A model with 10 predictors and a modest R² could have a lower F-statistic than a model with 3 predictors and the same R², because the F-statistic adjusts for the number of regressors.

• T. True
• F. False

Answer: True

Yes — with the same R² (same ESS/TSS ratio), a 10-predictor model has a larger k in the numerator denominator ESS/k, reducing the numerator of F. The 3-predictor model concentrates the same explanatory power across fewer degrees of freedom, yielding a higher F. This is why adding irrelevant regressors is detectable through F even when R² is unchanged — the adjustment for degrees of freedom is precisely designed to penalize model bloat.

Question 4

A statistically significant overall F-statistic confirms that the independent variables in a regression model have a causal effect on the dependent variable.

• T. True
• F. False

Answer: False

F tests whether predictors collectively explain variation — this is a statement about statistical association, not causation. A house price regression using zip code and school district ratings will produce an enormous F-statistic, but giving a house a better zip code doesn't cause its price to rise. Omitted variable bias, reverse causation, and spurious correlations can all produce high F-statistics with no causal content. Causal identification requires design features (randomization, instruments, discontinuities) that the F-test cannot provide.

Question 5

Explain why the F-statistic formula divides ESS and RSS by their respective degrees of freedom (k and n-k-1) rather than comparing the raw sums directly.

Model answer: Dividing by degrees of freedom converts sums of squares into averages (mean squares), which are comparable across models with different numbers of predictors or sample sizes. ESS mechanically increases as you add regressors — even useless ones absorb some random variation — so comparing raw ESS to RSS would always favor larger models. Dividing ESS by k (the number of regressors) and RSS by (n-k-1) accounts for how many 'free parameters' each piece used. Under the null hypothesis, both ESS/k and RSS/(n-k-1) estimate the error variance, so their ratio follows an F-distribution — enabling valid statistical testing.

The degrees-of-freedom adjustment is what makes the F-statistic a valid test statistic rather than just a fit measure. Without it, you could always inflate F by adding more variables. The adjustment enforces a penalty for model complexity, ensuring the test remains calibrated under the null.