Questions: Prediction Intervals and Out-of-Sample Forecasting

Question 1

A data scientist fits a salary regression on 50,000 observations and wants to predict the salary of a specific new employee with 5 years of experience. She expects the prediction interval to be extremely narrow because the sample is huge. She is:

• A. Wrong — the prediction interval includes irreducible error variance (the individual error term ε) that never shrinks with sample size; a large n only tightens the confidence interval for the mean, not the interval for an individual prediction
• B. Correct — with 50,000 observations the regression line is estimated so precisely that a prediction interval becomes indistinguishable from a point estimate
• C. Wrong — prediction intervals actually widen with sample size because more data reveals more variation in the outcome
• D. Correct — prediction intervals and confidence intervals for the mean both converge to zero width as n grows large

Answer: (A)

The prediction interval variance is σ²[1 + X_new′(X′X)⁻¹X_new]. As n → ∞, the second term (X_new′(X′X)⁻¹X_new) shrinks to zero, but the '1' remains — it represents the irreducible error variance of the new individual observation, which exists even if you knew the true regression line perfectly. No sample size eliminates this fundamental unpredictability of individual outcomes. The confidence interval for the mean omits this '1', so it does shrink toward a point with large n. The two intervals measure fundamentally different quantities.

Question 2

Which of the following correctly distinguishes what a confidence interval and a prediction interval estimate in regression?

• A. A confidence interval estimates where the true mean of Y lies for all units with a given X value; a prediction interval estimates where a specific new individual observation will fall
• B. A confidence interval is always wider because it must account for both parameter uncertainty and the individual error term
• C. A prediction interval is a special type of confidence interval used when the model's R² is below 0.5
• D. The two intervals are numerically equivalent whenever the sample size is large enough for the central limit theorem to apply

Answer: (A)

The conceptual distinction is: confidence intervals answer 'what is the average outcome for this type of unit?' while prediction intervals answer 'what will this specific unit's outcome be?' The prediction interval must be wider because it adds the irreducible individual error term to the parameter uncertainty — the confidence interval only captures parameter uncertainty. This means prediction intervals are always wider than confidence intervals at the same X value, regardless of sample size (making option B's explanation incorrect — the CI is not wider).

Question 3

A prediction interval for a new observation is always wider than the confidence interval for the mean at the same X value, even with a very large sample.

• T. True
• F. False

Answer: True

This follows from the formula: Var(prediction) = σ²[1 + X_new′(X′X)⁻¹X_new] vs. Var(mean estimate) = σ²[X_new′(X′X)⁻¹X_new]. The prediction interval always has the additional σ² term representing individual error variance, so it is strictly wider. As n → ∞, the X_new′(X′X)⁻¹X_new term shrinks to zero, but the prediction interval converges to ±1.96σ (not zero), while the confidence interval converges to a point. The gap between them actually grows in relative terms as n increases.

Question 4

As sample size grows toward infinity, both confidence intervals for the mean and prediction intervals for individual observations will eventually converge to a single point.

• T. True
• F. False

Answer: False

Only confidence intervals for the mean converge to a single point (the true conditional mean). Prediction intervals converge to ±1.96σ around the true mean — a non-zero width determined by the irreducible error variance σ². Even knowing the true regression line exactly, you cannot predict individual outcomes precisely, because each observation deviates from the line by its own error term ε, which is inherently random. A prediction interval reflects this fundamental uncertainty about the individual, not just uncertainty about the model parameters.

Question 5

Explain why prediction intervals widen as the predictor value X_new moves further from the mean of the training data. What are the statistical and practical implications of this widening?

Model answer: The width of a prediction interval depends on σ²[1 + X_new′(X′X)⁻¹X_new]. The term X_new′(X′X)⁻¹X_new is minimized when X_new equals the mean of X in the training data, and grows as X_new departs from that center. Geometrically, the regression line is 'anchored' most precisely at the center of the data; uncertainty in the estimated slope compounds as you move away. Practically, predictions made in the interior of the training data range are more reliable than extrapolations beyond it. For extrapolations, the formal prediction interval is wide, and this understates the true uncertainty because model misspecification risk (the true relationship may not be linear beyond the observed range) is not captured in the interval at all.

This is why analysts should always check whether new predictions lie within the range of the training data. A narrow prediction interval for in-sample predictions can create false confidence if the same model is applied extrapolatively — the interval widens formally, and unknown nonlinearities outside the data range add additional, unquantified risk.