Benign Overfitting Beyond Prediction: The Ordinary Least Squares Interpolator

Summary Recent advances in deep learning have highlighted the phenomenon of benign overfitting in overparameterized statistical models, sparking significant interest in understanding its foundations. Owing to its simplicity and practical relevance, the ordinary least-squares interpolator has become a key object of study for gaining theoretical insight into this phenomenon. While the properties of ordinary least squares are well understood in classical underparameterized settings, its behaviour in the overparameterized regime, unlike that of ridge regression or the lasso, remains comparatively less explored. We contribute to this growing literature by deriving new algebraic and statistical results for the minimum ℓ2-norm ordinary least-squares interpolator. In contrast to much of the existing work, which focuses on prediction risk, we centre our analysis on parameter estimation and inference, which are fundamental to many statistics and causal inference applications. Specifically, we establish overparameterized analogues of (i) the leave-k-out formulas, (ii) the omitted-variable bias formula and (iii) the Frisch–Waugh–Lovell theorem. Under the Gauss–Markov model, we further extend the Gauss–Markov theorem and analyse variance estimation under homoscedasticity in the overparameterized setting. Collectively, these results provide a systematic framework for studying parameter estimation and inference in overparameterized linear models, offering a novel perspective on benign overfitting beyond its implications for prediction.